Original: L EARNING O NE - HIDDEN - LAYER N EURAL N ETWORKS ON G AUSSIAN M IXTURE M ODELS WITH G UARANTEED G ENERALIZABILITY
Revised: L EARNING O NE - HIDDEN - LAYER N EURAL N ETWORKS ON G AUSSIAN M IXTURE M ODELS WITH G UARANTEED G ENERALIZABILITY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We analyze the learning problem of fully connected neural networks with the sigmoid activation function for binary classiﬁcation in the teacher-student setup, where the outputs are assumed to be generated by a ground-truth teacher neural network with unknown parameters, and the learning objective is to estimate the teacher network model by minimizing a non-convex cross-entropy risk function of the training data over a student neural network.
Revised: We analyze the learning problem of fully connected neural networks with the sigmoid activation function for binary classiﬁcation in the teacher-student setup, where the outputs are assumed to be generated by a ground-truth teacher neural network with unknown parameters, and the learning objective is to estimate the teacher network model by minimizing a non-convex cross-entropy risk function of the training data over a student neural network.

Original: This paper analyzes a general and practical scenario that the input features follow a Gaussian mixture model of a ﬁnite number of Gaussian distributions of various mean and variance.
Revised: This paper analyzes a general and practical scenario that the input features follow a Gaussian mixture model of a ﬁnite number of Gaussian distributions of various mean and variance.

Original: We propose a gradient descent algorithm with a tensor initialization approach and show that our algorithm converges linearly to a critical point that has a diminishing distance to the ground-truth model with guaranteed generalizability.
Revised: We propose a gradient descent algorithm with a tensor initialization approach and show that our algorithm converges linearly to a critical point that has a diminishing distance to the ground-truth model with guaranteed generalizability.

Original: We characterize the required number of samples for successful convergence, referred to as the sample complexity, as a function of the parameters of the Gaussian mixture model.
Revised: We characterize the required number of samples for successful convergence, referred to as the sample complexity, as a function of the parameters of the Gaussian mixture model.

Original: We prove analytically that when any mean or variance in the mixture model is large, or when all variances are close to zero, the sample complexity increases, and the convergence slows down, indicating a more challenging learning problem.
Revised: We prove analytically that when any mean or variance in the mixture model is large, or when all variances are close to zero, the sample complexity increases, and the convergence slows down, indicating a more challenging learning problem.

Original: Although focusing on one-hidden-layer neural networks, to the best of our knowledge, this paper provides the ﬁrst explicit characterization of the impact of the parameters of the input distributions on the sample complexity and learning rate
Revised: Although focusing on one-hidden-layer neural networks, to the best of our knowledge, this paper provides the ﬁrst explicit characterization of the impact of the parameters of the input distributions on the sample complexity and learning rate

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks (LeCun et al., 2015) have demonstrated superior empirical performance in various applications such as speech recognition (Krizhevsky et al., 2012) and computer vision (Graves et al., 2013;
Revised: Deep neural networks (LeCun et al., 2015) have demonstrated superior empirical performance in various applications such as speech recognition (Krizhevsky et al., 2012) and computer vision (Graves et al., 2013;

Original: He et al., 2016).
Revised: He et al., 2016).

Original: Despite the numerical success, the theoretical underpin of learning neural networks is much less investigated.
Revised: Despite the numerical success, the theoretical underpin of learning neural networks is much less investigated.

Original: One bottleneck for the wide acceptance of deep learning in critical applications is the lack of the theoretical generalization guarantees, i.e., why a model learned from the training data would achieve a high accuracy on the testing data.
Revised: One bottleneck for the wide acceptance of deep learning in critical applications is the lack of the theoretical generalization guarantees, i.e., why a model learned from the training data would achieve a high accuracy on the testing data.

Original: This paper studies the generalization performance of neural networks in the “teacher-student” setup, where the training data are generated by a teacher neural network, and the learning is performed on a student network by minimizing the empirical risk of the training data.
Revised: This paper studies the generalization performance of neural networks in the “teacher-student” setup, where the training data are generated by a teacher neural network, and the learning is performed on a student network by minimizing the empirical risk of the training data.

Original: This teacher-student setup has been studied in the statistical learning community for a long time (Engel & Broeck, 2001; Seung et al., 1992) and applied to neural networks recently (Goldt et al., 2019a; Zhong et al., 2017b;a;
Revised: This teacher-student setup has been studied in the statistical learning community for a long time (Engel & Broeck, 2001; Seung et al., 1992) and applied to neural networks recently (Goldt et al., 2019a; Zhong et al., 2017b;a;

Original: Zhang et al., 2019; 2020b; Fu et al., 2020; Zhang et al., 2020a).
Revised: Zhang et al., 2019; 2020b; Fu et al., 2020; Zhang et al., 2020a).

Original: Assuming that the student network has the same architecture as the teacher network, the existing generalization analyses mostly focus on one-hidden-layer networks, because the optimization problem is already nonconvex, and the analytical complexity increases tremendously
Revised: Assuming that the student network has the same architecture as the teacher network, the existing generalization analyses mostly focus on one-hidden-layer networks, because the optimization problem is already nonconvex, and the analytical complexity increases tremendously

Original: when the number of hidden layers increases.
Revised: when the number of hidden layers increases.

Original: One critical assumption of most works in this line is that the input features follow the standard Gaussian distribution.
Revised: One critical assumption of most works in this line is that the input features follow the standard Gaussian distribution.

Original: Although other distributions are considered in (Du et al., 2017; Ghorbani et al., 2006; Goldt et al., 2019b; Li & Liang, 2018; Mei et al., 2018b; Mignacco et al., 2006; Yoshida &
Revised: Although other distributions are considered in (Du et al., 2017; Ghorbani et al., 2006; Goldt et al., 2019b; Li & Liang, 2018; Mei et al., 2018b; Mignacco et al., 2006; Yoshida &

Original: Okada, 2019), the generalization performance beyond the standard Gaussian input is less investigated.
Revised: Okada, 2019), the generalization performance beyond the standard Gaussian input is less investigated.

Original: On the other hand, the learning performance clearly depends on the input data distribution.
Revised: On the other hand, the learning performance clearly depends on the input data distribution.

Original: (LeCun et al., 1998) states that the learning method converges faster if the inputs are whitened to be the standard Gaussian.
Revised: (LeCun et al., 1998) states that the learning method converges faster if the inputs are whitened to be the standard Gaussian.

Original: Batch normalization (Ioffe & Szegedy, 2015) modiﬁes the mean and variance in each layer and is a popular practical method to achieve a fast and stable convergence.
Revised: Batch normalization (Ioffe & Szegedy, 2015) modiﬁes the mean and variance in each layer and is a popular practical method to achieve a fast and stable convergence.

Original: Various explanations such as (Bjorck et al., 2018; Chai et al., 2020; Santurkar et al., 2018) have been proposed to explain the enormous success of Batch normalization, but little consensus exists on the exact mechanism.
Revised: Various explanations such as (Bjorck et al., 2018; Chai et al., 2020; Santurkar et al., 2018) have been proposed to explain the enormous success of Batch normalization, but little consensus exists on the exact mechanism.

Original: Contributions : This paper provides a theoretical analysis of learning one-hidden-layer neural networks when the input distribution follows a Gaussian mixture model containing an arbitrary number of Gaussian distributions with arbitrary mean and variance.
Revised: Contributions : This paper provides a theoretical analysis of learning one-hidden-layer neural networks when the input distribution follows a Gaussian mixture model containing an arbitrary number of Gaussian distributions with arbitrary mean and variance.

Original: The Gaussian mixture model has been employed in many applications such as data clustering and unsupervised learning (Dasgupta, 1999;
Revised: The Gaussian mixture model has been employed in many applications such as data clustering and unsupervised learning (Dasgupta, 1999;

Original: Figueiredo & Jain, 2002; Jain, 2010), and image classiﬁcation and segmentation (Permuter et al., 2006).
Revised: Figueiredo & Jain, 2002; Jain, 2010), and image classiﬁcation and segmentation (Permuter et al., 2006).

Original: The parameters of the mixture model can be estimated from data by the EM algorithm (Redner & Walker, 1984) or the moment-based method (Hsu & Kakade, 2013), with theoretical performance guarantees, see, e.g., (Ho & Nguyen, 2016; Ho et al., 2005; Dwivedi et al., 2020a;b).
Revised: The parameters of the mixture model can be estimated from data by the EM algorithm (Redner & Walker, 1984) or the moment-based method (Hsu & Kakade, 2013), with theoretical performance guarantees, see, e.g., (Ho & Nguyen, 2016; ? ; Dwivedi et al., 2020a;b).

Original: For the binary classiﬁcation problem with the cross entropy loss function, this paper proposes a gradient descent algorithm with tensor initialization to estimate the weights of the one-hidden-layer fully-connected neural network.
Revised: For the binary classiﬁcation problem with the cross entropy loss function, this paper proposes a gradient descent algorithm with tensor initialization to estimate the weights of the one-hidden-layer fully-connected neural network.

Original: Our algorithm converges to a critical point linearly, and the returned critical point converges to the ground-truth model at a rate of (cid:112) d log n/n , where d is the dimension of the feature, and n is the number of samples.
Revised: Our algorithm converges to a critical point linearly, and the returned critical point converges to the ground-truth model at a rate of (cid:112) d log n/n , where d is the dimension of the feature, and n is the number of samples.

Original: We also characterize the required number of samples for accurate estimation, referred to as the sample complexity, as a function of d , the number of neurons K , and the input distribution.
Revised: We also characterize the required number of samples for accurate estimation, referred to as the sample complexity, as a function of d , the number of neurons K , and the input distribution.

Original: Our explicit bounds imply (1) when the absolute value of any mean in the Gaussian mixture model increases from zero, the sample complexity increases, and the algorithm converges slower, indicating that it will be more challenging to learn a model with a small test error; (2) The same phenomenon happens when any variance in the mixture model increases to inﬁnity from a certain positive value, or if all the variances in the mixture model approach zero.
Revised: Our explicit bounds imply (1) when the absolute value of any mean in the Gaussian mixture model increases from zero, the sample complexity increases, and the algorithm converges slower, indicating that it will be more challenging to learn a model with a small test error; (2) The same phenomenon happens when any variance in the mixture model increases to inﬁnity from a certain positive value, or if all the variances in the mixture model approach zero.

Original: Our results indicate that the training converges faster and requires a less number of samples if the input data are zero mean with a certain non-zero variance.
Revised: Our results indicate that the training converges faster and requires a less number of samples if the input data are zero mean with a certain non-zero variance.

Original: This can be viewed as one theoretical explanation in one-hidden-layer for the success of Batch normalization.
Revised: This can be viewed as one theoretical explanation in one-hidden-layer for the success of Batch normalization.

Original: Moreover, to the best of our knowledge, this paper provides the ﬁrst theoretical and explicit characterization about how the mean and variance of the input distribution affect the sample complexity and learning rate .
Revised: Moreover, to the best of our knowledge, this paper provides the ﬁrst theoretical and explicit characterization about how the mean and variance of the input distribution affect the sample complexity and learning rate .

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: Learning over-parameterized neural networks.
Revised: Learning over-parameterized neural networks.

Original: One line of theoretical research on the learning performance considers the over-parameterized setting where the number of network parameters is greater than the number of training samples.
Revised: One line of theoretical research on the learning performance considers the over-parameterized setting where the number of network parameters is greater than the number of training samples.

Original: (Bousquet & Elisseeff, 2002; Hardt et al., 2016; Keskar et al., 2016; Livni et al., 2014; Neyshabur et al., 2017; Rumelhart et al., 1988; Soltanolkotabi et al., 2018; Allen-Zhu et al., 2019a).
Revised: (Bousquet & Elisseeff, 2002; Hardt et al., 2016; Keskar et al., 2016; Livni et al., 2014; Neyshabur et al., 2017; Rumelhart et al., 1988; Soltanolkotabi et al., 2018; Allen-Zhu et al., 2019a).

Original: (Allen-Zhu et al., 2019b; Du et al., 2019; Zou & Gu, 2019) show the deep neural networks can ﬁt all training samples in polynomial time.
Revised: (Allen-Zhu et al., 2019b; Du et al., 2019; Zou & Gu, 2019) show the deep neural networks can ﬁt all training samples in polynomial time.

Original: The optimization problem has no spurious local minima (Livni et al., 2014; Zhang et al., 2016; Soltanolkotabi et al., 2018), and the global minimum of the empirical risk function can be obtained by gradient descent (Li &
Revised: The optimization problem has no spurious local minima (Livni et al., 2014; Zhang et al., 2016; Soltanolkotabi et al., 2018), and the global minimum of the empirical risk function can be obtained by gradient descent (Li &

Original: Yuan, 2017; Du et al., 2018b; Zou et al., 2020).
Revised: Yuan, 2017; Du et al., 2018b; Zou et al., 2020).

Original: Although the returned model can achieve a zero training error, these works do not discuss whether it achieves a small test error or not.
Revised: Although the returned model can achieve a zero training error, these works do not discuss whether it achieves a small test error or not.

Original: (Allen-Zhu et al., 2019a; Li & Liang, 2018) analyze the generalization error by characterizing the training error and test error separately.
Revised: (Allen-Zhu et al., 2019a; Li & Liang, 2018) analyze the generalization error by characterizing the training error and test error separately.

Original: Still, there is no guarantee that a learned model with a small training error would have a small test error.
Revised: Still, there is no guarantee that a learned model with a small training error would have a small test error.

Original: (Cao & Gu, 2019) provides the bounds of the generalization error of the learned model by stochastic gradient descent (SGD) in deep neural networks, based on the assumption that there exists a good model with a small test error around the initialization of the SGD algorithm, and no discussion is provided about how to ﬁnd such an initialization.
Revised: (Cao & Gu, 2019) provides the bounds of the generalization error of the learned model by stochastic gradient descent (SGD) in deep neural networks, based on the assumption that there exists a good model with a small test error around the initialization of the SGD algorithm, and no discussion is provided about how to ﬁnd such an initialization.

Original: In contrast, our tensor initialization method in this paper provides an initialization that is close to the ground-truth teacher model such that our algorithm can ﬁnd this model with a zero test error.
Revised: In contrast, our tensor initialization method in this paper provides an initialization that is close to the ground-truth teacher model such that our algorithm can ﬁnd this model with a zero test error.

Original: Generalization performance with the standard Gaussian input.
Revised: Generalization performance with the standard Gaussian input.

Original: In the teacher-student setup of one-hidden-layer neural networks, (Brutzkus & Globerson, 2017; Du et al., 2018a; Ge et al., 2018; Liang et al., 2018; Li & Yuan, 2017; Shamir, 2018; Safran & Shamir, 2018; Tian, 2017) consider the ideal case of an inﬁnite number of training samples so that the training and test accuracy coincide and can be analyzed simultaneously.
Revised: In the teacher-student setup of one-hidden-layer neural networks, (Brutzkus & Globerson, 2017; Du et al., 2018a; Ge et al., 2018; Liang et al., 2018; Li & Yuan, 2017; Shamir, 2018; Safran & Shamir, 2018; Tian, 2017) consider the ideal case of an inﬁnite number of training samples so that the training and test accuracy coincide and can be analyzed simultaneously.

Original: When the number of training samples is ﬁnite, (Zhong et al., 2017b;a) characterize the sample complexity, i.e., the required number of samples, of learning one-hidden-layer fully connected neural networks with smooth activation functions and propose a gradient descent algorithm that converges to the ground-truth model linearly.
Revised: When the number of training samples is ﬁnite, (Zhong et al., 2017b;a) characterize the sample complexity, i.e., the required number of samples, of learning one-hidden-layer fully connected neural networks with smooth activation functions and propose a gradient descent algorithm that converges to the ground-truth model linearly.

Original: (Zhang et al., 2019; 2020b) extend the analyses to the non-smooth ReLU for fully-connected and convolutional neural networks, respectively.
Revised: (Zhang et al., 2019; 2020b) extend the analyses to the non-smooth ReLU for fully-connected and convolutional neural networks, respectively.

Original: (Zhang et al., 2020a) analyzes the generalizability of graph neural networks for both regression and binary classiﬁcation problems.
Revised: (Zhang et al., 2020a) analyzes the generalizability of graph neural networks for both regression and binary classiﬁcation problems.

Original: (Fu et al., 2020) analyzes the cross entropy loss function for binary classiﬁcation problems.
Revised: (Fu et al., 2020) analyzes the cross entropy loss function for binary classiﬁcation problems.

Original: Compared with other common loss functions such as the squared loss, the cross entropy loss function is harder to analyze due to the complicated forms and the saturation phenomenon of its Gradient and Hessian (Fu et al., 2020).
Revised: Compared with other common loss functions such as the squared loss, the cross entropy loss function is harder to analyze due to the complicated forms and the saturation phenomenon of its Gradient and Hessian (Fu et al., 2020).

Original: Theoretical characterization of learning performance from other input distributions.
Revised: Theoretical characterization of learning performance from other input distributions.

Original: (Du et al., 2017) considers rotationally invariant distributions, but the results only apply to a perceptron (i.e., a single-node network).
Revised: (Du et al., 2017) considers rotationally invariant distributions, but the results only apply to a perceptron (i.e., a single-node network).

Original: (Mei et al., 2018b) analyzes the generalization error of one-hidden-layer neural networks in the mean-ﬁeld limit trained on a large class of distributions, including a mixture of Gaussian distributions with the same mean.
Revised: (Mei et al., 2018b) analyzes the generalization error of one-hidden-layer neural networks in the mean-ﬁeld limit trained on a large class of distributions, including a mixture of Gaussian distributions with the same mean.

Original: The results only hold in the high-dimensional region where both the number of neurons K and the input dimension d are sufﬁciently large, and no sample complexity analysis is provided.
Revised: The results only hold in the high-dimensional region where both the number of neurons K and the input dimension d are sufﬁciently large, and no sample complexity analysis is provided.

Original: (Li & Liang, 2018) studies the generalization error of over-parameterized one-hidden-layer networks when the data come from mixtures of well-separated distribution, but the separation requirement excludes Gaussian distributions and Gaussian mixture models.
Revised: (Li & Liang, 2018) studies the generalization error of over-parameterized one-hidden-layer networks when the data come from mixtures of well-separated distribution, but the separation requirement excludes Gaussian distributions and Gaussian mixture models.

Original: (Yoshida & Okada, 2019) analyzes the Plateau Phenomenon that the decrease of the risk slows down signiﬁcantly partway and speeds up again in one-hidden-layer neural networks with inputs drawn from a single Gaussian with an arbitrary covariance.
Revised: (Yoshida & Okada, 2019) analyzes the Plateau Phenomenon that the decrease of the risk slows down signiﬁcantly partway and speeds up again in one-hidden-layer neural networks with inputs drawn from a single Gaussian with an arbitrary covariance.

Original: (Goldt et al., 2019b; 2006) analyze the dynamics of learning one-hidden-layer networks with SGD when the inputs are drawn from a wide class of generative models.
Revised: (Goldt et al., 2019b; 2006) analyze the dynamics of learning one-hidden-layer networks with SGD when the inputs are drawn from a wide class of generative models.

Original: (Mignacco et al., 2006) provides analytical equations for SGD evolution in a perceptron trained on the Gaussian mixture model.
Revised: (Mignacco et al., 2006) provides analytical equations for SGD evolution in a perceptron trained on the Gaussian mixture model.

Original: (Ghorbani et al., 2006) considers inputs with low-dimensional structures and compares neural networks with kernel methods.
Revised: (Ghorbani et al., 2006) considers inputs with low-dimensional structures and compares neural networks with kernel methods.

Original: Notations : Vectors are in bold lowercase, matrices and tensors in are bold uppercase.
Revised: Notations : Vectors are in bold lowercase, matrices and tensors in are bold uppercase.

Original: Scalars are in normal fonts.
Revised: Scalars are in normal fonts.

Original: For instance, Z is a matrix, and z is a vector.
Revised: For instance, Z is a matrix, and z is a vector.

Original: z i denotes the i -th entry of z , and
Revised: z i denotes the i -th entry of z , and

Original: Z i,j denotes the ( i, j ) -th entry of Z .
Revised: Z i,j denotes the ( i, j ) -th entry of Z .

Original: [ K ] ( K > 0 ) denotes the set including integers from 1 to
Revised: [ K ] ( K > 0 ) denotes the set including integers from 1 to

Original: K .
Revised: K .

Original: I d ∈ R d × d
Revised: I d ∈ R d × d

Original: and e i represent the identity matrix in R d × d
Revised: and e i represent the identity matrix in R d × d

Original: and the i -th standard basis vector, respectively.
Revised: and the i -th standard basis vector, respectively.

Original: We use δ i ( Z ) to denote the i -th largest singular value of Z .
Revised: We use δ i ( Z ) to denote the i -th largest singular value of Z .

Original: A (cid:23) 0 means A is a positive semi-deﬁnite (PSD) matrix.
Revised: A (cid:23) 0 means A is a positive semi-deﬁnite (PSD) matrix.

Original: The gradient and the Hessian of a function f ( W ) are denoted by ∇ f ( W ) and ∇ 2 f ( W ) , respectively.
Revised: The gradient and the Hessian of a function f ( W ) are denoted by ∇ f ( W ) and ∇ 2 f ( W ) , respectively.

Original: The outer product of vectors z i ∈ R n i , i ∈
Revised: The outer product of vectors z i ∈ R n i , i ∈

Original: [ l ] , is deﬁned as T =
Revised: [ l ] , is deﬁned as T =

Original: z 1 ⊗ · · · × z l ∈ R n 1 ×···× n l with T j 1 ··· j l = ( z 1 ) j 1 · · · ( z l ) j l .
Revised: z 1 ⊗ · · · × z l ∈ R n 1 ×···× n l with T j 1 ··· j l = ( z 1 ) j 1 · · · ( z l ) j l .

Original: Given a tensor T ∈ R n 1 × n 2 × n 3 and matrices A ∈ R n 1 × d 1 , B ∈ R n 2 × d 2 , C ∈ R n 3 × d 3 , the ( i 1 , i 2 , i 3 ) -th entry of the tensor T ( A , B , C ) is given by
Revised: Given a tensor T ∈ R n 1 × n 2 × n 3 and matrices A ∈ R n 1 × d 1 , B ∈ R n 2 × d 2 , C ∈ R n 3 × d 3 , the ( i 1 , i 2 , i 3 ) -th entry of the tensor T ( A , B , C ) is given by

Original: [Equation]
Revised: [Equation]

Original: We follow the convention that f ( x ) =
Revised: We follow the convention that f ( x ) =

Original: O ( g ( x ))
Revised: O ( g ( x ))

Original: (or Ω( g ( x ) , Θ( g ( x ))) means that f ( x ) increases at most, at least, or in the order of g ( x ) , respectively.
Revised: (or Ω( g ( x ) , Θ( g ( x ))) means that f ( x ) increases at most, at least, or in the order of g ( x ) , respectively.

Original:  P ROBLEM FORMULATION
Revised:  P ROBLEM FORMULATION

Original: We consider a one-hidden-layer fully connected neural network where all the weights in the second layer have the same ﬁxed value.
Revised: We consider a one-hidden-layer fully connected neural network where all the weights in the second layer have the same ﬁxed value.

Original: This structure is also known as the committee machine, see, e.g., (Aubin et al., 2018; Monasson & Zecchina, 1995; Schwarze & Hertz, 1992a;b).
Revised: This structure is also known as the committee machine, see, e.g., (Aubin et al., 2018; Monasson & Zecchina, 1995; Schwarze & Hertz, 1992; 1993).

Original: Let x ∈ R
Revised: Let x ∈ R

Original: d
Revised: d

Original: denote the input features.
Revised: denote the input features.

Original: Let K ≥ 1 be the number of neurons in the hidden layer.
Revised: Let K ≥ 1 be the number of neurons in the hidden layer.

Original: Following the teacherstudent setup, see e.g., (Fu et al., 2020), the output labels are generated by a teacher neural network with unknown ground-truth weights w ∗ j ∈ R d ( j ∈ [ K ] ).
Revised: Following the teacher-student setup, see e.g., (Fu et al., 2020), the output labels are generated by a teacher neural network with unknown ground-truth weights w ∗ j ∈ R d ( j ∈ [ K ] ).

Original: Let 
Revised: Let W ∗ =

Original: [Equation]
Revised: [ w ∗ 1 , ..., w ∗ K ] ∈

Original:  contain all the weights.
Revised: R d × K contain all the weights.

Original: Let δ i ( W ∗ ) denote the i -th largest singular value of W ∗ .
Revised: Let δ i ( W ∗ ) denote the i -th largest singular value of W ∗ .

Original: Let κ = δ 1 ( W ∗ ) δ K ( W ∗ ) , and deﬁne η 1 = (cid:81) Ki =
Revised: Let κ = δ 1 ( W ∗ ) δ K ( W ∗ ) , and deﬁne η 1 = (cid:81) Ki =

Original: δ i ( W ∗ ) δ K ( W ∗ ) .
Revised: δ i ( W ∗ ) δ K ( W ∗ ) .

Original: The nonlinear activation function here is the sigmoid function φ ( x ) = 1 1+exp( − x ) .
Revised: The nonlinear activation function here is the sigmoid function φ ( x ) = 1 1+exp( − x ) .

Original: We consider binary classiﬁcation, and the binary output y is generated by the teacher committee machine through
Revised: We consider binary classiﬁcation, and the binary output y is generated by the teacher committee machine through

Original: [Equation]
Revised: [Equation]

Original: Learning is performed over a student neural network that has the same architecture as the teacher network, and its weights are denoted by W ∈ R d × K .
Revised: Learning is performed over a student neural network that has the same architecture as the teacher network, and its weights are denoted by W ∈ R d × K .

Original: Given n pairs of training samples { x i , y i } ni =1 , the empirical risk function is
Revised: Given n pairs of training samples { x i , y i } ni =1 , the empirical risk function is

Original: [Equation]
Revised: [Equation]

Original: where (cid:96) ( W ; x i , y i ) is the cross-entropy loss function, i.e., (cid:96) (
Revised: where (cid:96) ( W ; x i , y i ) is the cross-entropy loss function, i.e., (cid:96) (

Original: [Equation]
Revised: [Equation]

Original: To estimate W ∗ from training samples, we solve the following nonconvex minimization problem
Revised: To estimate W ∗ from training samples, we solve the following nonconvex minimization problem

Original: [Equation]
Revised: [Equation]

Original: Here we assume the input features x i are generated i.i.d.
Revised: Here we assume the input features x i are generated i.i.d.

Original: from the Gaussian mixture model (Pearson, 1894; Titterington et al., 1985; Hsu & Kakade, 2013), which we denote as
Revised: from the Gaussian mixture model (Pearson, 1894; Titterington et al., 1985; Hsu & Kakade, 2013), which we denote as

Original: [Equation]
Revised: [Equation]

Original: where N denotes the multi-variate Gaussian distribution with mean µ l ∈ R d , and covariance σ l I d for σ l ∈ R + for all l ∈
Revised: where N denotes the multi-variate Gaussian distribution with mean µ l ∈ R d , and covariance σ l I d for σ l ∈ R + for all l ∈

Original: [ L ] .
Revised: [ L ] .

Original: The Gaussian mixture model can be viewed as
Revised: The Gaussian mixture model can be viewed as

Original: [Equation]
Revised: [Equation]

Original: where h is a discrete random variable with Pr( h = l )
Revised: where h is a discrete random variable with Pr( h = l )

Original: =
Revised: =

Original: λ l for l ∈
Revised: λ l for l ∈

Original: [ L ] , and z l follows the multivariate Gaussian N ( 0 , σ 2 l I d ) with zero mean and covariance σ 2 l I d 1 .
Revised: [ L ] , and z l follows the multivariate Gaussian N ( 0 , σ 2 l I d ) with zero mean and covariance σ 2 l I d 1 .

Original: If the Gaussian mixture model is symmetric, the symmetric distribution can be written as
Revised: If the Gaussian mixture model is symmetric, the symmetric distribution can be written as

Original: [Equation]
Revised: [Equation]

Original: We assume without loss of generality that µ l belongs to the column space of W ∗ for all l ∈ [ L ] .
Revised: We assume without loss of generality that µ l belongs to the column space of W ∗ for all l ∈ [ L ] .

Original: To see this, note that an arbitrary µ l can be written as µ l (cid:107) + µ l ⊥ , where µ l (cid:107) belongs to the column space of W ∗ , and µ l ⊥ is perpendicular to the column space.
Revised: To see this, note that an arbitrary µ l can be written as µ l (cid:107) + µ l ⊥ , where µ l (cid:107) belongs to the column space of W ∗ , and µ l ⊥ is perpendicular to the column space.

Original: Then, from (2) and (7) we have
Revised: Then, from (2) and (7) we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where x (cid:48) ∼ (cid:80) Ll =1 λ l N ( µ l (cid:107) , σ 2 l I d ) .
Revised: where x (cid:48) ∼ (cid:80) Ll =1 λ l N ( µ l (cid:107) , σ 2 l I d ) .

Original: Thus, these two cases are equivalent.
Revised: Thus, these two cases are equivalent.

Original:  P ROPOSED L EARNING A LGORITHM
Revised:  P ROPOSED L EARNING A LGORITHM

Original: We propose Algorithm 1 to solve (5) and defer its theoeretical analysis to Section 4.
Revised: We propose Algorithm 1 to solve (5) and defer its theoeretical analysis to Section 4.

Original: The method starts from a initialization W 0 ∈ R d × K computed based on the tensor initialization method (Subroutine 1) and then updates the iterates W t using gradient descent with the step size η 0 .
Revised: The method starts from a initialization W 0 ∈ R d × K computed based on the tensor initialization method (Subroutine 1) and then updates the iterates W t using gradient descent with the step size η 0 .

Original: Our tensor initialization method is extended from (Janzamin et al., 2014) and (Zhong et al., 2017b).
Revised: Our tensor initialization method is extended from (Janzamin et al., 2014) and (Zhong et al., 2017b).

Original: The idea is to compute quantities ( M j in (10)) that are tensors of w ∗ i and then apply the tensor decomposition method to estimate w ∗ i .
Revised: The idea is to compute quantities ( M j in (10)) that are tensors of w ∗ i and then apply the tensor decomposition method to estimate w ∗ i .

Original: Because M j can only be estimated from training samples, tensor decomposition does not return w ∗ i exactly but provides a close approximation.
Revised: Because M j can only be estimated from training samples, tensor decomposition does not return w ∗ i exactly but provides a close approximation.

Original: Because the existing method only applies to the standard Gaussian, we exploit the relationship between probability density functions and tensor expressions developed in (Janzamin et al., 2014) to design tensors suitable for the Gaussian mixture model.
Revised: Because the existing method only applies to the standard Gaussian, we exploit the relationship between probability density functions and tensor expressions developed in (Janzamin et al., 2014) to design tensors suitable for the Gaussian mixture model.

Original: Formally,
Revised: Formally,

Original: Input: Training data { ( x i , y i ) } ni =1 , the step size η 0 = O (cid:16) 1 (cid:80)
Revised: Input: Training data { ( x i , y i ) } ni =1 , the step size η 0 = O (cid:16) 1 (cid:80)

Original: Ll
Revised: Ll

Original: =1 λ l ( (cid:107) µ l (cid:107) ∞ + σ l )(cid:17) , iteration T
Revised: =1 λ l ( (cid:107) µ l (cid:107) ∞ + σ l )(cid:17) , iteration T

Original: Initialization: W 0 ← Tensor initialization method via Subroutine
Revised: Initialization: W 0 ← Tensor initialization method via Subroutine

Original: Gradient Descent: for t = 0 , 1 , · · · , T −
Revised: Gradient Descent: for t = 0 , 1 , · · · , T −

Original: [Equation]
Revised: [Equation]

Original: Output: W T
Revised: Output: W T

Original: Deﬁnition 1 Let p
Revised: Deﬁnition 1 Let p

Original: [Equation]
Revised: [Equation]

Original: x ) =
Revised: x ) =

Original: [Equation]
Revised: [Equation]

Original: ) be the probability density function of the Gaussian mixture model in (6).
Revised: ) be the probability density function of the Gaussian mixture model in (6).

Original: We deﬁne
Revised: We deﬁne

Original: [Equation]
Revised: [Equation]

Original: Let α ∈ R d denote an arbitrary vector.
Revised: Let α ∈ R d denote an arbitrary vector.

Original: If the Gaussian Mixture Model is symmetric as in (8), then
Revised: If the Gaussian Mixture Model is symmetric as in (8), then

Original: P 2 :=
Revised: P 2 :=

Original: M 3 ( I d , I d , α ) .
Revised: M 3 ( I d , I d , α ) .

Original: Otherwise, P 2 :=
Revised: Otherwise, P 2 :=

Original: M 2 .
Revised: M 2 .

Original: M j is a j th-order tensor of w ∗ i , e.g., M
Revised: M j is a j th-order tensor of w ∗ i , e.g., M

Original: [Equation]
Revised: [Equation]

Original: These quantiﬁes cannot be directly computed from (10) but can be estimated by sample means, denoted by (cid:99) M i ( i = 1 , 2 , 3 ) and (cid:98) P 2 , from samples { x i , y i } ni =1 .
Revised: These quantiﬁes cannot be directly computed from (10) but can be estimated by sample means, denoted by (cid:99) M i ( i = 1 , 2 , 3 ) and (cid:98) P 2 , from samples { x i , y i } ni =1 .

Original: The following assumption guarantees that these tensors are nonzero and can thus be leveraged to estimate W ∗ .
Revised: The following assumption guarantees that these tensors are nonzero and can thus be leveraged to estimate W ∗ .

Original: Assumption 1 The Gaussian Mixture Model in (6) satisﬁes the following conditions:
Revised: Assumption 1 The Gaussian Mixture Model in (6) satisﬁes the following conditions:

Original: E x ∼ (cid:80) Ll
Revised: E x ∼ (cid:80) Ll

Original: =1 λ l N ( µ l ,σ 2 l I )
Revised: =1 λ l N ( µ l ,σ 2 l I )

Original: [ φ (cid:48)(cid:48)(cid:48) ( w ∗ i (cid:62) x )] (cid:54) = 0 for i ∈
Revised: [ φ (cid:48)(cid:48)(cid:48) ( w ∗ i (cid:62) x )] (cid:54) = 0 for i ∈

Original: [ K ] , which implies that M 3 is nonzero.
Revised: [ K ] , which implies that M 3 is nonzero.

Original: If the distribution is not symmetric, then E x ∼ (cid:80) Ll =1 λ l N ( µ l ,σ 2 l I ) [ φ (cid:48)(cid:48) ( w ∗ i (cid:62) x )] (cid:54) = 0 for i ∈
Revised: If the distribution is not symmetric, then E x ∼ (cid:80) Ll =1 λ l N ( µ l ,σ 2 l I ) [ φ (cid:48)(cid:48) ( w ∗ i (cid:62) x )] (cid:54) = 0 for i ∈

Original: [ K ] , which implies M 2 and P 2 in this case are nonzero.
Revised: [ K ] , which implies M 2 and P 2 in this case are nonzero.

Original: Note that Assumption 1 is a very mild assumption 2 .
Revised: Note that Assumption 1 is a very mild assumption 2 .

Original: Moreover, as indicted in (Janzamin et al., 2014), in the rare case that some quantities M i ( i = 1 , 2 , 3 ) and P 2 are zero, one can construct higher-order tensors in a similar way as in Deﬁnition 1 and then estimate W ∗ from higher-order tensors.
Revised: Moreover, as indicted in (Janzamin et al., 2014), in the rare case that some quantities M i ( i = 1 , 2 , 3 ) and P 2 are zero, one can construct higher-order tensors in a similar way as in Deﬁnition 1 and then estimate W ∗ from higher-order tensors.

Original: Subroutine 1 estimates the direction and magnitude of w ∗ j , j ∈
Revised: Subroutine 1 estimates the direction and magnitude of w ∗ j , j ∈

Original: [ K ] , separately.
Revised: [ K ] , separately.

Original: The key steps are as follows.
Revised: The key steps are as follows.

Original: We ﬁrst use the power method to decompose (cid:98) P 2 to approximate the subspace spanned by { w ∗ 1 , w ∗ 2 , · · · , w ∗ K } , denoted by (cid:98) U .
Revised: We ﬁrst use the power method to decompose (cid:98) P 2 to approximate the subspace spanned by { w ∗ 1 , w ∗ 2 , · · · , w ∗ K } , denoted by (cid:98) U .

Original: Then, we project (cid:99) M 3 ∈ R d × d × d to (cid:98) R 3 ∈ R K × K × K using (cid:98) U to reduce the computational and sample complexity for decomposing a third-order tensor in the next step.
Revised: Then, we project (cid:99) M 3 ∈ R d × d × d to (cid:98) R 3 ∈ R K × K × K using (cid:98) U to reduce the computational and sample complexity for decomposing a third-order tensor in the next step.

Original: We then apply the KCL algorithm to decompose (cid:98) R 3 into vectors (cid:98) v i .
Revised: We then apply the KCL algorithm to decompose (cid:98) R 3 into vectors (cid:98) v i .

Original: Note that (cid:98) U (cid:62) ˆ v i
Revised: Note that (cid:98) U (cid:62) ˆ v i

Original: =
Revised: =

Original: s i ¯ w ∗
Revised: s i ¯ w ∗

Original: i , where s i ∈ { 1 , − 1 } is a random sign.
Revised: i , where s i ∈ { 1 , − 1 } is a random sign.

Original: Then the direction of w ∗ j is determined.
Revised: Then the direction of w ∗ j is determined.

Original: Finally, the magnitude of w ∗ i ’s and the signs of s
Revised: Finally, the magnitude of w ∗ i ’s and the signs of s

Original: i ’s are determined by solving a linear system of equations using the RecMagSign method.
Revised: i ’s are determined by solving a linear system of equations using the RecMagSign method.

Original: Please refer to (Zhong et al., 2017b) and (Kuleshov et al., 2015) for more details on the power method, KCL and RecMagSign methods.
Revised: Please refer to (Zhong et al., 2017b) and (Kuleshov et al., 2015) for more details on the power method, KCL and RecMagSign methods.

Original:  M AIN T HEORETICAL R ESULTS
Revised:  M AIN T HEORETICAL R ESULTS

Original: The main idea of our analysis is to show that the empirical risk function in (3) is strongly convex in a region near W ∗ .
Revised: The main idea of our analysis is to show that the empirical risk function in (3) is strongly convex in a region near W ∗ .

Original: Then W 0 returned by Subroutine 3 is in this convex region, and the iterates returned by Algorithm 1 converge to a critical point in this region.
Revised: Then W 0 returned by Subroutine 3 is in this convex region, and the iterates returned by Algorithm 1 converge to a critical point in this region.

Original: Before formally stating our result in Theorem 1, we summarize the key implications of Theorem 1 as follow.
Revised: Before formally stating our result in Theorem 1, we summarize the key implications of Theorem 1 as follow.

Original: Subroutine 1 Tensor Initialization Method
Revised: Subroutine 1 Tensor Initialization Method

Original: Input: Partition n pairs of data { ( x i , y i ) } ni =1 into three subsets D 1 , D 2 , D
Revised: Input: Partition n pairs of data { ( x i , y i ) } ni =1 into three subsets D 1 , D 2 , D

Original: Output:
Revised: Output:

Original: Compute (cid:98) P 2 using D 1 and an arbitrary vector α (cid:98) U
Revised: Compute (cid:98) P 2 using D 1 and an arbitrary vector α (cid:98) U

Original: ←−
Revised: ←−

Original: PowerMethod
Revised: PowerMethod

Original: (cid:98) P 2 , K )
Revised: (cid:98) P 2 , K )

Original: Compute (cid:98) R 3 = (cid:99) M 3 ( (cid:98) U , (cid:98) U , (cid:98) U ) from data set D{ (cid:98) v i } i ∈
Revised: Compute (cid:98) R 3 = (cid:99) M 3 ( (cid:98) U , (cid:98) U , (cid:98) U ) from data set D{ (cid:98) v i } i ∈

Original: [ K ] ←− KCL ( (cid:98) R 3 ) { W 0 } ←−
Revised: [ K ] ←− KCL ( (cid:98) R 3 ) { W 0 } ←−

Original: RecMagSign ( (cid:98) U , { ˆ v i } i ∈
Revised: RecMagSign ( (cid:98) U , { ˆ v i } i ∈

Original: [ K ] , D 3 )
Revised: [ K ] , D 3 )

Original: Return: W1.
Revised: Return: W1.

Original: Convergence rate and estimation accuracy : The iterates W t converge to a critical point (cid:99) W n linearly, and the distance between (cid:99) W n and W ∗ decays in the rate of (cid:112) d log n/n .
Revised: Convergence rate and estimation accuracy : The iterates W t converge to a critical point (cid:99) W n linearly, and the distance between (cid:99) W n and W ∗ decays in the rate of (cid:112) d log n/n .

Original: For example, when the number of samples n is Θ( d log 2 d ) , the estimation error decays in the order of 1 log d .
Revised: For example, when the number of samples n is Θ( d log 2 d ) , the estimation error decays in the order of 1 log d .

Original: Sample complexity : The sample complexity for accurate estimation is Θ( d log 2 d ) where d is the feature dimension.
Revised: Sample complexity : The sample complexity for accurate estimation is Θ( d log 2 d ) where d is the feature dimension.

Original: This result is in the same order as the sample complexity for the standard
Revised: This result is in the same order as the sample complexity for the standard

Original: Gaussian input in (Fu et al., 2020) and (Zhong et al., 2017b), indicating that our method can handle input from the Gaussian mixture model without increasing the order of the sample complexity.
Revised: Gaussian input in (Fu et al., 2020) and (Zhong et al., 2017b), indicating that our method can handle input from the Gaussian mixture model without increasing the order of the sample complexity.

Original: Our bound is almost order-wise optimal with respect to d because the degree of freedom is dK .
Revised: Our bound is almost order-wise optimal with respect to d because the degree of freedom is dK .

Original: The additional multiplier of log 2 d results from the concentration bound in the proof technique.
Revised: The additional multiplier of log 2 d results from the concentration bound in the proof technique.

Original: Impact of the mean : If everything else is ﬁxed, and at least one entry of a mean µ l ( i ) (the i th entry of µ l ) of the Gaussian mixture model increases from 0 (in terms of the absolute value), the sample complexity increases to inﬁnity and the convergence slows down.
Revised: Impact of the mean : If everything else is ﬁxed, and at least one entry of a mean µ l ( i ) (the i th entry of µ l ) of the Gaussian mixture model increases from 0 (in terms of the absolute value), the sample complexity increases to inﬁnity and the convergence slows down.

Original: The intuition is that as the absolute value of some mean increases, some training samples have signiﬁcantly large magnitude such that the sigmoid function saturates.
Revised: The intuition is that as the absolute value of some mean increases, some training samples have signiﬁcantly large magnitude such that the sigmoid function saturates.

Original: These training samples are not informative for the estimation of
Revised: These training samples are not informative for the estimation of

Original: W ∗ , and the gradient of these samples is close to zero.
Revised: W ∗ , and the gradient of these samples is close to zero.

Original: Therefore, the required number of samples to estimate W ∗ needs to increase, and the gradient descent algorithm slows down.
Revised: Therefore, the required number of samples to estimate W ∗ needs to increase, and the gradient descent algorithm slows down.

Original: Impact of the variance : If everything else is ﬁxed, and at least one variance σ l of the Gaussian mixture model increases from a certain positive value, the sample complexity increases to inﬁnity and the convergence slows down.
Revised: Impact of the variance : If everything else is ﬁxed, and at least one variance σ l of the Gaussian mixture model increases from a certain positive value, the sample complexity increases to inﬁnity and the convergence slows down.

Original: The intuition is the same as increasing | µ l ( i ) | in point 3.
Revised: The intuition is the same as increasing | µ l ( i ) | in point 3.

Original: On the other hand, when all variances in the Gaussian mixture model approach zero, the sample complexity increases to inﬁnity, and the convergence slows down.
Revised: On the other hand, when all variances in the Gaussian mixture model approach zero, the sample complexity increases to inﬁnity, and the convergence slows down.

Original: The intuition is that when the input data are concentrated on a few vectors, the optimization problem does not have a benign landscape.
Revised: The intuition is that when the input data are concentrated on a few vectors, the optimization problem does not have a benign landscape.

Original: Combining points 3 and 4, one can see that to learn the teacher network characterized by (2), the training samples shall have zero mean and a medium level of variance to reduce the sample complexity and speed up the convergence.
Revised: Combining points 3 and 4, one can see that to learn the teacher network characterized by (2), the training samples shall have zero mean and a medium level of variance to reduce the sample complexity and speed up the convergence.

Original: If the variance is too large, some samples become non-informative and affect the learning negatively.
Revised: If the variance is too large, some samples become non-informative and affect the learning negatively.

Original: If the variance is too small, the learning problem becomes mathematically challenging to solve.
Revised: If the variance is too small, the learning problem becomes mathematically challenging to solve.

Original: This theoretical characterization can be viewed as one motivation of the empirical techniques to improve learning rate such as whiting (LeCun et al., 1998) and Batch normalization (Ioffe & Szegedy, 2015).
Revised: This theoretical characterization can be viewed as one motivation of the empirical techniques to improve learning rate such as whiting (LeCun et al., 1998) and Batch normalization (Ioffe & Szegedy, 2015).

Original: We state our main theoretical result as follows.
Revised: We state our main theoretical result as follows.

Original: Theorem 1 Consider the binary classiﬁcation problem with one hidden-layer fully connected neural network as in (2).
Revised: Theorem 1 Consider the binary classiﬁcation problem with one hidden-layer fully connected neural network as in (2).

Original: Suppose Assumption 1 holds, then there exist (cid:15) 0 ∈ (0 , 14 ) and positive value functions B ( λ , M , σ , W ∗ ) and q ( λ , M , σ , W ∗ ) such that as long as the sample size n satisﬁes
Revised: Suppose Assumption 1 holds, then there exist (cid:15) 0 ∈ (0 , 14 ) and positive value functions B ( λ , M , σ , W ∗ ) and q ( λ , M , σ , W ∗ ) such that as long as the sample size n satisﬁes

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: we have that with probability at least 1 − d − 10 , the iterates { W t } Tt =1 returned by Algorithm 1 with step size η 

Revised: we have that with probability at least 1 − d − 10 , the iterates { W t } Tt =1 returned by Algorithm 1 with step size η

Original: = O (cid:16)
Revised: = O (cid:16)

Original: [Equation]
Revised: [Equation]

Original: converge linearly to a critical point (cid:99) W n with the rate of convergence v = 1 − K − 2 q ( λ , M , σ , W ∗ ) , i.e.,
Revised: converge linearly to a critical point (cid:99) W n with the rate of convergence v = 1 − K − 2 q ( λ , M , σ , W ∗ ) , i.e.,

Original: [Equation]
Revised: [Equation]

Original: Moreover, the distance between W ∗ and (cid:99) W n is bounded by
Revised: Moreover, the distance between W ∗ and (cid:99) W n is bounded by

Original: [Equation]
Revised: [Equation]

Original: We next quantify the impact of the parameters of the Gaussian mixture model on the sample complexity n sc and the convergence rate v discussed in Theorem 1 as follows.
Revised: We next quantify the impact of the parameters of the Gaussian mixture model on the sample complexity n sc and the convergence rate v discussed in Theorem 1 as follows.

Original: Corollary 1 (Impact of the Gaussian mixture model on n sc and v ) (1) When everything else is ﬁxed, n sc increases to inﬁnity, and v increases to 1, as | µ l ( i ) | with any l ∈ [ L ] and i ∈ [ d ] increases, where µ l ( i ) is the i -th entry of µ l .
Revised: Corollary 1 (Impact of the Gaussian mixture model on n sc and v ) (1) When everything else is ﬁxed, n sc increases to inﬁnity, and v increases to 1, as | µ l ( i ) | with any l ∈ [ L ] and i ∈ [ d ] increases, where µ l ( i ) is the i -th entry of µ l .

Original: When everything else is ﬁxed except for some σ l for any l ∈ [ L ] , n sc increases to inﬁnity, and v increases to 1, as σ l increases from ζ s for some constant ζ s > 0 .
Revised: When everything else is ﬁxed except for some σ l for any l ∈ [ L ] , n sc increases to inﬁnity, and v increases to 1, as σ l increases from ζ s for some constant ζ s > 0 .

Original: (3) n sc increases to inﬁnity, and v increases to 1 if all σ l ’s go to zero for all l ∈
Revised: (3) n sc increases to inﬁnity, and v increases to 1 if all σ l ’s go to zero for all l ∈

Original: [ L ] .
Revised: [ L ] .

Original: To the best of our knowledge, Theorem 1 provides the ﬁrst explicit characterization of the sample complexity and learning rate when the input follows the Gaussian mixture model.
Revised: To the best of our knowledge, Theorem 1 provides the ﬁrst explicit characterization of the sample complexity and learning rate when the input follows the Gaussian mixture model.

Original: Although we consider the sigmoid activation in this paper, our results apply to any activation function φ provided that φ (cid:48) is an even function, and φ , φ (cid:48) and φ (cid:48)(cid:48) are bounded.
Revised: Although we consider the sigmoid activation in this paper, our results apply to any activation function φ provided that φ (cid:48) is an even function, and φ , φ (cid:48) and φ (cid:48)(cid:48) are bounded.

Original: Examples include tanh and erf .
Revised: Examples include tanh and erf .

Original: Algorithm 1 employs a constant step size.
Revised: Algorithm 1 employs a constant step size.

Original: One can potentially speed up the convergence, i.e., reduce v , by using a variable step size.
Revised: One can potentially speed up the convergence, i.e., reduce v , by using a variable step size.

Original: We leave the corresponding theoretical analysis for future work.
Revised: We leave the corresponding theoretical analysis for future work.

Original: If we scale the weights W ∗ (cid:48)
Revised: If we scale the weights W ∗ (cid:48)

Original: =
Revised: =

Original: W ∗ /c and the input feature x (cid:48) =
Revised: W ∗ /c and the input feature x (cid:48) =

Original: c x simultaneously, the output remains the same for any nonzero constant c .
Revised: c x simultaneously, the output remains the same for any nonzero constant c .

Original: Therefore, the learning problems in these two cases are equivalent in terms of the sample complexity and convergence rate.
Revised: Therefore, the learning problems in these two cases are equivalent in terms of the sample complexity and convergence rate.

Original: Theorem 1 reﬂects such equivalence.
Revised: Theorem 1 reﬂects such equivalence.

Original: One can check that B ( λ , M , σ , W ∗ ) =
Revised: One can check that B ( λ , M , σ , W ∗ ) =

Original: B ( λ , M (cid:48) , σ (cid:48) , W ∗(cid:48) )
Revised: B ( λ , M (cid:48) , σ (cid:48) , W ∗(cid:48) )

Original: from the proof in Section
Revised: from the proof in Section

Original: B. Similarly, the convergence rate in (12) remains the same in both cases.
Revised: B. Similarly, the convergence rate in (12) remains the same in both cases.

Original: One main component in the proof of Theorem 1 to show that if (11) holds, the landscape of the empirical risk is close to that of the population risk in a local neighborhood of W ∗ .
Revised: One main component in the proof of Theorem 1 to show that if (11) holds, the landscape of the empirical risk is close to that of the population risk in a local neighborhood of W ∗ .

Original: (Mei et al., 2018a) quantiﬁed the similarity of these two functions when K = 1 , but it is not clear if their approach can be extended to the case K > 1 .
Revised: (Mei et al., 2018a) quantiﬁed the similarity of these two functions when K = 1 , but it is not clear if their approach can be extended to the case K > 1 .

Original: Here, focusing on the Gaussian mixture model, we explicitly quantify the impact of the parameters of the input distribution on the landscapes of these functions.
Revised: Here, focusing on the Gaussian mixture model, we explicitly quantify the impact of the parameters of the input distribution on the landscapes of these functions.

Original: Please see Appendix-C for details.
Revised: Please see Appendix-C for details.

Original: Compared with the analyses for the standard Gaussian in (Fu et al., 2020; Zhong et al., 2017b), we develop new techniques in the following aspects.
Revised: Compared with the analyses for the standard Gaussian in (Fu et al., 2020; Zhong et al., 2017b), we develop new techniques in the following aspects.

Original: First, a direct extension of the matrix concentration inequalities in these works leads to a sample complexity bound of O ( d 3 ) , while we develop new concentration bounds to tighten it to O
Revised: First, a direct extension of the matrix concentration inequalities in these works leads to a sample complexity bound of O ( d 3 ) , while we develop new concentration bounds to tighten it to O

Original: ( d log 2 d ) .
Revised: ( d log 2 d ) .

Original: Second, the existing analysis to bound the Hessian of the population risk function does not extend to the Gaussian mixture model.
Revised: Second, the existing analysis to bound the Hessian of the population risk function does not extend to the Gaussian mixture model.

Original: We develop new tools that also apply to other activation functions like tanh or erf.
Revised: We develop new tools that also apply to other activation functions like tanh or erf.

Original: Third, we design new tensors for the initialization, and the proof about the tensor initialization is revised accordingly.
Revised: Third, we design new tensors for the initialization, and the proof about the tensor initialization is revised accordingly.

Original: The above results assume the parameters of the Gaussian mixture are known.
Revised: The above results assume the parameters of the Gaussian mixture are known.

Original: In practice, they can be estimated by the EM algorithm (Redner & Walker, 1984) and the moment-based method (Hsu & Kakade, 2013).
Revised: In practice, they can be estimated by the EM algorithm (Redner & Walker, 1984) and the moment-based method (Hsu & Kakade, 2013).

Original: The EM algorithm returns model parameters within Euclidean distance O (( dn ) 12 ) when the number of mixture components L is known.
Revised: The EM algorithm returns model parameters within Euclidean distance O (( dn ) 12 ) when the number of mixture components L is known.

Original: When L is unknown, one usually overspeciﬁes an estimate ¯ L > L , then the estimation error by the EM algorithm scales as O (( dn ) 14 ) .
Revised: When L is unknown, one usually overspeciﬁes an estimate ¯ L > L , then the estimation error by the EM algorithm scales as O (( dn ) 14 ) .

Original: Please refer to (Ho & Nguyen, 2016;
Revised: Please refer to (Ho & Nguyen, 2016;

Original: Ho et al., 2005; Dwivedi et al., 2020a;b) for details.
Revised: Ho et al., 2020; Dwivedi et al., 2020a;b) for details.

Original:  N UMERICAL E XPERIMENTS
Revised:  N UMERICAL E XPERIMENTS

Original: We verify Theorem 1 through numerical experiments.
Revised: We verify Theorem 1 through numerical experiments.

Original: We generate a ground-truth W ∗ ∈ R d × K from the Gaussian distribution.
Revised: We generate a ground-truth W ∗ ∈ R d × K from the Gaussian distribution.

Original: The training samples { x i , y i } ni =1 are generated using (6) and (2).
Revised: The training samples { x i , y i } ni =1 are generated using (6) and (2).

Original: The maximum number of iterations of Algorithm 1 is set as 12000 .
Revised: The maximum number of iterations of Algorithm 1 is set as 12000 .

Original: 5. T ENSOR INITIALIZATION
Revised: 5. T ENSOR INITIALIZATION

Original: Fig.
Revised: Fig.

Original: 1 shows the accuracy of the returned model by Algorithm 1.
Revised: 1 shows the accuracy of the returned model by Algorithm 1.

Original: Here d = 5 , K = 2 , λ 1 = λ 2 = 0 .
Revised: Here d = 5 , K = 2 , λ 1 = λ 2 = 0 .

Original: 5 , µ 1 = − 1 and µ 2 = 0 .
Revised: 5 , µ 1 = − 1 and µ 2 = 0 .

Original: We compare the tensor initialization with a random initialization in a local region { W ∈
Revised: We compare the tensor initialization with a random initialization in a local region { W ∈

Original: [Equation]
Revised: [Equation]

Original: F ≤ (cid:15) } .
Revised: F ≤ (cid:15) } .

Original: Tensor initialization in Subroutine 1 returns an initial point close to W ∗ with a relative error of 0 . 61 .
Revised: Tensor initialization in Subroutine 1 returns an initial point close to W ∗ with a relative error of 0 . 61 .

Original: If the random initialization is also close to W ∗ , e.g., (cid:15) = 0 .
Revised: If the random initialization is also close to W ∗ , e.g., (cid:15) = 0 .

Original: 1 , then the gradient descent algorithm converges to a critical point from both initializations, and the linear convergence rate is the same.
Revised: 1 , then the gradient descent algorithm converges to a critical point from both initializations, and the linear convergence rate is the same.

Original: If the random initialization is far away, e.g., (cid:15) = 1 . 5 , the algorithm does not converge.
Revised: If the random initialization is far away, e.g., (cid:15) = 1 . 5 , the algorithm does not converge.

Original: On a MacBook Pro with Intel(R) Core(TM) i5-7360U CPU at 2.30GHz and MATLAB 2017a, it takes 0.55 second to compute the tensor initialization.
Revised: On a MacBook Pro with Intel(R) Core(TM) i5-7360U CPU at 2.30GHz and MATLAB 2017a, it takes 0.55 second to compute the tensor initialization.

Original: We consider a random initialization with (cid:15) = 0 .
Revised: We consider a random initialization with (cid:15) = 0 .

Original: 1 in the following experiments to simplify the computation.
Revised: 1 in the following experiments to simplify the computation.

Original: 5. S AMPLE COMPLEXITY
Revised: 5. S AMPLE COMPLEXITY

Original: Consider the case that K = 3 , L = 2 , λ 1 = λ 2 = 12 .
Revised: Consider the case that K = 3 , L = 2 , λ 1 = λ 2 = 12 .

Original: Let µ 1 be an all one vector in R d and let µ 2 = − µ 1 .
Revised: Let µ 1 be an all one vector in R d and let µ 2 = − µ 1 .

Original: Let σ 1 =
Revised: Let σ 1 =

Original: σ 2 = 1 .
Revised: σ 2 = 1 .

Original: We vary d and evaluate the sample complexity bound in (11) with respect to d .
Revised: We vary d and evaluate the sample complexity bound in (11) with respect to d .

Original: We randomly initialize M times and let (cid:99) W ( m ) n denote the output of Algorithm 1 in the m th trail.
Revised: We randomly initialize M times and let (cid:99) W ( m ) n denote the output of Algorithm 1 in the m th trail.

Original: Let ¯ W n denote the mean values of all (cid:99) W ( m ) n , and let d W = (cid:113)(cid:80)
Revised: Let ¯ W n denote the mean values of all (cid:99) W ( m ) n , and let d W = (cid:113)(cid:80)

Original: Mm =1 || (cid:98)
Revised: Mm =1 || (cid:98)

Original: w mn
Revised: w mn

Original: − ¯ W n || 2 /M denote the variance.
Revised: − ¯ W n || 2 /M denote the variance.

Original: An experiment is successful if d W ≤ 10 − 4 and fails otherwise.
Revised: An experiment is successful if d W ≤ 10 − 4 and fails otherwise.

Original: M is set as 20 .
Revised: M is set as 20 .

Original: We vary d and the number of samples n .
Revised: We vary d and the number of samples n .

Original: For each pair of d and n , 20 independent sets of W ∗ and the corresponding training samples are generated.
Revised: For each pair of d and n , 20 independent sets of W ∗ and the corresponding training samples are generated.

Original: Fig.
Revised: Fig.

Original: 2 shows the success rate of these independent experiments.
Revised: 2 shows the success rate of these independent experiments.

Original: A black block means that all the experiments fail.
Revised: A black block means that all the experiments fail.

Original: A white block means that they all succeed.
Revised: A white block means that they all succeed.

Original: The sample complexity is indeed almost linear in d , as predicted by (11).
Revised: The sample complexity is indeed almost linear in d , as predicted by (11).

Original: Moreover, the coefﬁcient n/d can be large depending on the problem setup.
Revised: Moreover, the coefﬁcient n/d can be large depending on the problem setup.

Original: The sample complexity against the feature dimension d
Revised: The sample complexity against the feature dimension d

Original: We then ﬁx d = 5 and study the impact on the sample complexity when the mean and variance in the Gaussian mixture model change.
Revised: We then ﬁx d = 5 and study the impact on the sample complexity when the mean and variance in the Gaussian mixture model change.

Original: In Fig. 3.(a), we ﬁx σ 1 = σ 2 = 1 and let µ 1 = µ · 1 , µ 2 = − 1 .
Revised: In Fig. 3.(a), we ﬁx σ 1 = σ 2 = 1 and let µ 1 = µ · 1 , µ 2 = − 1 .

Original: µ varies from 0 to 7 . 5 .
Revised: µ varies from 0 to 7 . 5 .

Original: Fig. 3.(a) shows that when the mean increases, the sample complexity increases.
Revised: Fig. 3.(a) shows that when the mean increases, the sample complexity increases.

Original: This coincides with our theoretical analyses in Section 4.
Revised: This coincides with our theoretical analyses in Section 4.

Original: In Fig. 3.(b), we ﬁx µ 1 = 1 , µ 2 = − 1 , and let σ 1 = σ and σ 2 = 1 .
Revised: In Fig. 3.(b), we ﬁx µ 1 = 1 , µ 2 = − 1 , and let σ 1 = σ and σ 2 = 1 .

Original: σ varies from 10 − 1 . 4 to 10 1 . 4 .
Revised: σ varies from 10 − 1 . 4 to 10 1 . 4 .

Original: The sample complexity increases both when σ increases and when σ approaches zero.
Revised: The sample complexity increases both when σ increases and when σ approaches zero.

Original: The results match our theoretical prediction in Section 4.
Revised: The results match our theoretical prediction in Section 4.

Original: [Figure]
Revised: [Figure]

Original: 5. C ONVERGENCE ANALYSIS
Revised: 5. C ONVERGENCE ANALYSIS

Original: We next study the convergence rate of Algorithm 1.
Revised: We next study the convergence rate of Algorithm 1.

Original: d is ﬁxed as 5.
Revised: d is ﬁxed as 5.

Original: Fig.
Revised: Fig.

Original: 4.(a) shows the impact of the mean of the Gaussian mixture model on the convergence rate.
Revised: 4.(a) shows the impact of the mean of the Gaussian mixture model on the convergence rate.

Original: We set λ 1 = λ 2 = 0 .
Revised: We set λ 1 = λ 2 = 0 .

Original: 5 , µ 1 = µ · 1 , µ 2 = − 1 , and σ 1 = σ 2 = 1 .
Revised: 5 , µ 1 = µ · 1 , µ 2 = − 1 , and σ 1 = σ 2 = 1 .

Original: The sample complexity n is set to 10000 .
Revised: The sample complexity n is set to 10000 .

Original: One can see that Algorithm 1 always converges linearly when µ changes.
Revised: One can see that Algorithm 1 always converges linearly when µ changes.

Original: Moreover, as µ increases, Algorithm 1 converges slower, as predicted by our theoretical analyses in Section 4.
Revised: Moreover, as µ increases, Algorithm 1 converges slower, as predicted by our theoretical analyses in Section 4.

Original: In Fig. 4.(b) shows the impact of the variance of the Gaussian mixture model.
Revised: In Fig. 4.(b) shows the impact of the variance of the Gaussian mixture model.

Original: We set λ 1 = λ 2 = 0 .
Revised: We set λ 1 = λ 2 = 0 .

Original: 5 , µ 1 = 1 , µ 2 = − 1 , σ 1 = σ 2 = σ .
Revised: 5 , µ 1 = 1 , µ 2 = − 1 , σ 1 = σ 2 = σ .

Original: The sample complexity n is set to 50000 .
Revised: The sample complexity n is set to 50000 .

Original: Among different σ we test, Algorithm 1 converges fastest when σ = 1 .
Revised: Among different σ we test, Algorithm 1 converges fastest when σ = 1 .

Original: The convergence rate slows down when σ increases to 2 or when σ decreases to 0.5.
Revised: The convergence rate slows down when σ increases to 2 or when σ decreases to 0.5.

Original: The result is consistent with our theoretical results in Section 4.
Revised: The result is consistent with our theoretical results in Section 4.

Original: We then verify the convergence rate in (12), which shows that v = 1 − Θ( K − 2 ) .
Revised: We then verify the convergence rate in (12), which shows that v = 1 − Θ( K − 2 ) .

Original: We set λ 1 = λ 2 = 0 .
Revised: We set λ 1 = λ 2 = 0 .

Original: 5 , µ 1 = 1 , µ 2 = − 1 , σ 1 = σ 2 = 1 .
Revised: 5 , µ 1 = 1 , µ 2 = − 1 , σ 1 = σ 2 = 1 .

Original: K ranges from 2 to 8 .
Revised: K ranges from 2 to 8 .

Original: One can see from Fig.
Revised: One can see from Fig.

Original: 5 that, as predicted, the convergence rate is almost linear in 1 /K 2 .
Revised: 5 that, as predicted, the convergence rate is almost linear in 1 /K 2 .

Original: [Figure]
Revised: [Figure]

Original: We then evaluate the distance between (cid:99) W n returned by Algorithm 1 and W ∗ , measured by || (cid:99) W n −
Revised: We then evaluate the distance between (cid:99) W n returned by Algorithm 1 and W ∗ , measured by || (cid:99) W n −

Original: W ∗ || F .
Revised: W ∗ || F .

Original: d is 5.
Revised: d is 5.

Original: n ranges from 2 × 10 3 to 6 × 10 4 .
Revised: n ranges from 2 × 10 3 to 6 × 10 4 .

Original: σ 1 = σ 2 = 3 , µ 1 = 1 , µ 2 = − 1 .
Revised: σ 1 = σ 2 = 3 , µ 1 = 1 , µ 2 = − 1 .

Original: Each point in
Revised: Each point in

Original: Fig. 6 is averaged over 100 independent experiments of different W ∗ and the corresponding training set.
Revised: Fig. 6 is averaged over 100 independent experiments of different W ∗ and the corresponding training set.

Original: (cid:107) W ∗ (cid:107) F is normalized to 1.
Revised: (cid:107) W ∗ (cid:107) F is normalized to 1.

Original: The error is indeed linear in (cid:112) log( n ) /n , as predicted by (12).
Revised: The error is indeed linear in (cid:112) log( n ) /n , as predicted by (12).

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: This paper analyzes the theoretical performance guarantee of learning one-hidden-layer neural networks for binary classiﬁcation when the input follows the Gaussian mixture model.
Revised: This paper analyzes the theoretical performance guarantee of learning one-hidden-layer neural networks for binary classiﬁcation when the input follows the Gaussian mixture model.

Original: We develop an algorithm that converges linearly to a model that has a diminishing difference from the ground-truth model that has guaranteed generalizability.
Revised: We develop an algorithm that converges linearly to a model that has a diminishing difference from the ground-truth model that has guaranteed generalizability.

Original: We also provide the ﬁrst explicit characterization of the impact of the input distribution on the sample complexity and convergence rate.
Revised: We also provide the ﬁrst explicit characterization of the impact of the input distribution on the sample complexity and convergence rate.

Original: Future works include the analysis of multiple-hidden-layer neural networks and multi-class classiﬁcation.
Revised: Future works include the analysis of multiple-hidden-layer neural networks and multi-class classiﬁcation.

Original: Because of the concatenation of nonlinear activation functions, the analysis of the landscape of the empirical risk and the design of a proper initialization is more challenging and requires the development of new tools.
Revised: Because of the concatenation of nonlinear activation functions, the analysis of the landscape of the empirical risk and the design of a proper initialization is more challenging and requires the development of new tools.

Original: A P RELIMINARIES
Revised: A P RELIMINARIES

Original: In this section, we introduce some deﬁnitions and properties that will be used in proving the main results.
Revised: In this section, we introduce some deﬁnitions and properties that will be used in proving the main results.

Original: First we deﬁne the following three quantities.
Revised: First we deﬁne the following three quantities.

Original: ρ ( µ , σ ) is motivated by the ρ parameter for the standard Gaussian distribution in (Zhong et al., 2017b), and we generalize it to a Gaussian with an arbitrary mean and variance.
Revised: ρ ( µ , σ ) is motivated by the ρ parameter for the standard Gaussian distribution in (Zhong et al., 2017b), and we generalize it to a Gaussian with an arbitrary mean and variance.

Original: We deﬁne the new quantities Γ( λ , M , σ , W ∗ ) and D m ( λ , M , σ ) for the Gaussian mixture model.
Revised: We deﬁne the new quantities Γ( λ , M , σ , W ∗ ) and D m ( λ , M , σ ) for the Gaussian mixture model.

Original: Deﬁnition 2 ( ρ -function).
Revised: Deﬁnition 2 ( ρ -function).

Original: Let z ∼ N ( u ,
Revised: Let z ∼ N ( u ,

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: d .
Revised: d .

Original: Deﬁne α q
Revised: Deﬁne α q

Original: ( i, u ,
Revised: ( i, u ,

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: E
Revised: E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [ φ
Revised: [ φ

Original: [Equation]
Revised: [Equation]

Original: ( σ · z i ) z qi ] and β q ( i, u , σ ) =
Revised: ( σ · z i ) z qi ] and β q ( i, u , σ ) =

Original: E z i ∼N ( u i , 1) [ φ (cid:48) 2 ( σ · z i )
Revised: E z i ∼N ( u i , 1) [ φ (cid:48) 2 ( σ · z i )

Original: z qi ] , ∀ q ∈ { 0 , 1 , 2 } , where z i and u i is the i -th entry of z and u , respectively.
Revised: z qi ] , ∀ q ∈ { 0 , 1 , 2 } , where z i and u i is the i -th entry of z and u , respectively.

Original: Deﬁne ρ ( u , σ ) as
Revised: Deﬁne ρ ( u , σ ) as

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 3 ( Γ -function).
Revised: Deﬁnition 3 ( Γ -function).

Original: With (6), (14) and κ , η deﬁned in Section 2, we deﬁne
Revised: With (6), (14) and κ , η deﬁned in Section 2, we deﬁne

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 4 (D-function).
Revised: Deﬁnition 4 (D-function).

Original: Given the Gaussian Mixture Model in (6) and any positive integer m , deﬁne D m ( λ , M , σ ) as
Revised: Given the Gaussian Mixture Model in (6) and any positive integer m , deﬁne D m ( λ , M , σ ) as

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where λ = ( λ 1 , · · · , λ L ) ∈ R L , M = ( µ 1 , · · · , µ L ) ∈ R d × L and σ = ( σ 1 , · · · , σ L ) ∈ R L .
Revised: where λ = ( λ 1 , · · · , λ L ) ∈ R L , M = ( µ 1 , · · · , µ L ) ∈ R d × L and σ = ( σ 1 , · · · , σ L ) ∈ R L .

Original: ρ -function is deﬁned to compute the lower bound of the Hessian of the population risk with Gaussian input.
Revised: ρ -function is deﬁned to compute the lower bound of the Hessian of the population risk with Gaussian input.

Original: Γ function is the weighted sum of ρ -function under mixture Gaussian distribution.
Revised: Γ function is the weighted sum of ρ -function under mixture Gaussian distribution.

Original: This function is positive and upper bounded by a small value.
Revised: This function is positive and upper bounded by a small value.

Original: It is increasing when | µ l ( i ) | increases.
Revised: It is increasing when | µ l ( i ) | increases.

Original: When σ l increases, Γ increases ﬁrst and then decreases.
Revised: When σ l increases, Γ increases ﬁrst and then decreases.

Original: Γ goes to zero if all (cid:107) µ l (cid:107) ∞ or all σ l goes to inﬁnity.
Revised: Γ goes to zero if all (cid:107) µ l (cid:107) ∞ or all σ l goes to inﬁnity.

Original: D -function is a normalized parameter for the means and variances.
Revised: D -function is a normalized parameter for the means and variances.

Original: It is lower bounded by 1.
Revised: It is lower bounded by 1.

Original: D -function is an increasing function of (cid:107) µ l (cid:107) ∞ and a decreasing function of σ l .
Revised: D -function is an increasing function of (cid:107) µ l (cid:107) ∞ and a decreasing function of σ l .

Original: Property 1 ρ ( u , σ ) in Deﬁnition 2 satisﬁes the following properties,
Revised: Property 1 ρ ( u , σ ) in Deﬁnition 2 satisﬁes the following properties,

Original: ρ ( u , σ ) > 0 for any u ∈ R d and σ (cid:54) = 0 .
Revised: ρ ( u , σ ) > 0 for any u ∈ R d and σ (cid:54) = 0 .

Original: ρ ( u , σ ) converges to a positive value function of σ as u
Revised: ρ ( u , σ ) converges to a positive value function of σ as u

Original: i goes to 0, i.e. lim u i → 0 ρ ( u , σ ) :
Revised: i goes to 0, i.e. lim u i → 0 ρ ( u , σ ) :

Original: =
Revised: =

Original: C m ( σ ) .
Revised: C m ( σ ) .

Original: When all u i (cid:54) = 

Revised: When all u i (cid:54) =

Original: ( i ∈ [ d ]) , ρ ( u σ , σ ) converges to a positive value function of u as σ goes to 0,
Revised: ( i ∈ [ d ]) , ρ ( u σ , σ ) converges to a positive value function of u as σ goes to 0,

Original: i
Revised: i

Original: e. lim σ → 0 ρ ( u σ , σ ) :
Revised: e. lim σ → 0 ρ ( u σ , σ ) :

Original: = C s ( u ) .
Revised: = C s ( u ) .

Original: When u i = 0 for some i ∈
Revised: When u i = 0 for some i ∈

Original: [ d ] , lim σ → 0 ρ ( u σ , σ ) = 0 .
Revised: [ d ] , lim σ → 0 ρ ( u σ , σ ) = 0 .

Original: When everything else except | u
Revised: When everything else except | u

Original: i | is ﬁxed, ρ (
Revised: i | is ﬁxed, ρ (

Original: [Equation]
Revised: [Equation]

Original: W ∗ ) , σδ K ( W ∗ )) is lower bounded by a positive value function, L m ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) , which is monotonically decreasing to 0 as | u i | increases.
Revised: W ∗ ) , σδ K ( W ∗ )) is lower bounded by a positive value function, L m ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) , which is monotonically decreasing to 0 as | u i | increases.

Original: When everything else except σ is ﬁxed, ρ ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) is lower bounded by a positive value function, L s ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) , which satisﬁes the following conditions: (a) there exists ζ s (cid:48) > 0 , such that σ − 1 L s ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) is an increasing function of σ when σ ∈ (0 , ζ s (cid:48) ) ; (b) there exists ζ s > 0 such that L s ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) is an decreasing function of σ when σ ∈ ( ζ s , + ∞ ) .
Revised: When everything else except σ is ﬁxed, ρ ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) is lower bounded by a positive value function, L s ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) , which satisﬁes the following conditions: (a) there exists ζ s (cid:48) > 0 , such that σ − 1 L s ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) is an increasing function of σ when σ ∈ (0 , ζ s (cid:48) ) ; (b) there exists ζ s > 0 such that L s ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) is an decreasing function of σ when σ ∈ ( ζ s , + ∞ ) .

Original: Proof: (1) From the Cauchy Schwarz’s inequality, we have
Revised: Proof: (1) From the Cauchy Schwarz’s inequality, we have

Original: [Equation]
Revised: [Equation]

Original: The equalities of the (17) and (18) hold if and only if φ (cid:48) is a constant function.
Revised: The equalities of the (17) and (18) hold if and only if φ (cid:48) is a constant function.

Original: Since that φ is the sigmoid function, the equalities of (17) and (18) cannot hold.
Revised: Since that φ is the sigmoid function, the equalities of (17) and (18) cannot hold.

Original: By the deﬁnition of ρ ( u , σ ) in Deﬁnition 2, we have β 

Revised: By the deﬁnition of ρ ( u , σ ) in Deﬁnition 2, we have β

Original: ( i, u , σ ) − α 20 ( i, u , σ ) > 0 and β 2 ( i, u , σ ) − α 22 ( i, u ,σ ) ui + > 0 .
Revised: ( i, u , σ ) − α 20 ( i, u , σ ) > 0 and β 2 ( i, u , σ ) − α 22 ( i, u ,σ ) ui + > 0 .

Original: Therefore,
Revised: Therefore,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Combining (20) and (21)
Revised: Combining (20) and (21)

Original: [Equation]
Revised: [Equation]

Original: we can derive that ρ ( u , σ ) converges to a positive value function of σ as u i goes to 0, i.e. lim u
Revised: we can derive that ρ ( u , σ ) converges to a positive value function of σ as u i goes to 0, i.e. lim u

Original: → 0 ρ ( u , σ ) :
Revised: → 0 ρ ( u , σ ) :

Original: = C m ( σ )
Revised: = C m ( σ )

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: The third step of (22) is by the fact that the Gaussian distribution goes to a Dirac delta function when σ goes to 0 .
Revised: The third step of (22) is by the fact that the Gaussian distribution goes to a Dirac delta function when σ goes to 0 .

Original: Then the integral will take the value when x i = 1 .
Revised: Then the integral will take the value when x i = 1 .

Original: Similarly, we can obtain the following
Revised: Similarly, we can obtain the following

Original: [Equation]
Revised: [Equation]

Original: Therefore, by L’Hopital’s rule and (23), (24), we have
Revised: Therefore, by L’Hopital’s rule and (23), (24), we have

Original: [Equation]
Revised: [Equation]

Original: Combining (25) and (22), we can derive that ρ ( u σ , σ ) converges to a positive value function of u as σ goes to 0, i.e. lim σ → 0 ρ ( u σ , σ ) :
Revised: Combining (25) and (22), we can derive that ρ ( u σ , σ ) converges to a positive value function of u as σ goes to 0, i.e. lim σ → 0 ρ ( u σ , σ ) :

Original: = C s ( u ) .
Revised: = C s ( u ) .

Original: When u i = 0 for some i ∈ [ d ] , lim σ → 0 ( u 2 i σ 2 + 1)( β 0 ( j, u σ , σ ) − α 2 ( j, u σ , σ ))
Revised: When u i = 0 for some i ∈ [ d ] , lim σ → 0 ( u 2 i σ 2 + 1)( β 0 ( j, u σ , σ ) − α 2 ( j, u σ , σ ))

Original: = 

Revised: =

Original: by (23).
Revised: by (23).

Original: Then from the Deﬁnition 2, we have lim σ → 0 ρ ( u σ , σ ) = 0 .
Revised: Then from the Deﬁnition 2, we have lim σ → 0 ρ ( u σ , σ ) = 0 .

Original: (4) We show the statement by contradiction.
Revised: (4) We show the statement by contradiction.

Original: Suppose that for any positive value function, h ( u i ) , which is monotonically decreasing to 0 as | u i | increases, there exists a u ∗ i ∈
Revised: Suppose that for any positive value function, h ( u i ) , which is monotonically decreasing to 0 as | u i | increases, there exists a u ∗ i ∈

Original: R such that h ( u i ) ≥ ρ ( W ∗(cid:62) u σδ K
Revised: R such that h ( u i ) ≥ ρ ( W ∗(cid:62) u σδ K

Original: ( W
Revised: ( W

Original: [Equation]
Revised: [Equation]

Original: ) , σδ
Revised: ) , σδ

Original: [Equation]
Revised: [Equation]

Original: ( W
Revised: ( W

Original: [Equation]
Revised: [Equation]

Original: )) (cid:12)(cid:12)(cid:12)
Revised: )) (cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: ∗ .
Revised: ∗ .

Original: Then we can derive that
Revised: Then we can derive that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: (cid:12) (cid:12)
Revised: (cid:12) (cid:12)

Original: [Equation]
Revised: [Equation]

Original: = 0 .
Revised: = 0 .

Original: Since that ρ ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) is continuous, we can obtain that ρ ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ ))
Revised: Since that ρ ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) is continuous, we can obtain that ρ ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ ))

Original: (cid:12)(cid:12)(cid:12) u i =
Revised: (cid:12)(cid:12)(cid:12) u i =

Original: u
Revised: u

Original: i ∗ = 0 , which contradicts to the conclusion in
Revised: i ∗ = 0 , which contradicts to the conclusion in

Original: Property 1.1.
Revised: Property 1.1.

Original: The condition (b) can be easily proved as (4).
Revised: The condition (b) can be easily proved as (4).

Original: Therefore, we only need to show the condition (a).
Revised: Therefore, we only need to show the condition (a).

Original: When ( W ∗(cid:62) u ) i (cid:54) = 

Revised: When ( W ∗(cid:62) u ) i (cid:54) =

Original: for all i ∈ [ K ] , lim σ → 0 ρ ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) =
Revised: for all i ∈ [ K ] , lim σ → 0 ρ ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) =

Original: C s ( u ) > 0 .
Revised: C s ( u ) > 0 .

Original: Therefore, there exists ζ s > 0 , such that when 0 < σ < ζ s , ρ ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) >
Revised: Therefore, there exists ζ s > 0 , such that when 0 < σ < ζ s , ρ ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) >

Original: C s ( W ∗(cid:62) u ).
Revised: C s ( W ∗(cid:62) u ).

Original: Then we can deﬁne L s ( W ∗(cid:62) u
Revised: Then we can deﬁne L s ( W ∗(cid:62) u

Original: [Equation]
Revised: [Equation]

Original: , σδ K ( W ∗
Revised: , σδ K ( W ∗

Original: [Equation]
Revised: [Equation]

Original: = C s (
Revised: = C s (

Original: [Equation]
Revised: [Equation]

Original: 2 ζ s σ 2 such that σ − 1 L s ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) is an increasing function of σ below ρ ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) .
Revised: 2 ζ s σ 2 such that σ − 1 L s ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) is an increasing function of σ below ρ ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) .

Original: When ( W ∗(cid:62) u ) i = 0 for some i ∈
Revised: When ( W ∗(cid:62) u ) i = 0 for some i ∈

Original: [ K ] , then lim
Revised: [ K ] , then lim

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: = 0 .
Revised: = 0 .

Original: We can derive
Revised: We can derive

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The last step of (26) is because if the limit is negative, then ρ ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) will be negative in a small neighborhood around σ = 0 , which contradicts to the fact that ρ ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) > 0 .
Revised: The last step of (26) is because if the limit is negative, then ρ ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) will be negative in a small neighborhood around σ = 0 , which contradicts to the fact that ρ ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) > 0 .

Original: If the limit in (26) is 0, then lim σ → 0 ∂∂σ ρ ( W ∗(cid:62) u σδK ( W ∗ ) ,σδ K ( W ∗ )) σ > 0 otherwise there will be a small neighborhood around σ = 0 in which ρ ( W ∗(cid:62) u σδK ( W ∗ ) ,σδ K ( W ∗ )) σ < 0 .
Revised: If the limit in (26) is 0, then lim σ → 0 ∂∂σ ρ ( W ∗(cid:62) u σδK ( W ∗ ) ,σδ K ( W ∗ )) σ > 0 otherwise there will be a small neighborhood around σ = 0 in which ρ ( W ∗(cid:62) u σδK ( W ∗ ) ,σδ K ( W ∗ )) σ < 0 .

Original: In this case we only need to let σ −
Revised: In this case we only need to let σ −

Original: [Equation]
Revised: [Equation]

Original: s ( W ∗(cid:62)
Revised: s ( W ∗(cid:62)

Original: [Equation]
Revised: [Equation]

Original: If the limit in (26) is positive, we can ﬁnd a positive lower bound of ρ (
Revised: If the limit in (26) is positive, we can ﬁnd a positive lower bound of ρ (

Original: [Equation]
Revised: [Equation]

Original: σ in a small neighborhood around σ = 0 and an increasing function of σ , σ − 1 L ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) can be deﬁned to be less than this positive lower bound.
Revised: σ in a small neighborhood around σ = 0 and an increasing function of σ , σ − 1 L ( W ∗(cid:62) u σδ K ( W ∗ ) , σδ K ( W ∗ )) can be deﬁned to be less than this positive lower bound.

Original: In conclusion, the condition (a) is proved.
Revised: In conclusion, the condition (a) is proved.

Original: Property 2 With the notation in (6), if a function f ( x ) is an even function, then
Revised: Property 2 With the notation in (6), if a function f ( x ) is an even function, then

Original: [Equation]
Revised: [Equation]

Original: Proof:
Revised: Proof:

Original: Denote
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Property 3 Under Gaussian Mixture Model x ∼ (cid:80) Ll
Revised: Property 3 Under Gaussian Mixture Model x ∼ (cid:80) Ll

Original: =1 λ l N ( µ l , σ 2 l I d ) , we have the following upper bound.
Revised: =1 λ l N ( µ l , σ 2 l I d ) , we have the following upper bound.

Original: [Equation]
Revised: [Equation]

Original: Proof:
Revised: Proof:

Original: The main idea is to ﬁnd an upper bound with symmetric distribution assumption ﬁrst, and then apply
Revised: The main idea is to ﬁnd an upper bound with symmetric distribution assumption ﬁrst, and then apply

Original: Property 2 to extend the conclusion to the general case.
Revised: Property 2 to extend the conclusion to the general case.

Original: ( a ) If the Mixed-Gaussian distribution is symmetric and L = 2 ,
Revised: ( a ) If the Mixed-Gaussian distribution is symmetric and L = 2 ,

Original: i
Revised: i

Original: e. x ∼ 12 (cid:16) N ( µ , σ
Revised: e. x ∼ 12 (cid:16) N ( µ , σ

Original: [Equation]
Revised: [Equation]

Original: d ) + N ( − µ , σ 2 I d ) (cid:17) , then
Revised: d ) + N ( − µ , σ 2 I d ) (cid:17) , then

Original: we
Revised: we

Original: ﬁrst
Revised: ﬁrst

Original: need to analyse the distribution of u (cid:62) x by computing the moment generating function
Revised: need to analyse the distribution of u (cid:62) x by computing the moment generating function

Original: [Equation]
Revised: [Equation]

Original: which is the Moment Generating Function of (cid:80)
Revised: which is the Moment Generating Function of (cid:80)

Original: [Equation]
Revised: [Equation]

Original: N ( µ (cid:48)
Revised: N ( µ (cid:48)

Original: i , σ (cid:48) 2 ) .
Revised: i , σ (cid:48) 2 ) .

Original: The last step of (32) is by expanding the multiplication of d terms.
Revised: The last step of (32) is by expanding the multiplication of d terms.

Original: Speciﬁcally, let { s i } 2 d i =1 denote all 2 d vectors in R d taking values from 0 and 1 .
Revised: Speciﬁcally, let { s i } 2 d i =1 denote all 2 d vectors in R d taking values from 0 and 1 .

Original: Let s ik ( k ∈ [ d ]) denote the k -th entry of s i .
Revised: Let s ik ( k ∈ [ d ]) denote the k -th entry of s i .

Original: We deﬁne µ (cid:48)
Revised: We deﬁne µ (cid:48)

Original: i =
Revised: i =

Original: (cid:80) dk
Revised: (cid:80) dk

Original: =1 ( − 1) s ik u k µ k ∈ R for i ∈ [2 d ] , and σ (cid:48)
Revised: =1 ( − 1) s ik u k µ k ∈ R for i ∈ [2 d ] , and σ (cid:48)

Original: =
Revised: =

Original: σ || u || ∈ R , where u k and µ k are the k -th entry of the vector u and µ , respectively.
Revised: σ || u || ∈ R , where u k and µ k are the k -th entry of the vector u and µ , respectively.

Original: Then we can derive the ﬁrst few steps of E [( u (cid:62) x ) 2 t ]
Revised: Then we can derive the ﬁrst few steps of E [( u (cid:62) x ) 2 t ]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The ﬁrst step is by the distribution of u (cid:62) x we obtain from (32).
Revised: The ﬁrst step is by the distribution of u (cid:62) x we obtain from (32).

Original: The third step follows from the binomial expansion.
Revised: The third step follows from the binomial expansion.

Original: The forth step results from the calculation of high-order moment of Gaussian distribution.
Revised: The forth step results from the calculation of high-order moment of Gaussian distribution.

Original: The second to last step is derived from the inverse of binomial expansion.
Revised: The second to last step is derived from the inverse of binomial expansion.

Original: The last step is due to the substitution of summation.
Revised: The last step is due to the substitution of summation.

Original: To compute the inner summation in the last step of (33), we have
Revised: To compute the inner summation in the last step of (33), we have

Original: [Equation]
Revised: [Equation]

Original: all the p i are even
Revised: all the p i are even

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Firstly we explain the third step.
Revised: Firstly we explain the third step.

Original: For any odd p (cid:96) , there is a term a 

Revised: For any odd p (cid:96) , there is a term a

Original: = ( u (cid:96) ( − 1) s i(cid:96) µ (cid:96) )
Revised: = ( u (cid:96) ( − 1) s i(cid:96) µ (cid:96) )

Original: p (cid:96) · (cid:81) k (cid:54) =
Revised: p (cid:96) · (cid:81) k (cid:54) =

Original: (cid:96) ( u k ( − 1) s ik µ k ) p k
Revised: (cid:96) ( u k ( − 1) s ik µ k ) p k

Original: among the expansion of µ (cid:48) 2 t i , whose corresponding vector s i is ( s ii , · · · , s i(cid:96) , · · · , s id ) .
Revised: among the expansion of µ (cid:48) 2 t i , whose corresponding vector s i is ( s ii , · · · , s i(cid:96) , · · · , s id ) .

Original: We can ﬁnd a µ (cid:48)
Revised: We can ﬁnd a µ (cid:48)

Original: j such that its corresponding vector is ( s i 1 , · · · , 1 − s i(cid:96) , · · · , s id ) , which is only different from the tuple of µ (cid:48)
Revised: j such that its corresponding vector is ( s i 1 , · · · , 1 − s i(cid:96) , · · · , s id ) , which is only different from the tuple of µ (cid:48)

Original: i in the (cid:96) -th entry.
Revised: i in the (cid:96) -th entry.

Original: Therefore, in the expansion of µ (cid:48) 2 t j , there exists a term a (cid:48) 

Revised: Therefore, in the expansion of µ (cid:48) 2 t j , there exists a term a (cid:48)

Original: = ( u (cid:96) ( − 1) 1 − s i(cid:96) µ (cid:96) )
Revised: = ( u (cid:96) ( − 1) 1 − s i(cid:96) µ (cid:96) )

Original: p (cid:96) · (cid:81) k (cid:54) =
Revised: p (cid:96) · (cid:81) k (cid:54) =

Original: (cid:96) ( u k ( − 1) s ik µ k ) p
Revised: (cid:96) ( u k ( − 1) s ik µ k ) p

Original: k that can be cancelled out by a 0 .
Revised: k that can be cancelled out by a 0 .

Original: Therefore, there will be no odd power terms left.
Revised: Therefore, there will be no odd power terms left.

Original: The third to last step of (34) is by the inverse binomial expansion.
Revised: The third to last step of (34) is by the inverse binomial expansion.

Original: The second to last step is by the inequality (cid:80) N i =1 a i b i ≤ max { b i } · (cid:80) N i =
Revised: The second to last step is by the inequality (cid:80) N i =1 a i b i ≤ max { b i } · (cid:80) N i =

Original: 1 a i , where a i and b i are positive.
Revised: 1 a i , where a i and b i are positive.

Original: The last step is because
Revised: The last step is because

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In the ﬁrst equality of (35), p d 1 , ..., p d m denote all the positive p i .
Revised: In the ﬁrst equality of (35), p d 1 , ..., p d m denote all the positive p i .

Original: Thus, we have (cid:80) mi =1 p d i = 2 t where p d i ≥ 2 .
Revised: Thus, we have (cid:80) mi =1 p d i = 2 t where p d i ≥ 2 .

Original: Therefore, m ≤ 2 t 2 = t which is used in the second inequality.
Revised: Therefore, m ≤ 2 t 2 = t which is used in the second inequality.

Original: Therefore, combining
Revised: Therefore, combining

Original: (34), we can continue the derivation of (33) as follows.
Revised: (34), we can continue the derivation of (33) as follows.

Original: [Equation]
Revised: [Equation]

Original: The last step is because that
Revised: The last step is because that

Original: [Equation]
Revised: [Equation]

Original: ( b ) From Property 2, since that ( u (cid:62) x ) 2 t is an even function, we have a result for a general Gaussian distribution
Revised: ( b ) From Property 2, since that ( u (cid:62) x ) 2 t is an even function, we have a result for a general Gaussian distribution

Original: [Equation]
Revised: [Equation]

Original: Therefore, if there are L components in the Gaussian Mixture Model, then
Revised: Therefore, if there are L components in the Gaussian Mixture Model, then

Original: [Equation]
Revised: [Equation]

Original: Property 4 With Gaussian Mixture Model (7), we have
Revised: Property 4 With Gaussian Mixture Model (7), we have

Original: [Equation]
Revised: [Equation]

Original: Proof:
Revised: Proof:

Original: [Equation]
Revised: [Equation]

Original: In the 3rd step, we apply Jensen inequality because f ( x ) = x t is convex when x ≥ 0 and t ≥ 1 .
Revised: In the 3rd step, we apply Jensen inequality because f ( x ) = x t is convex when x ≥ 0 and t ≥ 1 .

Original: In the 4th step we apply the Binomial theorem and the result of k-order central moment of Gaussian variable.
Revised: In the 4th step we apply the Binomial theorem and the result of k-order central moment of Gaussian variable.

Original: Property 

Revised: Property

Original: The population risk function f ( W ) is deﬁned as
Revised: The population risk function f ( W ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Based on (2)
Revised: Based on (2)

Original: [Equation]
Revised: [Equation]

Original: (3) and (4)
Revised: (3) and (4)

Original: [Equation]
Revised: [Equation]

Original: we can derive its gradient and Hessian as follows.
Revised: we can derive its gradient and Hessian as follows.

Original: [Equation]
Revised: [Equation]

Original: Property 

Revised: Property

Original: With D m ( λ , M , σ ) deﬁned in deﬁnition 4, we have
Revised: With D m ( λ , M , σ ) deﬁned in deﬁnition 4, we have

Original: [Equation]
Revised: [Equation]

Original: Proof:
Revised: Proof:

Original: To prove (45), we can ﬁrst compare the terms (cid:80) Li
Revised: To prove (45), we can ﬁrst compare the terms (cid:80) Li

Original: =1 λ i a i (cid:80) Li =
Revised: =1 λ i a i (cid:80) Li =

Original: λ i a 2 i and (cid:80) Li =
Revised: λ i a 2 i and (cid:80) Li =

Original: λ i a 3 i , where a i ≥ 1 , i ∈
Revised: λ i a 3 i , where a i ≥ 1 , i ∈

Original: [ L ] and (cid:80) Li =

Revised: [ L ] and (cid:80) Li =

Original: λ i = 1 .
Revised: λ i = 1 .

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: λ i a 

Revised: λ i a

Original: i −
Revised: i −

Original: L (cid:88) i
Revised: L (cid:88) i

Original: = λ
Revised: = λ

Original: i a i
Revised: i a i

Original: L (cid:88) i =
Revised: L (cid:88) i =

Original: λ
Revised: λ

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: 2 i =
Revised: 2 i =

Original: L (cid:88)
Revised: L (cid:88)

Original: i
Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: (cid:1) =
Revised: (cid:1) =

Original: L (cid:88) i
Revised: L (cid:88) i

Original: = λ
Revised: = λ

Original: i a i
Revised: i a i

Original: [Equation]
Revised: [Equation]

Original: (1 − λ i )
Revised: (1 − λ i )

Original: [Equation]
Revised: [Equation]

Original: − (cid:88) 1 ≤ j ≤ L
Revised: − (cid:88) 1 ≤ j ≤ L

Original: [Equation]
Revised: [Equation]

Original: j (cid:54) =
Revised: j (cid:54) =

Original: i λ j a 2 j (cid:1) =
Revised: i λ j a 2 j (cid:1) =

Original: L (cid:88) i
Revised: L (cid:88) i

Original: = λ
Revised: = λ

Original: i a i · (cid:0) (cid:88) 1 ≤ j ≤ L,j (cid:54) =
Revised: i a i · (cid:0) (cid:88) 1 ≤ j ≤ L,j (cid:54) =

Original: i λ j a 

Revised: i λ j a

Original: i − (cid:88) 1 ≤ j ≤ L,j (cid:54) =
Revised: i − (cid:88) 1 ≤ j ≤ L,j (cid:54) =

Original: i λ j a 2 j (cid:1) =
Revised: i λ j a 2 j (cid:1) =

Original: L (cid:88)
Revised: L (cid:88)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: The second to last step is because we can ﬁnd the pairwise terms
Revised: The second to last step is because we can ﬁnd the pairwise terms

Original: λ i a i · λ j ( a 2 i − a 2 j ) and λ j a j · λ i ( a 2 j − a 2 i ) in the summation that can be putted together.
Revised: λ i a i · λ j ( a 2 i − a 2 j ) and λ j a j · λ i ( a 2 j − a 2 i ) in the summation that can be putted together.

Original: From (47), we can obtain
Revised: From (47), we can obtain

Original: [Equation]
Revised: [Equation]

Original: Combining (48) and the deﬁnition of D m ( λ , M , σ ) in (4), we can derive (45).
Revised: Combining (48) and the deﬁnition of D m ( λ , M , σ ) in (4), we can derive (45).

Original: Similarly, to prove (46), we can ﬁrst compare the terms ( (cid:80) Li =
Revised: Similarly, to prove (46), we can ﬁrst compare the terms ( (cid:80) Li =

Original: 1 λ i a i ) 2 and (cid:80) Li =
Revised: 1 λ i a i ) 2 and (cid:80) Li =

Original: λ i a 2 i , where a i ≥ 1 , i ∈
Revised: λ i a 2 i , where a i ≥ 1 , i ∈

Original: [ L ] and (cid:80) Li =

Revised: [ L ] and (cid:80) Li =

Original: λ i = 1 .
Revised: λ i = 1 .

Original: [Equation]
Revised: [Equation]

Original: The derivation of (49) is close to (47).
Revised: The derivation of (49) is close to (47).

Original: By (49) we have
Revised: By (49) we have

Original: [Equation]
Revised: [Equation]

Original: Combining (50) and the deﬁnition of D m ( λ , M , σ ) in (4), we can derive (46).
Revised: Combining (50) and the deﬁnition of D m ( λ , M , σ ) in (4), we can derive (46).

Original: B P ROOF OF T HEOREM
Revised: B P ROOF OF T HEOREM

Original: Theorem 1 is built upon three lemmas.
Revised: Theorem 1 is built upon three lemmas.

Original: Lemma 1 shows that with O ( dK 5 log 2 d ) samples, the empirical risk function is strongly convex in the neighborhood of W ∗ .
Revised: Lemma 1 shows that with O ( dK 5 log 2 d ) samples, the empirical risk function is strongly convex in the neighborhood of W ∗ .

Original: Lemma 2 shows that if initialized in the convex region, that the gradient descent algorithm converges linearly to a critical point (cid:99) W n , that is close to W ∗ .
Revised: Lemma 2 shows that if initialized in the convex region, that the gradient descent algorithm converges linearly to a critical point (cid:99) W n , that is close to W ∗ .

Original: Lemma 3 shows that the Tensor Initialization Method in Subroutine 1 initializes
Revised: Lemma 3 shows that the Tensor Initialization Method in Subroutine 1 initializes

Original: W 0 ∈ R d × K in the local convex region.
Revised: W 0 ∈ R d × K in the local convex region.

Original: Theorem 1 follows naturally by combining these three lemmas.
Revised: Theorem 1 follows naturally by combining these three lemmas.

Original: This proving approach is built upon those in (Fu et al., 2020).
Revised: This proving approach is built upon those in (Fu et al., 2020).

Original: One of our major technical contribution is extending Lemmas 1 and 2 to the Gaussian mixture model, while the results in (Fu et al., 2020) only apply to Standard Gaussian models.
Revised: One of our major technical contribution is extending Lemmas 1 and 2 to the Gaussian mixture model, while the results in (Fu et al., 2020) only apply to Standard Gaussian models.

Original: The second major contribution is a new tensor initialization method for Gaussian mixture model such that the initial point is in the convex region (see Lemma 3).
Revised: The second major contribution is a new tensor initialization method for Gaussian mixture model such that the initial point is in the convex region (see Lemma 3).

Original: Both contributions require the development of new tools, and our analyses are much more involved than those for the standard Gaussian due to the complexity introduced by the
Revised: Both contributions require the development of new tools, and our analyses are much more involved than those for the standard Gaussian due to the complexity introduced by the

Original: Gaussian mixture model.
Revised: Gaussian mixture model.

Original: To present these lemmas, the Euclidean ball B ( W ∗ , r ) is used to denote the neighborhood of W ∗ , where r is the radius of the ball.
Revised: To present these lemmas, the Euclidean ball B ( W ∗ , r ) is used to denote the neighborhood of W ∗ , where r is the radius of the ball.

Original: [Equation]
Revised: [Equation]

Original: The radius of the convex region is
Revised: The radius of the convex region is

Original: [Equation]
Revised: [Equation]

Original: Lemma 1 (Strongly local convexity) Consider the classiﬁcation model with FCN (2) and the sigmoid activation function.
Revised: Lemma 1 (Strongly local convexity) Consider the classiﬁcation model with FCN (2) and the sigmoid activation function.

Original: There exists a constant C such that as long as the sample size
Revised: There exists a constant C such that as long as the sample size

Original: [Equation]
Revised: [Equation]

Original: for some constant C 1 > 0 and (cid:15) 0 ∈ (0 , 14 ) , we have for all W ∈ B ( W ∗ , r FCN
Revised: for some constant C 1 > 0 and (cid:15) 0 ∈ (0 , 14 ) , we have for all W ∈ B ( W ∗ , r FCN

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: with probability at least 1 − d − 10 for some constant C 2 >
Revised: with probability at least 1 − d − 10 for some constant C 2 >

Original: [Equation]
Revised: [Equation]

Original: Lemma 2 (Linear convergence of gradient descent) Assume the conditions in Lemma 1 hold.
Revised: Lemma 2 (Linear convergence of gradient descent) Assume the conditions in Lemma 1 hold.

Original: If the local convexity holds, there exists a critical point in B ( W ∗ , r ) for some constant C 3 > 0 and (cid:15) 0 ∈ (0 , 12 ) , such that
Revised: If the local convexity holds, there exists a critical point in B ( W ∗ , r ) for some constant C 3 > 0 and (cid:15) 0 ∈ (0 , 12 ) , such that

Original: [Equation]
Revised: [Equation]

Original: If the initial point W 0 ∈ B ( W ∗ , r ) , the gradient descent linearly converges to (cid:99) W n , i.e.,
Revised: If the initial point W 0 ∈ B ( W ∗ , r ) , the gradient descent linearly converges to (cid:99) W n , i.e.,

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − d − 10 .
Revised: with probability at least 1 − d − 10 .

Original: Lemma 3 (Tensor initialization)
Revised: Lemma 3 (Tensor initialization)

Original: For classiﬁcation model, with D 6 ( λ , M , σ ) deﬁned in Deﬁnition 4, we have that if the sample size
Revised: For classiﬁcation model, with D 6 ( λ , M , σ ) deﬁned in Deﬁnition 4, we have that if the sample size

Original: [Equation]
Revised: [Equation]

Original: then the output W 0 ∈ R d × K satisﬁes
Revised: then the output W 0 ∈ R d × K satisﬁes

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − n − Ω( δ 41 )
Revised: with probability at least 1 − n − Ω( δ 41 )

Original: Proof of Theorem 1 and Corollary 1:
Revised: Proof of Theorem 1 and Corollary 1:

Original: From Lemma 2 and Lemma 3, we know that if n is sufﬁciently large such that the initialization Wby the tensor method is in the region B ( W ∗ , r ) , then the gradient descent method converges to a critical point (cid:99) W n that is sufﬁciently close to W ∗ .
Revised: From Lemma 2 and Lemma 3, we know that if n is sufﬁciently large such that the initialization Wby the tensor method is in the region B ( W ∗ , r ) , then the gradient descent method converges to a critical point (cid:99) W n that is sufﬁciently close to W ∗ .

Original: To achieve that, one sufﬁcient condition is
Revised: To achieve that, one sufﬁcient condition is

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality follows from || W || F ≤ √ K
Revised: where the ﬁrst inequality follows from || W || F ≤ √ K

Original: || W || for W ∈ R d × K , the second inequality comes from Lemma 3, and the third inequality comes from the requirement to be in the region
Revised: || W || for W ∈ R d × K , the second inequality comes from Lemma 3, and the third inequality comes from the requirement to be in the region

Original: B ( W ∗ , r ) .
Revised: B ( W ∗ , r ) .

Original: That is equivalent to the following condition
Revised: That is equivalent to the following condition

Original: [Equation]
Revised: [Equation]

Original: where C 0 = max { C 4 , C − 2 3 } .
Revised: where C 0 = max { C 4 , C − 2 3 } .

Original: By the deﬁnition
Revised: By the deﬁnition

Original: [Equation]
Revised: [Equation]

Original: we can obtain
Revised: we can obtain

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: From Property
Revised: From Property

Original: [Equation]
Revised: [Equation]

Original: we have that
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: Plugging (61), (62) into (60), we have
Revised: Plugging (61), (62) into (60), we have

Original: [Equation]
Revised: [Equation]

Original: Considering the requirements on the sample complexity in (53), (57) and (63)
Revised: Considering the requirements on the sample complexity in (53), (57) and (63)

Original: [Equation]
Revised: [Equation]

Original: (63) shows a sufﬁcient number of samples.
Revised: (63) shows a sufﬁcient number of samples.

Original: Taking the union bound of all the failure probabilities in Lemma 1, and 3, (63) holds with probability 1 − d − 10 .
Revised: Taking the union bound of all the failure probabilities in Lemma 1, and 3, (63) holds with probability 1 − d − 10 .

Original: By Property 1.3, ρ ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) can be lower bounded by positive and monotonically decreasing functions L m ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) when everything else except | µ l ( i ) | is ﬁxed, or
Revised: By Property 1.3, ρ ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) can be lower bounded by positive and monotonically decreasing functions L m ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) when everything else except | µ l ( i ) | is ﬁxed, or

Original: L s ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) when everything else except σ l is ﬁxed.
Revised: L s ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) when everything else except σ l is ﬁxed.

Original: Then, by substituting the lower bound of ρ ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) for itself in Γ( λ , M , σ , W ∗ ) , we can have an upper bound of ( σ max δ 1 ( W ∗ ))
Revised: Then, by substituting the lower bound of ρ ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) for itself in Γ( λ , M , σ , W ∗ ) , we can have an upper bound of ( σ max δ 1 ( W ∗ ))

Original: 2 τ 12 Γ( λ , M , σ , W ∗ ) − 2 D 12 ( λ , M , σ ) , denoted as B ( λ , M , σ , W ∗ ) .
Revised: 2 τ 12 Γ( λ , M , σ , W ∗ ) − 2 D 12 ( λ , M , σ ) , denoted as B ( λ , M , σ , W ∗ ) .

Original: To be more speciﬁc, when everything else except | µ l ( i ) | is ﬁxed, L m ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) is plugged in
Revised: To be more speciﬁc, when everything else except | µ l ( i ) | is ﬁxed, L m ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) is plugged in

Original: B ( λ , M , σ , W ∗ ) .
Revised: B ( λ , M , σ , W ∗ ) .

Original: Then since that
Revised: Then since that

Original: D 12 ( λ , M , σ , W ∗ ) and
Revised: D 12 ( λ , M , σ , W ∗ ) and

Original: L m ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) − 2 are both increasing function of µ l ( i ) , B ( λ , M , σ , W ∗ ) is an increasing function of | µ l ( i ) | .
Revised: L m ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) − 2 are both increasing function of µ l ( i ) , B ( λ , M , σ , W ∗ ) is an increasing function of | µ l ( i ) | .

Original: When everything else except σ l is ﬁxed, if σ l = σ max > ζ s , then σ 2max τ 12 D 12 ( λ , M , σ , W ∗ ) is an increasing function of σ l .
Revised: When everything else except σ l is ﬁxed, if σ l = σ max > ζ s , then σ 2max τ 12 D 12 ( λ , M , σ , W ∗ ) is an increasing function of σ l .

Original: Since that L s ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) is a decreasing function,
Revised: Since that L s ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) is a decreasing function,

Original: L s ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) − 2 is an increasing function of σ l .
Revised: L s ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) − 2 is an increasing function of σ l .

Original: Hence, B ( λ , M , σ , W ∗ ) is an increasing function of σ l .
Revised: Hence, B ( λ , M , σ , W ∗ ) is an increasing function of σ l .

Original: Moreover, when all σ l < ζ s (cid:48) and go to 0 , two decreasing functions of σ l , σ 2max L s ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) − 2 and D 12 ( λ , M , σ ) will be the dominant term of
Revised: Moreover, when all σ l < ζ s (cid:48) and go to 0 , two decreasing functions of σ l , σ 2max L s ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) − 2 and D 12 ( λ , M , σ ) will be the dominant term of

Original: B ( λ , M , σ , W ∗ ) .
Revised: B ( λ , M , σ , W ∗ ) .

Original: Therefore, B ( λ , M , σ , W ∗ ) increases to inﬁnity as all σ l ’s
Revised: Therefore, B ( λ , M , σ , W ∗ ) increases to inﬁnity as all σ l ’s

Original: go to 0 .
Revised: go to 0 .

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: Similarly, by replacing ρ ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) with L m ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) when everything else except | µ l ( i ) | is ﬁxed, or L s ( W ∗(cid:62)
Revised: Similarly, by replacing ρ ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) with L m ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) when everything else except | µ l ( i ) | is ﬁxed, or L s ( W ∗(cid:62)

Original: µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ ))
Revised: µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ ))

Original: (or σ − 2 L s ( W ∗(cid:62)
Revised: (or σ − 2 L s ( W ∗(cid:62)

Original: µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) for σ ≥ 1 ) when everything else except σ l is ﬁxed, (56) can also be transferred to another feasible upper bound.
Revised: µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) for σ ≥ 1 ) when everything else except σ l is ﬁxed, (56) can also be transferred to another feasible upper bound.

Original: We denote the modiﬁed version of the convergence rate as v = 1 − K − 2 q ( λ , M , σ , W ∗ ) .
Revised: We denote the modiﬁed version of the convergence rate as v = 1 − K − 2 q ( λ , M , σ , W ∗ ) .

Original: Since that q ( λ , M , σ , W ∗ ) is a ratio between the smallest and the largest singular value of ∇ 2 f ( W ∗ ) , we have q ( λ , M , σ , W ∗ ) ∈ (0 , 1) .
Revised: Since that q ( λ , M , σ , W ∗ ) is a ratio between the smallest and the largest singular value of ∇ 2 f ( W ∗ ) , we have q ( λ , M , σ , W ∗ ) ∈ (0 , 1) .

Original: Hence, we can obtain 1 − K − 2 q ( λ , M , σ , W ∗ ) ∈ (0 , 1) by K ≥ 1 .
Revised: Hence, we can obtain 1 − K − 2 q ( λ , M , σ , W ∗ ) ∈ (0 , 1) by K ≥ 1 .

Original: When everything else except | µ l ( i ) | is ﬁxed, since that L m ( W ∗(cid:62) µ l σ l δ K
Revised: When everything else except | µ l ( i ) | is ﬁxed, since that L m ( W ∗(cid:62) µ l σ l δ K

Original: ( W ∗ ) , σ l δ K ( W ∗ )) is monotonically decreasing and
Revised: ( W ∗ ) , σ l δ K ( W ∗ )) is monotonically decreasing and

Original: (cid:80) Ll =
Revised: (cid:80) Ll =

Original: 1 λ ( (cid:107) µ (cid:107) ∞ + σ l ) 2 is increasing as | µ l ( i ) | increases, v is an increasing function of | µ l ( i ) | to 1 .
Revised: 1 λ ( (cid:107) µ (cid:107) ∞ + σ l ) 2 is increasing as | µ l ( i ) | increases, v is an increasing function of | µ l ( i ) | to 1 .

Original: Similarly, when everything else except σ l is ﬁxed where σ l ≥ max { 1 , ζ s } ,
Revised: Similarly, when everything else except σ l is ﬁxed where σ l ≥ max { 1 , ζ s } ,

Original: [Equation]
Revised: [Equation]

Original: decreases to 0 as σ l increases.
Revised: decreases to 0 as σ l increases.

Original: We replace ρ ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) by σ − 2 L s ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) and then σ 2 · σ − 2 L s ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ ))
Revised: We replace ρ ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) by σ − 2 L s ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) and then σ 2 · σ − 2 L s ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ ))

Original: =
Revised: =

Original: L s ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) is an decreasing function less than ρ ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) .
Revised: L s ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) is an decreasing function less than ρ ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ )) .

Original: Therefore, v is an increasing function of σ l to 1 when σ l ≥ max { 1 , ζ s } .
Revised: Therefore, v is an increasing function of σ l to 1 when σ l ≥ max { 1 , ζ s } .

Original: When everything else except all σ l ≤ ζ s (cid:48)
Revised: When everything else except all σ l ≤ ζ s (cid:48)

Original: ’s go to 0 , all L s ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ ) ’s and
Revised: ’s go to 0 , all L s ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , σ l δ K ( W ∗ ) ’s and

Original: [Equation]
Revised: [Equation]

Original: ’s decrease to 0 .
Revised: ’s decrease to 0 .

Original: Therefore, v increases to 1 .
Revised: Therefore, v increases to 1 .

Original: The bound of (cid:107) (cid:99) W n − W ∗ (cid:107) F is directly from (55).
Revised: The bound of (cid:107) (cid:99) W n − W ∗ (cid:107) F is directly from (55).

Original: C P ROOF OF L EMMA
Revised: C P ROOF OF L EMMA

Original: We ﬁrst state some important lemmas used in proof in Section C.1 and describe the proof in Section
Revised: We ﬁrst state some important lemmas used in proof in Section C.1 and describe the proof in Section

Original: C.2.
Revised: C.2.

Original: The proofs of these lemmas are provided in Section C.3 to C.7 in sequence.
Revised: The proofs of these lemmas are provided in Section C.3 to C.7 in sequence.

Original: The proof idea mainly follows from (Fu et al.
Revised: The proof idea mainly follows from (Fu et al.

Original: Lemma 6 shows the Hessian ∇ 2 f ( W ) of the population risk function is smooth.
Revised: Lemma 6 shows the Hessian ∇ 2 f ( W ) of the population risk function is smooth.

Original: Lemma 7 illustrates that ∇ 2 f ( W ) is strongly convex in the neighborhood around µ ∗ .
Revised: Lemma 7 illustrates that ∇ 2 f ( W ) is strongly convex in the neighborhood around µ ∗ .

Original: Lemma 8 shows the Hessian of the empirical risk function ∇ 2 f n ( W ∗ ) is close to its population risk ∇ 2 f ( W ∗ ) in the local convex region.
Revised: Lemma 8 shows the Hessian of the empirical risk function ∇ 2 f n ( W ∗ ) is close to its population risk ∇ 2 f ( W ∗ ) in the local convex region.

Original: Summing up these three lemmas, we can derive the proof of Lemma 1.
Revised: Summing up these three lemmas, we can derive the proof of Lemma 1.

Original: Lemma 4 is used in the proof of Lemma 7.
Revised: Lemma 4 is used in the proof of Lemma 7.

Original: Lemma 5 is used in the proof of Lemma 8.
Revised: Lemma 5 is used in the proof of Lemma 8.

Original: The analysis of the Hessian matrix of the population loss in (Fu et al., 2020) and (Zhong et al., 2017b) can not be extended to the Gaussian mixture model.
Revised: The analysis of the Hessian matrix of the population loss in (Fu et al., 2020) and (Zhong et al., 2017b) can not be extended to the Gaussian mixture model.

Original: To solve this problem, we develop new tools using some good properties of symmetric distribution and even function.
Revised: To solve this problem, we develop new tools using some good properties of symmetric distribution and even function.

Original: Our approach can also be applied to other activations like tanh or erf.
Revised: Our approach can also be applied to other activations like tanh or erf.

Original: Moreover, if we directly apply the existing matrix concentration inequalities in these works in bounding the error between the empirical loss and the population loss, the resulting sample complexity would be O ( d 3 ) and cannot reﬂect the inﬂuence of each component of the Gaussian mixture distribution.
Revised: Moreover, if we directly apply the existing matrix concentration inequalities in these works in bounding the error between the empirical loss and the population loss, the resulting sample complexity would be O ( d 3 ) and cannot reﬂect the inﬂuence of each component of the Gaussian mixture distribution.

Original: We develop a new version of Bernstein’s inequality (see (136)) so that the ﬁnal bound is O
Revised: We develop a new version of Bernstein’s inequality (see (136)) so that the ﬁnal bound is O

Original: ( d log 2 d ) .
Revised: ( d log 2 d ) .

Original: (Mei et al. (2018a)) showed that the landscape of the empirical risk is close to that of the population risk when the number of samples is sufﬁciently large for the special case
Revised: (Mei et al. (2018a)) showed that the landscape of the empirical risk is close to that of the population risk when the number of samples is sufﬁciently large for the special case

Original: that K = 1 .
Revised: that K = 1 .

Original: Focusing on
Revised: Focusing on

Original: Gaussian mixture models, our result explicitly shows how the parameters of the input distribution, including the proportion, mean and variance of each component will affect the error bound between the empirical loss and the population loss in Lemma 8.
Revised: Gaussian mixture models, our result explicitly shows how the parameters of the input distribution, including the proportion, mean and variance of each component will affect the error bound between the empirical loss and the population loss in Lemma 8.

Original: C. U SEFUL L EMMAS IN THE PROOF OF L EMMA
Revised: C. U SEFUL L EMMAS IN THE PROOF OF L EMMA

Original: Lemma
Revised: Lemma

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ρ ( µ , σ ) is deﬁned in Deﬁnition 2.
Revised: where ρ ( µ , σ ) is deﬁned in Deﬁnition 2.

Original: Lemma 5 With the FCN model (2) and the Gaussian Mixture Model (7), for some constant C 12 > 0 , we have
Revised: Lemma 5 With the FCN model (2) and the Gaussian Mixture Model (7), for some constant C 12 > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 6 (Hessian smoothness of population loss)In the FCN model (2), assume || w ∗ k || 2 ≤ 1 for all k .
Revised: Lemma 6 (Hessian smoothness of population loss)In the FCN model (2), assume || w ∗ k || 2 ≤ 1 for all k .

Original: Then for some constant C 5 > 0 , we have
Revised: Then for some constant C 5 > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 7 (Local strong convexity of population loss)
Revised: Lemma 7 (Local strong convexity of population loss)

Original: In the FCN model (2) , if || W − W ∗ || F ≤ r for an (cid:15) 0 ∈ (0 , 14 ) , then for some constant C 4 > 0 ,
Revised: In the FCN model (2) , if || W − W ∗ || F ≤ r for an (cid:15) 0 ∈ (0 , 14 ) , then for some constant C 4 > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Lemma 

Revised: Lemma

Original: In the FCN model (2) , as long as n ≥ C (cid:48) · dK log dK for some constant C (cid:48) > 0 , we have
Revised: In the FCN model (2) , as long as n ≥ C (cid:48) · dK log dK for some constant C (cid:48) > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − d − 10 for some constant C 6 > 0 .
Revised: with probability at least 1 − d − 10 for some constant C 6 > 0 .

Original: C. P ROOF OF L EMMA
Revised: C. P ROOF OF L EMMA

Original: From Lemma 7 and 8, with probability at least 1 − d − 10 ,
Revised: From Lemma 7 and 8, with probability at least 1 − d − 10 ,

Original: [Equation]
Revised: [Equation]

Original: As long as the sample complexity is set to satisfy
Revised: As long as the sample complexity is set to satisfy

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e.,
Revised: e.,

Original: [Equation]
Revised: [Equation]

Original: for some constant C 1 >
Revised: for some constant C 1 >

Original: [Equation]
Revised: [Equation]

Original: then we have the lower bound of the Hessian with probability at least 1 − d − 10 .
Revised: then we have the lower bound of the Hessian with probability at least 1 − d − 10 .

Original: [Equation]
Revised: [Equation]

Original: By (68) and (69)
Revised: By (68) and (69)

Original: [Equation]
Revised: [Equation]

Original: we can also derive the upper bound as follows,
Revised: we can also derive the upper bound as follows,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for some constant C 2 > 0 .
Revised: for some constant C 2 > 0 .

Original: Combining (73) and (74), we have
Revised: Combining (73) and (74), we have

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − d − 10 .
Revised: with probability at least 1 − d − 10 .

Original: C. P ROOF OF L EMMA
Revised: C. P ROOF OF L EMMA

Original: Following the proof idea in Lemma D.4 of (Zhong et al., 2017b)
Revised: Following the proof idea in Lemma D.4 of (Zhong et al., 2017b)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: In A 0 , we know that E x ∼ 12 N ( µ , I d )+ 12 N ( − µ , I d ) x j = 0 .
Revised: In A 0 , we know that E x ∼ 12 N ( µ , I d )+ 12 N ( − µ , I d ) x j = 0 .

Original: Therefore,
Revised: Therefore,

Original: A 0 = k (cid:88) i =
Revised: A 0 = k (cid:88) i =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: In B 0 , α 1 ( i, µ , σ ) =
Revised: In B 0 , α 1 ( i, µ , σ ) =

Original: E x ∼ 12 N ( µ , I
Revised: E x ∼ 12 N ( µ , I

Original: [Equation]
Revised: [Equation]

Original: )+ 12 N ( − µ , I d ) ( x i φ
Revised: )+ 12 N ( − µ , I d ) ( x i φ

Original: [Equation]
Revised: [Equation]

Original: ( x i ))
Revised: ( x i ))

Original: = 0 .
Revised: = 0 .

Original: By the equation in Page 30 of (Zhong et al., 2017b), we have
Revised: By the equation in Page 30 of (Zhong et al., 2017b), we have

Original: B 0 =
Revised: B 0 =

Original: k (cid:88) i (cid:54)
Revised: k (cid:88) i (cid:54)

Original: =
Revised: =

Original: l
Revised: l

Original: E x ∼ 12 N
Revised: E x ∼ 12 N

Original: [Equation]
Revised: [Equation]

Original: 12 N
Revised: 12 N

Original: [Equation]
Revised: [Equation]

Original: I d )
Revised: I d )

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: (cid:105) =
Revised: (cid:105) =

Original: (cid:88) i (cid:54)
Revised: (cid:88) i (cid:54)

Original: =
Revised: =

Original: l p ii
Revised: l p ii

Original: [Equation]
Revised: [Equation]

Original: i (cid:54)
Revised: i (cid:54)

Original: [Equation]
Revised: [Equation]

Original: l
Revised: l

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: A 0 + B
Revised: A 0 + B

Original: [Equation]
Revised: [Equation]

Original: (cid:88) i = (cid:16) p ii
Revised: (cid:88) i = (cid:16) p ii

Original: [Equation]
Revised: [Equation]

Original: 2 ( i
Revised: 2 ( i

Original: [Equation]
Revised: [Equation]

Original: µ
Revised: µ

Original: [Equation]
Revised: [Equation]

Original: σ ) (cid:112) 1 + µ 2 i + (cid:88) l (cid:54) =
Revised: σ ) (cid:112) 1 + µ 2 i + (cid:88) l (cid:54) =

Original: i p li
Revised: i p li

Original: [Equation]
Revised: [Equation]

Original: 0 ( l
Revised: 0 ( l

Original: [Equation]
Revised: [Equation]

Original: µ
Revised: µ

Original: [Equation]
Revised: [Equation]

Original: σ ) (cid:113) 1 + µ 2 i (cid:17)
Revised: σ ) (cid:113) 1 + µ 2 i (cid:17)

Original: [Equation]
Revised: [Equation]

Original: − k (cid:88) i =
Revised: − k (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: ii
Revised: ii

Original: [Equation]
Revised: [Equation]

Original: 22 ( i
Revised: 22 ( i

Original: [Equation]
Revised: [Equation]

Original: σ ) 1 + µ 

Revised: σ ) 1 + µ

Original: i − k (cid:88) i = (cid:88) l (cid:54)
Revised: i − k (cid:88) i = (cid:88) l (cid:54)

Original: =
Revised: =

Original: i p 2 li α 0 ( l, µ , σ ) 2 (1 + µ 2 i ) + k (cid:88)
Revised: i p 2 li α 0 ( l, µ , σ ) 2 (1 + µ 2 i ) + k (cid:88)

Original: [Equation]
Revised: [Equation]

Original: ≥
Revised: ≥

Original: [Equation]
Revised: [Equation]

Original: C. P ROOF OF L EMMA
Revised: C. P ROOF OF L EMMA

Original: Following the equation (92) in Lemma 8 of (Fu et al., 2020) and by (44)
Revised: Following the equation (92) in Lemma 8 of (Fu et al., 2020) and by (44)

Original: [Equation]
Revised: [Equation]

Original: By Lagrange’s inequality
Revised: By Lagrange’s inequality

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: From Lemma
Revised: From Lemma

Original: [Equation]
Revised: [Equation]

Original: we know
Revised: we know

Original: [Equation]
Revised: [Equation]

Original: By Property 4, we have
Revised: By Property 4, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: for some constant C 12 >
Revised: for some constant C 12 >

Original: [Equation]
Revised: [Equation]

Original: C. P ROOF OF L EMMA
Revised: C. P ROOF OF L EMMA

Original: Let a = ( a (cid:62) 1 , · · · , a (cid:62) K ) (cid:62) ∈ R dK .
Revised: Let a = ( a (cid:62) 1 , · · · , a (cid:62) K ) (cid:62) ∈ R dK .

Original: Let ∆ j,l ∈ R d
Revised: Let ∆ j,l ∈ R d

Original: ×
Revised: ×

Original: d be the ( j, l ) -th block of ∇ 2 f ( W ) − ∇ 2 f ( W ∗ ) ∈
Revised: d be the ( j, l ) -th block of ∇ 2 f ( W ) − ∇ 2 f ( W ∗ ) ∈

Original: R dK ×
Revised: R dK ×

Original: dK .
Revised: dK .

Original: By deﬁnition,
Revised: By deﬁnition,

Original: [Equation]
Revised: [Equation]

Original: By the mean value theorem and (44),
Revised: By the mean value theorem and (44),

Original: [Equation]
Revised: [Equation]

Original: where W (cid:48)
Revised: where W (cid:48)

Original: =
Revised: =

Original: γ W + (1 − γ ) W ∗ for some γ ∈ (0 , 1) and T j,l,k is deﬁned such that ∂ξ j,l ( W (cid:48) )
Revised: γ W + (1 − γ ) W ∗ for some γ ∈ (0 , 1) and T j,l,k is deﬁned such that ∂ξ j,l ( W (cid:48) )

Original: ∂ w (cid:48) k
Revised: ∂ w (cid:48) k

Original: =
Revised: =

Original: T j,l,k · x ∈ R d .
Revised: T j,l,k · x ∈ R d .

Original: Then we provide an upper bound for ξ j,l .
Revised: Then we provide an upper bound for ξ j,l .

Original: Since that y = 1 or 0 , we ﬁrst compute the case in which y = 1 .
Revised: Since that y = 1 or 0 , we ﬁrst compute the case in which y = 1 .

Original: From (44) we can obtain
Revised: From (44) we can obtain

Original: [Equation]
Revised: [Equation]

Original: We can bound ξ j,l ( W ) by bounding each components of (89).
Revised: We can bound ξ j,l ( W ) by bounding each components of (89).

Original: Note that we have
Revised: Note that we have

Original: [Equation]
Revised: [Equation]

Original: where (90) holds for any j, l ∈
Revised: where (90) holds for any j, l ∈

Original: [ K ] .
Revised: [ K ] .

Original: The case y = 0 can be computed with the same upper bound by substituting (1 − H ( W ))
Revised: The case y = 0 can be computed with the same upper bound by substituting (1 − H ( W ))

Original: = 1 K (cid:80) Kj =1 (1 − φ ( w (cid:62) j x )) for H ( W ) in (89), (90) and (91).
Revised: = 1 K (cid:80) Kj =1 (1 − φ ( w (cid:62) j x )) for H ( W ) in (89), (90) and (91).

Original: Therefore, there exists a constant C 9 > 0 , such that
Revised: Therefore, there exists a constant C 9 > 0 , such that

Original: [Equation]
Revised: [Equation]

Original: We then need to calculate T j,l,k .
Revised: We then need to calculate T j,l,k .

Original: Following the analysis of ξ j,l ( W ) , we only consider the case of y = 1 here for simplicity.
Revised: Following the analysis of ξ j,l ( W ) , we only consider the case of y = 1 here for simplicity.

Original: [Equation]
Revised: [Equation]

Original: k x ) , where j, l, k are not equal to each other (93)
Revised: k x ) , where j, l, k are not equal to each other (93)

Original: T j,j,k
Revised: T j,j,k

Original: [Equation]
Revised: [Equation]

Original: (95) for some constant C 8 > 0 .
Revised: (95) for some constant C 8 > 0 .

Original: All the three inequalities of (95) are derived from Cauchy-Schwarz inequality.
Revised: All the three inequalities of (95) are derived from Cauchy-Schwarz inequality.

Original: Note that we have (cid:12)(cid:12)(cid:12) −
Revised: Note that we have (cid:12)(cid:12)(cid:12) −

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: ≤(98)
Revised: ≤(98)

Original: [Equation]
Revised: [Equation]

Original: Therefore, by combining (93), (94) and (96) to (99), we have
Revised: Therefore, by combining (93), (94) and (96) to (99), we have

Original: [Equation]
Revised: [Equation]

Original: for some constants C 7 > 0 .
Revised: for some constants C 7 > 0 .

Original: By (87), (88), (95), (100) and the Cauchy-Schwarz’s Inequality
Revised: By (87), (88), (95), (100) and the Cauchy-Schwarz’s Inequality

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: for some constant C 5 > 0 .
Revised: for some constant C 5 > 0 .

Original: C. P ROOF OF L EMMA
Revised: C. P ROOF OF L EMMA

Original: From (Fu et al.
Revised: From (Fu et al.

Original: , we know
Revised: , we know

Original: [Equation]
Revised: [Equation]

Original: (103) with a = ( a (cid:62) 1 , · · · , a (cid:62) K ) (cid:62) ∈ R dK .
Revised: (103) with a = ( a (cid:62) 1 , · · · , a (cid:62) K ) (cid:62) ∈ R dK .

Original: And ∇ 2 f ( W ∗ ) (cid:22) (cid:16) max || a
Revised: And ∇ 2 f ( W ∗ ) (cid:22) (cid:16) max || a

Original: [Equation]
Revised: [Equation]

Original: for some constant C 4 > 0 .
Revised: for some constant C 4 > 0 .

Original: By applying Property 3, we can derive the upper bound in (104) as
Revised: By applying Property 3, we can derive the upper bound in (104) as

Original: [Equation]
Revised: [Equation]

Original: To ﬁnd a lower bound for (103), we can ﬁrst transfer the expectation of the Gaussian Mixture Model to the weight sum of the expectations over general Gaussian distributions.
Revised: To ﬁnd a lower bound for (103), we can ﬁrst transfer the expectation of the Gaussian Mixture Model to the weight sum of the expectations over general Gaussian distributions.

Original: [Equation]
Revised: [Equation]

Original: Denote U ∈ R
Revised: Denote U ∈ R

Original: d × k as the orthogonal basis of W ∗ .
Revised: d × k as the orthogonal basis of W ∗ .

Original: For any vector a i ∈ R d , there exists two vectors b i ∈ R K and c
Revised: For any vector a i ∈ R d , there exists two vectors b i ∈ R K and c

Original: i ∈ R
Revised: i ∈ R

Original: d − K such that
Revised: d − K such that

Original: [Equation]
Revised: [Equation]

Original: where U ⊥ ∈ R d × ( d − K ) denotes the complement of U .
Revised: where U ⊥ ∈ R d × ( d − K ) denotes the complement of U .

Original: We also have U (cid:62)⊥ µ l = 0 by (9).
Revised: We also have U (cid:62)⊥ µ l = 0 by (9).

Original: Plugging (107) into RHS of (106), and then we have
Revised: Plugging (107) into RHS of (106), and then we have

Original: [Equation]
Revised: [Equation]

Original: where the last step is by U (cid:62)⊥ µ l = 0 by (9).
Revised: where the last step is by U (cid:62)⊥ µ l = 0 by (9).

Original: [Equation]
Revised: [Equation]

Original: by deﬁning t
Revised: by deﬁning t

Original: [Equation]
Revised: [Equation]

Original: The last step is by U (cid:62)⊥ µ l = 0 .
Revised: The last step is by U (cid:62)⊥ µ l = 0 .

Original: The 4th step is because that s i is independent of t i , thus E
Revised: The 4th step is because that s i is independent of t i , thus E

Original: [ t i t
Revised: [ t i t

Original: [Equation]
Revised: [Equation]

Original: Since (cid:16) (cid:80) ki =1 p (cid:62) i x · φ (cid:48) ( σ · x i ) (cid:17) 2 is an even function, so from Property 2 we have
Revised: Since (cid:16) (cid:80) ki =1 p (cid:62) i x · φ (cid:48) ( σ · x i ) (cid:17) 2 is an even function, so from Property 2 we have

Original: [Equation]
Revised: [Equation]

Original: Combining Lemma 4 and Property 2, we next follow the derivation for the standard Gaussian distribution in Page 36 of (Zhong et al., 2017b) and generalize the result to a Gaussian distribution with an arbitrary mean and variance as follows.
Revised: Combining Lemma 4 and Property 2, we next follow the derivation for the standard Gaussian distribution in Page 36 of (Zhong et al., 2017b) and generalize the result to a Gaussian distribution with an arbitrary mean and variance as follows.

Original: [Equation]
Revised: [Equation]

Original: The second step is by letting z
Revised: The second step is by letting z

Original: = U (cid:62) x .
Revised: = U (cid:62) x .

Original: The third step is by letting s
Revised: The third step is by letting s

Original: = V (cid:62) z .
Revised: = V (cid:62) z .

Original: The last to second step follows from g =
Revised: The last to second step follows from g =

Original: s σ l δ K ( W ∗ ) ,
Revised: s σ l δ K ( W ∗ ) ,

Original: where
Revised: where

Original: g ∼ N ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , I K ) and the last inequality is by
Revised: g ∼ N ( W ∗(cid:62) µ l σ l δ K ( W ∗ ) , I K ) and the last inequality is by

Original: Lemma 4.
Revised: Lemma 4.

Original: Similarly, we extend the derivation in Page 37 of (Zhong et al., 2017b) for the standard
Revised: Similarly, we extend the derivation in Page 37 of (Zhong et al., 2017b) for the standard

Original: Gaussian distribution to a general Gaussian distribution as follows.
Revised: Gaussian distribution to a general Gaussian distribution as follows.

Original: [Equation]
Revised: [Equation]

Original: Combining (108) - (111), (114) and (115), we have min
Revised: Combining (108) - (111), (114) and (115), we have min

Original: [Equation]
Revised: [Equation]

Original: For the Gaussian Mixture Model x ∼ (cid:80) Ll =1 N ( µ l , σ 2 l I d ) , we have min || a ||
Revised: For the Gaussian Mixture Model x ∼ (cid:80) Ll =1 N ( µ l , σ 2 l I d ) , we have min || a ||

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: x ∼
Revised: x ∼

Original: [Equation]
Revised: [Equation]

Original: λ
Revised: λ

Original: [Equation]
Revised: [Equation]

Original: (cid:104)
Revised: (cid:104)

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: From (67) in Lemma 6, since that we have the condition (cid:107) W − W ∗ (cid:107) F ≤ r and (52), we can obtain ||∇
Revised: From (67) in Lemma 6, since that we have the condition (cid:107) W − W ∗ (cid:107) F ≤ r and (52), we can obtain ||∇

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: ||
Revised: ||

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) 0 ∈ (0 , 14 ) .
Revised: where (cid:15) 0 ∈ (0 , 14 ) .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: The last inequality of (121) holds since C 4 · (cid:80)
Revised: The last inequality of (121) holds since C 4 · (cid:80)

Original: [Equation]
Revised: [Equation]

Original: =1 λ l
Revised: =1 λ l

Original: [Equation]
Revised: [Equation]

Original: 4 K
Revised: 4 K

Original: [Equation]
Revised: [Equation]

Original: Ll =1 λ l σ 2 l ηκ 2 ρ ( W ∗(cid:62) µ l σ l δ K
Revised: Ll =1 λ l σ 2 l ηκ 2 ρ ( W ∗(cid:62) µ l σ l δ K

Original: ( W ∗ ) , σ l δ K ( W ∗ ))
Revised: ( W ∗ ) , σ l δ K ( W ∗ ))

Original: =
Revised: =

Original: O ( σ 2max K 2 ) and O ( σ 2max ) ≥ Ω( σ 2max K 2 ) .
Revised: O ( σ 2max K 2 ) and O ( σ 2max ) ≥ Ω( σ 2max K 2 ) .

Original: Combining (120) and (121), we have
Revised: Combining (120) and (121), we have

Original: [Equation]
Revised: [Equation]

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: δ K ( W ∗ ) , σ l δ K ( W ∗ ))
Revised: δ K ( W ∗ ) , σ l δ K ( W ∗ ))

Original: · I (cid:22) ∇ 2 f ( W ) (cid:22) C 4 ·
Revised: · I (cid:22) ∇ 2 f ( W ) (cid:22) C 4 ·

Original: L (cid:88) l = λ l ( (cid:107) µ (cid:107) ∞ + σ l ) 2 · I (122)
Revised: L (cid:88) l = λ l ( (cid:107) µ (cid:107) ∞ + σ l ) 2 · I (122)

Original: C. P ROOF OF L EMMA
Revised: C. P ROOF OF L EMMA

Original: Let N (cid:15) be the (cid:15) -covering number of the Euclidean ball B ( W ∗ , r ) .
Revised: Let N (cid:15) be the (cid:15) -covering number of the Euclidean ball B ( W ∗ , r ) .

Original: It is known that log N (cid:15) ≤ dK log( 3 r(cid:15) ) from (Vershynin, 2010).
Revised: It is known that log N (cid:15) ≤ dK log( 3 r(cid:15) ) from (Vershynin, 2010).

Original: Let W (cid:15) =
Revised: Let W (cid:15) =

Original: { W 1 , ..., W N (cid:15) } be the (cid:15) -cover set with N (cid:15) elements.
Revised: { W 1 , ..., W N (cid:15) } be the (cid:15) -cover set with N (cid:15) elements.

Original: For any W ∈ B ( W ∗ , r ) , let j ( W ) =
Revised: For any W ∈ B ( W ∗ , r ) , let j ( W ) =

Original: arg min j ∈ [ N (cid:15) ]
Revised: arg min j ∈ [ N (cid:15) ]

Original: || W − W j ( W ) || F ≤ (cid:15) for all W ∈ B ( W ∗ , r ) .
Revised: || W − W j ( W ) || F ≤ (cid:15) for all W ∈ B ( W ∗ , r ) .

Original: Then for any W ∈ B ( W ∗ , r ) , we have
Revised: Then for any W ∈ B ( W ∗ , r ) , we have

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: Hence, we have
Revised: Hence, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where A t , B t and C t are deﬁned as
Revised: where A t , B t and C t are deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Then we bound P ( A t ) , P ( B t ) and P ( C t ) separately.
Revised: Then we bound P ( A t ) , P ( B t ) and P ( C t ) separately.

Original: 1) Upper bound on P ( B t ) .
Revised: 1) Upper bound on P ( B t ) .

Original: By Lemma 6 in (Fu et al., 2020), we obtain
Revised: By Lemma 6 in (Fu et al., 2020), we obtain

Original: [Equation]
Revised: [Equation]

Original: where V 14 is a 14 -cover of the unit-Euclidean-norm ball B ( 0 , 1) with log | V 14 | ≤ dK log 12 .
Revised: where V 14 is a 14 -cover of the unit-Euclidean-norm ball B ( 0 , 1) with log | V 14 | ≤ dK log 12 .

Original: Taking the union bound over W (cid:15) and V 14 , we have
Revised: Taking the union bound over W (cid:15) and V 14 , we have

Original: [Equation]
Revised: [Equation]

Original: ) where G i = (cid:68) v , ( ∇ 2 (cid:96) ( W , x i ) − E x ∼ (cid:80) Ll =1 λ l N ( µ l ,σ 2 l I d ) [ ∇ 2 (cid:96) ( W , x i )] v ) (cid:69) and E [ G i ] = 0 .
Revised: ) where G i = (cid:68) v , ( ∇ 2 (cid:96) ( W , x i ) − E x ∼ (cid:80) Ll =1 λ l N ( µ l ,σ 2 l I d ) [ ∇ 2 (cid:96) ( W , x i )] v ) (cid:69) and E [ G i ] = 0 .

Original: Here v = ( u (cid:62) 1 , · · · , u (cid:62) K ) (cid:62) ∈ R dK .
Revised: Here v = ( u (cid:62) 1 , · · · , u (cid:62) K ) (cid:62) ∈ R dK .

Original: [Equation]
Revised: [Equation]

Original: for some C 9 > 0 .
Revised: for some C 9 > 0 .

Original: The ﬁrst step of (130) is by (43).
Revised: The ﬁrst step of (130) is by (43).

Original: The last step is by (92) and the CauchySchwarz’s Inequality.
Revised: The last step is by (92) and the CauchySchwarz’s Inequality.

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: 131) where the second to last inequality results from Property 3.
Revised: 131) where the second to last inequality results from Property 3.

Original: The last inequality is because v ∈ V 14 ,
Revised: The last inequality is because v ∈ V 14 ,

Original: [Equation]
Revised: [Equation]

Original: K j =
Revised: K j =

Original: [Equation]
Revised: [Equation]

Original: ||
Revised: ||

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: ||
Revised: ||

Original: [Equation]
Revised: [Equation]

Original: ||
Revised: ||

Original: [Equation]
Revised: [Equation]

Original: ≤ 1 .
Revised: ≤ 1 .

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality holds from p !
Revised: where the ﬁrst inequality holds from p !

Original: ≥ ( pe ) p and (131), and the third line holds provided that
Revised: ≥ ( pe ) p and (131), and the third line holds provided that

Original: [Equation]
Revised: [Equation]

Original: Note that the quantity inside the maximization in (133) achieves its maximum when p = 2 , because it is monotonously decreasing.
Revised: Note that the quantity inside the maximization in (133) achieves its maximum when p = 2 , because it is monotonously decreasing.

Original: Therefore, (133) holds if θ ≤ 274 e (cid:80) Ll
Revised: Therefore, (133) holds if θ ≤ 274 e (cid:80) Ll

Original: =1 λ l ( (cid:107) µ l (cid:107) ∞ + σ l ) 2 .
Revised: =1 λ l ( (cid:107) µ l (cid:107) ∞ + σ l ) 2 .

Original: Then
Revised: Then

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: for some constant C 10 > 0 .
Revised: for some constant C 10 > 0 .

Original: The ﬁrst inequality follows from the Markov’s Inequality.
Revised: The ﬁrst inequality follows from the Markov’s Inequality.

Original: When θ = min {
Revised: When θ = min {

Original: [Equation]
Revised: [Equation]

Original: } , we have a modiﬁed Bernstein’s Inequality for the Gaussian Mixture Model as follows
Revised: } , we have a modiﬁed Bernstein’s Inequality for the Gaussian Mixture Model as follows

Original: [Equation]
Revised: [Equation]

Original: 135) for some constant C 11 > 0 .
Revised: 135) for some constant C 11 > 0 .

Original: We can obtain the same bound for P ( − 1 n (cid:80) ni
Revised: We can obtain the same bound for P ( − 1 n (cid:80) ni

Original: =

Revised: =

Original: G i ≥ t 6 ) by replacing
Revised: G i ≥ t 6 ) by replacing

Original: G i as − G i .
Revised: G i as − G i .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: Thus, as long as
Revised: Thus, as long as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for some large constant C 6 > 0 , we have P ( B t ) ≤ δ 2 .
Revised: for some large constant C 6 > 0 , we have P ( B t ) ≤ δ 2 .

Original: 2) Upper bound on P ( A t ) and P ( C t ) .
Revised: 2) Upper bound on P ( A t ) and P ( C t ) .

Original: From Lemma 5, we can obtain sup
Revised: From Lemma 5, we can obtain sup

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: C t holds if
Revised: C t holds if

Original: [Equation]
Revised: [Equation]

Original: We can bound the A t as below.
Revised: We can bound the A t as below.

Original: [Equation]
Revised: [Equation]

Original: Thus, taking
Revised: Thus, taking

Original: [Equation]
Revised: [Equation]

Original: δ (141) ensures that P ( A t ) ≤ δ 2 .
Revised: δ (141) ensures that P ( A t ) ≤ δ 2 .

Original: 3) Final step
Revised: 3) Final step

Original: Let (cid:15) =
Revised: Let (cid:15) =

Original: δ
Revised: δ

Original: [Equation]
Revised: [Equation]

Original: and δ = d
Revised: and δ = d

Original: [Equation]
Revised: [Equation]

Original: 10 , then from (137) and (141) we need
Revised: 10 , then from (137) and (141) we need

Original: [Equation]
Revised: [Equation]

Original: So by setting t = (cid:80) Ll =
Revised: So by setting t = (cid:80) Ll =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ∞ +
Revised: ∞ +

Original: [Equation]
Revised: [Equation]

Original: dK log n n , as long as n ≥ C (cid:48)
Revised: dK log n n , as long as n ≥ C (cid:48)

Original: dK log dK , we have
Revised: dK log dK , we have

Original: [Equation]
Revised: [Equation]

Original: D P ROOF OF L EMMA
Revised: D P ROOF OF L EMMA

Original: We ﬁrst present a lemma used in proving Lemma 2 in Section D.1 and then prove Lemma 2 in
Revised: We ﬁrst present a lemma used in proving Lemma 2 in Section D.1 and then prove Lemma 2 in

Original: Section D.2.
Revised: Section D.2.

Original: D. A USEFUL LEMMA USED IN THE PROOF
Revised: D. A USEFUL LEMMA USED IN THE PROOF

Original: Lemma 9 If r is deﬁned in (52) for (cid:15) 0 ∈ (0 , 14 ) , then with probability at least 1 − d − 10 , we have
Revised: Lemma 9 If r is deﬁned in (52) for (cid:15) 0 ∈ (0 , 14 ) , then with probability at least 1 − d − 10 , we have

Original: [Equation]
Revised: [Equation]

Original: for some constant C 13 > 0 .
Revised: for some constant C 13 > 0 .

Original: Proof:
Revised: Proof:

Original: Similar to the idea of the proof of Lemma
Revised: Similar to the idea of the proof of Lemma

Original: [Equation]
Revised: [Equation]

Original: we adopt an (cid:15) -covering net of the ball B ( W ∗ , r ) to build a relationship between any arbitrary point in the ball and the points in the covering set.
Revised: we adopt an (cid:15) -covering net of the ball B ( W ∗ , r ) to build a relationship between any arbitrary point in the ball and the points in the covering set.

Original: We can then divide the distance between ∇ f n ( W ) and ∇ f ( W ) into three parts, similar to (123).
Revised: We can then divide the distance between ∇ f n ( W ) and ∇ f ( W ) into three parts, similar to (123).

Original: (145) to (147) can be derived in a similar way as (125) to (127), with “ ∇ 2 ” replaced by “ ∇ ”.
Revised: (145) to (147) can be derived in a similar way as (125) to (127), with “ ∇ 2 ” replaced by “ ∇ ”.

Original: Then we need to bound P ( A (cid:48) t ) , P ( B (cid:48) t ) and P ( C (cid:48) t ) respectively, where A (cid:48) t , B (cid:48) t and C (cid:48) t are deﬁned below.
Revised: Then we need to bound P ( A (cid:48) t ) , P ( B (cid:48) t ) and P ( C (cid:48) t ) respectively, where A (cid:48) t , B (cid:48) t and C (cid:48) t are deﬁned below.

Original: [Equation]
Revised: [Equation]

Original: a) Upper bound of P ( B (cid:48) t ) .
Revised: a) Upper bound of P ( B (cid:48) t ) .

Original: Applying Lemma 3 in (Mei et al., 2018a), we have
Revised: Applying Lemma 3 in (Mei et al., 2018a), we have

Original: [Equation]
Revised: [Equation]

Original: Deﬁne G (cid:48)
Revised: Deﬁne G (cid:48)

Original: i = (cid:68) v , ( ∇
Revised: i = (cid:68) v , ( ∇

Original: [Equation]
Revised: [Equation]

Original: W , x
Revised: W , x

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: ) [ ∇ (cid:96) ( W , x i )]) (cid:69) .
Revised: ) [ ∇ (cid:96) ( W , x i )]) (cid:69) .

Original: Here v ∈ R d .
Revised: Here v ∈ R d .

Original: To compute ∇ (cid:96) ( W , x i ) , we require the derivation in Property 5.
Revised: To compute ∇ (cid:96) ( W , x i ) , we require the derivation in Property 5.

Original: Then we can have an upper bound of ζ ( W ) in (42).
Revised: Then we can have an upper bound of ζ ( W ) in (42).

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then we have an upper bound of G (cid:48) i .
Revised: Then we have an upper bound of G (cid:48) i .

Original: [Equation]
Revised: [Equation]

Original: Following the idea of (131) and (132), and by v ∈ V 1 2 , we have
Revised: Following the idea of (131) and (132), and by v ∈ V 1 2 , we have

Original: [Equation]
Revised: [Equation]

Original: 152) where (152) holds if θ ≤ 274 e (cid:113)(cid:80)
Revised: 152) where (152) holds if θ ≤ 274 e (cid:113)(cid:80)

Original: Ll
Revised: Ll

Original: =1 λ l ( (cid:107) µ l (cid:107) ∞ + σ l ) 2 .
Revised: =1 λ l ( (cid:107) µ l (cid:107) ∞ + σ l ) 2 .

Original: Following the derivation of (129) and (134) to (137), we have
Revised: Following the derivation of (129) and (134) to (137), we have

Original: P ( | 1 n n (cid:88) i =
Revised: P ( | 1 n n (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: ) ≤
Revised: ) ≤

Original: [Equation]
Revised: [Equation]

Original: (cid:16) max
Revised: (cid:16) max

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: C 14 nt144 (cid:80)
Revised: C 14 nt144 (cid:80)

Original: [Equation]
Revised: [Equation]

Original: for some constant C 14 > 0 and C 15 > 0 .
Revised: for some constant C 14 > 0 and C 15 > 0 .

Original: Moreover, we can obtain P ( B (cid:48) t ) ≤ δ 2 as long as
Revised: Moreover, we can obtain P ( B (cid:48) t ) ≤ δ 2 as long as

Original: [Equation]
Revised: [Equation]

Original: (b) For the upper bound of P ( A (cid:48) t ) and P ( C (cid:48) t ) , we can ﬁrst derive
Revised: (b) For the upper bound of P ( A (cid:48) t ) and P ( C (cid:48) t ) , we can ﬁrst derive

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst inequality is by (42).
Revised: The ﬁrst inequality is by (42).

Original: The second inequality is by the Mean Value Theorem.
Revised: The second inequality is by the Mean Value Theorem.

Original: The third step is by (92).
Revised: The third step is by (92).

Original: The last inequality is by Property 4.
Revised: The last inequality is by Property 4.

Original: Therefore, following the steps in part (2) of Lemma
Revised: Therefore, following the steps in part (2) of Lemma

Original: [Equation]
Revised: [Equation]

Original: we can conclude that C (cid:48) t holds if
Revised: we can conclude that C (cid:48) t holds if

Original: [Equation]
Revised: [Equation]

Original: Moreover, from (141) in Lemma 8 we have that
Revised: Moreover, from (141) in Lemma 8 we have that

Original: [Equation]
Revised: [Equation]

Original: ensures P ( A (cid:48) t ) ≤ δ 2 .
Revised: ensures P ( A (cid:48) t ) ≤ δ 2 .

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: let (cid:15) =
Revised: let (cid:15) =

Original: [Equation]
Revised: [Equation]

Original: δ
Revised: δ

Original: [Equation]
Revised: [Equation]

Original: and t
Revised: and t

Original: [Equation]
Revised: [Equation]

Original: d log n n , if n ≥ C
Revised: d log n n , if n ≥ C

Original: [Equation]
Revised: [Equation]

Original: · dK log dK for some constant C (cid:48)(cid:48) > 0 , we have
Revised: · dK log dK for some constant C (cid:48)(cid:48) > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: D. P ROOF OF L EMMA
Revised: D. P ROOF OF L EMMA

Original: Following the proof of Theorem 2 in [Fu et al.
Revised: Following the proof of Theorem 2 in [Fu et al.

Original: (2020)], ﬁrst
Revised: (2020)], ﬁrst

Original: we have the Taylor’s expansion of f n ( (cid:99) W n )
Revised: we have the Taylor’s expansion of f n ( (cid:99) W n )

Original: [Equation]
Revised: [Equation]

Original: Here W (cid:48) is on the straight line connecting W ∗ and (cid:99) W n .
Revised: Here W (cid:48) is on the straight line connecting W ∗ and (cid:99) W n .

Original: By the fact that f n ( (cid:99) W n ) ≤ f n ( W ∗ ) , we have
Revised: By the fact that f n ( (cid:99) W n ) ≤ f n ( W ∗ ) , we have

Original: [Equation]
Revised: [Equation]

Original: From Lemma 7 and Lemma
Revised: From Lemma 7 and Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: The second to last step of (162) comes from the triangle inequality and the last step follows from the fact ∇ f ( W ∗ )
Revised: The second to last step of (162) comes from the triangle inequality and the last step follows from the fact ∇ f ( W ∗ )

Original: = 0 .
Revised: = 0 .

Original: Combining (160), (161) and (162), we have
Revised: Combining (160), (161) and (162), we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have concluded that there indeed exists a critical point (cid:99) W in B ( W ∗ , r ) .
Revised: Therefore, we have concluded that there indeed exists a critical point (cid:99) W in B ( W ∗ , r ) .

Original: Then we show the linear convergence of Algorithm 1 as below.
Revised: Then we show the linear convergence of Algorithm 1 as below.

Original: By the update rule, we have
Revised: By the update rule, we have

Original: [Equation]
Revised: [Equation]

Original: where W ( γ ) = γ (cid:99) W n + (1 − γ ) W t for γ ∈ (0 , 1) .
Revised: where W ( γ ) = γ (cid:99) W n + (1 − γ ) W t for γ ∈ (0 , 1) .

Original: Since W ( γ ) ∈ B ( W ∗ , r ) , by Lemma
Revised: Since W ( γ ) ∈ B ( W ∗ , r ) , by Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where H min = Ω
Revised: where H min = Ω

Original: [Equation]
Revised: [Equation]

Original: µ
Revised: µ

Original: [Equation]
Revised: [Equation]

Original: ∞ + σ l ) 2 .
Revised: ∞ + σ l ) 2 .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: By setting η 0 =
Revised: By setting η 0 =

Original: [Equation]
Revised: [Equation]

Original: , we obtain
Revised: , we obtain

Original: [Equation]
Revised: [Equation]

Original: Therefore, Algorithm 1 converges linearly to the local minimizer.
Revised: Therefore, Algorithm 1 converges linearly to the local minimizer.

Original: E P ROOF OF L EMMA
Revised: E P ROOF OF L EMMA

Original: We need Lemma 10 to Lemma 14, which are stated in Section E.1, for the proof of Lemma 3.
Revised: We need Lemma 10 to Lemma 14, which are stated in Section E.1, for the proof of Lemma 3.

Original: Section E.2 summarizes the proof of Lemma 3.
Revised: Section E.2 summarizes the proof of Lemma 3.

Original: The proofs of Lemma 10 to Lemma 12 are provided in Section E.3 to Section E.5.
Revised: The proofs of Lemma 10 to Lemma 12 are provided in Section E.3 to Section E.5.

Original: Lemma 13 and Lemma 14 are cited from (Zhong et al., 2017b).
Revised: Lemma 13 and Lemma 14 are cited from (Zhong et al., 2017b).

Original: Although (Zhong et al., 2017b) considers the standard Gaussian distribution, the proofs of Lemmaand 14 hold for any data distribution.
Revised: Although (Zhong et al., 2017b) considers the standard Gaussian distribution, the proofs of Lemmaand 14 hold for any data distribution.

Original: Therefore, these two lemmas can be applied here directly.
Revised: Therefore, these two lemmas can be applied here directly.

Original: The tensor initialization in (Zhong et al., 2017b) only holds for the standard Gaussian distribution.
Revised: The tensor initialization in (Zhong et al., 2017b) only holds for the standard Gaussian distribution.

Original: We exploit a more general deﬁnition of tensors from (Janzamin et al.
Revised: We exploit a more general deﬁnition of tensors from (Janzamin et al.

Original: for the tensor initialization in our algorithm.
Revised: for the tensor initialization in our algorithm.

Original: We also develop new error bounds for the initialization.
Revised: We also develop new error bounds for the initialization.

Original: E. U SEFUL LEMMAS IN THE PROOF
Revised: E. U SEFUL LEMMAS IN THE PROOF

Original: Lemma 10 Let P 2 follow Deﬁnition 1.
Revised: Lemma 10 Let P 2 follow Deﬁnition 1.

Original: Let S be a set of i.i.d.
Revised: Let S be a set of i.i.d.

Original: samples generated from the mixed
Revised: samples generated from the mixed

Original: Gaussian distribution (cid:80) Ll
Revised: Gaussian distribution (cid:80) Ll

Original: =1 λ l N ( µ l , σ 2 l I ) .
Revised: =1 λ l N ( µ l , σ 2 l I ) .

Original: Let (cid:98) P 2 be the empirical version of P 2 using data set S .
Revised: Let (cid:98) P 2 be the empirical version of P 2 using data set S .

Original: Then with probability at least 1 − 2 n − Ω( δ 41 d ) , we have
Revised: Then with probability at least 1 − 2 n − Ω( δ 41 d ) , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 11 Let U ∈
Revised: Lemma 11 Let U ∈

Original: E
Revised: E

Original: d × K be the orthogonal column span of W ∗ .
Revised: d × K be the orthogonal column span of W ∗ .

Original: Let α be a ﬁxed unit vector and (cid:98)
Revised: Let α be a ﬁxed unit vector and (cid:98)

Original: U ∈ R
Revised: U ∈ R

Original: d × K denote an orthogonal matrix satisfying || UU (cid:62) − (cid:98)
Revised: d × K denote an orthogonal matrix satisfying || UU (cid:62) − (cid:98)

Original: U (cid:98) U (cid:62) || ≤ 14 .
Revised: U (cid:98) U (cid:62) || ≤ 14 .

Original: Deﬁne
Revised: Deﬁne

Original: R 3 =
Revised: R 3 =

Original: M 3 ( (cid:98) U , (cid:98) U , (cid:98) U ) , where M 3 is deﬁned in Deﬁnition 1.
Revised: M 3 ( (cid:98) U , (cid:98) U , (cid:98) U ) , where M 3 is deﬁned in Deﬁnition 1.

Original: Let (cid:98) R 3 be the empirical version of
Revised: Let (cid:98) R 3 be the empirical version of

Original: R 3 using data set S , where each sample of S is i.i.d.
Revised: R 3 using data set S , where each sample of S is i.i.d.

Original: sampled from the mixed Gaussian distribution (cid:80)
Revised: sampled from the mixed Gaussian distribution (cid:80)

Original: Ll =1 λ l N ( µ l , σ 2 l I ) .
Revised: Ll =1 λ l N ( µ l , σ 2 l I ) .

Original: Then with probability at least 1 − n − Ω( δ 4 ) , we have
Revised: Then with probability at least 1 − n − Ω( δ 4 ) , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 12 Let (cid:99)
Revised: Lemma 12 Let (cid:99)

Original: M 1 be the empirical version of M 1 using dataset S .
Revised: M 1 be the empirical version of M 1 using dataset S .

Original: Then with probability at least 1 − 2 n − Ω( d ) , we have
Revised: Then with probability at least 1 − 2 n − Ω( d ) , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 13 ((Zhong et al., 2017b), Lemma E.6)
Revised: Lemma 13 ((Zhong et al., 2017b), Lemma E.6)

Original: Let P 2 be deﬁned in Deﬁnition 1 and (cid:98) P 2 be its empirical version.
Revised: Let P 2 be deﬁned in Deﬁnition 1 and (cid:98) P 2 be its empirical version.

Original: Let
Revised: Let

Original: U ∈ R
Revised: U ∈ R

Original: d × K be the column span of W ∗ .
Revised: d × K be the column span of W ∗ .

Original: Assume || P 2 − (cid:98) P 2 || ≤ δ K
Revised: Assume || P 2 − (cid:98) P 2 || ≤ δ K

Original: ( P 2 ).
Revised: ( P 2 ).

Original: Then after T = O (log( 1 (cid:15) )) iterations, the output of the Tensor Initialization Method 3, (cid:98) U will satisfy
Revised: Then after T = O (log( 1 (cid:15) )) iterations, the output of the Tensor Initialization Method 3, (cid:98) U will satisfy

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Lemma 14 ((Zhong et al., 2017b), Lemma E.13) Let U ∈ R
Revised: Lemma 14 ((Zhong et al., 2017b), Lemma E.13) Let U ∈ R

Original: d
Revised: d

Original: × K be the orthogonal column span of W ∗ .
Revised: × K be the orthogonal column span of W ∗ .

Original: Let (cid:98) U ∈ R
Revised: Let (cid:98) U ∈ R

Original: d
Revised: d

Original: × K be an orthogonal matrix such that || UU (cid:62) − (cid:98)
Revised: × K be an orthogonal matrix such that || UU (cid:62) − (cid:98)

Original: U (cid:98) U
Revised: U (cid:98) U

Original: (cid:62)
Revised: (cid:62)

Original: ||
Revised: ||

Original: (cid:46) γ 1 (cid:46) 1 κ 2 √ K .
Revised: (cid:46) γ 1 (cid:46) 1 κ 2 √ K .

Original: For each i ∈ [ K ] , let (cid:98) v
Revised: For each i ∈ [ K ] , let (cid:98) v

Original: i denote the vector satisfying || (cid:98) v
Revised: i denote the vector satisfying || (cid:98) v

Original: i − (cid:98) U (cid:62) ¯ w i ∗ || ≤ γ 

Revised: i − (cid:98) U (cid:62) ¯ w i ∗ || ≤ γ

Original: (cid:46) 1 κ 2 √ K .
Revised: (cid:46) 1 κ 2 √ K .

Original: Let M 1 be deﬁned in Lemma 12 and (cid:99) M 1 be its empirical version.
Revised: Let M 1 be deﬁned in Lemma 12 and (cid:99) M 1 be its empirical version.

Original: If || M 1 − (cid:99) M 1 || ≤ γ 3 || M 

Revised: If || M 1 − (cid:99) M 1 || ≤ γ 3 || M

Original: ||
Revised: ||

Original: (cid:46)
Revised: (cid:46)

Original: 14 || M 

Revised: 14 || M

Original: || , then we have
Revised: || , then we have

Original: [Equation]
Revised: [Equation]

Original: P
Revised: P

Original: L EMMA
Revised: L EMMA

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: j ∗ − (cid:98) v j (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:17) + (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ||
Revised: j ∗ − (cid:98) v j (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:17) + (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ||

Original: w ∗ j || − (cid:98) α
Revised: w ∗ j || − (cid:98) α

Original: j
Revised: j

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (174)
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (174)

Original: By Lemma 10, Lemma 13 and δ K ( P 2 ) (cid:46) δ 2 K , we have
Revised: By Lemma 10, Lemma 13 and δ K ( P 2 ) (cid:46) δ 2 K , we have

Original: [Equation]
Revised: [Equation]

Original: Moreover, we have
Revised: Moreover, we have

Original: [Equation]
Revised: [Equation]

Original: in which the ﬁrst step is by Theorem 3 in [Kuleshov et al.
Revised: in which the ﬁrst step is by Theorem 3 in [Kuleshov et al.

Original: and the second step is by Lemma 11.
Revised: and the second step is by Lemma 11.

Original: By Lemma 14, we have
Revised: By Lemma 14, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, taking the union bound of failure probabilities in Lemmas 10, 11 and 12 and by
Revised: Therefore, taking the union bound of failure probabilities in Lemmas 10, 11 and 12 and by

Original: D 2 ( λ , M , σ ) D 4 ( λ , M , σ ) ≤ D 6 ( λ , M , σ ) from Property 6, we have that if the sample size n ≥ κ 8 K 4 τ 12 D 6 ( λ , M , σ ) · d log 2 d , then the output W 0 ∈ R d × K satisﬁes
Revised: D 2 ( λ , M , σ ) D 4 ( λ , M , σ ) ≤ D 6 ( λ , M , σ ) from Property 6, we have that if the sample size n ≥ κ 8 K 4 τ 12 D 6 ( λ , M , σ ) · d log 2 d , then the output W 0 ∈ R d × K satisﬁes

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − n − Ω( δ 41 )
Revised: with probability at least 1 − n − Ω( δ 41 )

Original: E. P ROOF OF L EMMA
Revised: E. P ROOF OF L EMMA

Original: From Assumption 1, if the Gaussian Mixture Model is a symmetric probability distribution deﬁned in (8), then P 2 = M 3 ( I , I , α ) .
Revised: From Assumption 1, if the Gaussian Mixture Model is a symmetric probability distribution deﬁned in (8), then P 2 = M 3 ( I , I , α ) .

Original: Therefore, by Deﬁnition 1, we have
Revised: Therefore, by Deﬁnition 1, we have

Original: [Equation]
Revised: [Equation]

Original: Following (Zhong et al., 2017b), (cid:101) ⊗ is deﬁned such that for any v ∈ R d 1 and Z ∈ R d 1 × d 2 ,
Revised: Following (Zhong et al., 2017b), (cid:101) ⊗ is deﬁned such that for any v ∈ R d 1 and Z ∈ R d 1 × d 2 ,

Original: [Equation]
Revised: [Equation]

Original: where z i is the i -th column of Z .
Revised: where z i is the i -th column of Z .

Original: By Deﬁnition
Revised: By Deﬁnition

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst step of (181) is because ( x − µ l σ 2 l ) ⊗ 2 ( α (cid:62) ( x − µ l σ 2 l )) is the dominant term of the entire expression, and y ≤ 1 .
Revised: The ﬁrst step of (181) is because ( x − µ l σ 2 l ) ⊗ 2 ( α (cid:62) ( x − µ l σ 2 l )) is the dominant term of the entire expression, and y ≤ 1 .

Original: The second step is because the expression can be considered as a normalized weighted summation of ( x − µ l σ 2 l ) ⊗ 2 ( α (cid:62) ( x − µ l σ 2 l )) and ( x (cid:62) i α ) x i x (cid:62) i is its dominant term.
Revised: The second step is because the expression can be considered as a normalized weighted summation of ( x − µ l σ 2 l ) ⊗ 2 ( α (cid:62) ( x − µ l σ 2 l )) and ( x (cid:62) i α ) x i x (cid:62) i is its dominant term.

Original: Deﬁne
Revised: Deﬁne

Original: S m ( x ) =
Revised: S m ( x ) =

Original: ( − 1) m ∇ m x p ( x ) p ( x ) , where p ( x ) is the probability density function of the random variable x .
Revised: ( − 1) m ∇ m x p ( x ) p ( x ) , where p ( x ) is the probability density function of the random variable x .

Original: From Deﬁnition 1, we can verify that
Revised: From Deﬁnition 1, we can verify that

Original: [Equation]
Revised: [Equation]

Original: Then deﬁne Gp i = (cid:10) v , ([ y i · S 3 ( x i )]
Revised: Then deﬁne Gp i = (cid:10) v , ([ y i · S 3 ( x i )]

Original: ( I d , I d , α ) − E (cid:2) [ y i · S 3 ( x i )]
Revised: ( I d , I d , α ) − E (cid:2) [ y i · S 3 ( x i )]

Original: ( I d , I d , α ) (cid:3) v ) (cid:11) , where || v
Revised: ( I d , I d , α ) (cid:3) v ) (cid:11) , where || v

Original: ||
Revised: ||

Original: = 1 , then E
Revised: = 1 , then E

Original: [ Gp i ] = 0 .
Revised: [ Gp i ] = 0 .

Original: Similar to the proof of (130), (131) and (132) in Lemma 8, we have
Revised: Similar to the proof of (130), (131) and (132) in Lemma 8, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Hence, similar to the derivation of (134)
Revised: Hence, similar to the derivation of (134)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: P (cid:16)
Revised: P (cid:16)

Original: [Equation]
Revised: [Equation]

Original: for some constant C 16 > 0 .
Revised: for some constant C 16 > 0 .

Original: Let θ =
Revised: Let θ =

Original: [Equation]
Revised: [Equation]

Original: M
Revised: M

Original: [Equation]
Revised: [Equation]

Original: (cid:1) 2 and t =
Revised: (cid:1) 2 and t =

Original: δ 21 · (cid:0) τ
Revised: δ 21 · (cid:0) τ

Original: [Equation]
Revised: [Equation]

Original: D
Revised: D

Original: [Equation]
Revised: [Equation]

Original: D
Revised: D

Original: [Equation]
Revised: [Equation]

Original: M , σ )
Revised: M , σ )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: d log n n , then we have
Revised: d log n n , then we have

Original: [Equation]
Revised: [Equation]

Original: n n
Revised: n n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: with probability at least 1 − 2 n − Ω( δ 41 d ) .
Revised: with probability at least 1 − 2 n − Ω( δ 41 d ) .

Original: If the Gaussian Mixture Model is not a symmetric distribution which is deﬁned in (8), then
Revised: If the Gaussian Mixture Model is not a symmetric distribution which is deﬁned in (8), then

Original: P 2 = M 2 .
Revised: P 2 = M 2 .

Original: We would have a similar result as follows.
Revised: We would have a similar result as follows.

Original: [Equation]
Revised: [Equation]

Original: Then deﬁne Gp (cid:48) i
Revised: Then deﬁne Gp (cid:48) i

Original: = (cid:10) v , ([ y i · S 2 ( x i )]
Revised: = (cid:10) v , ([ y i · S 2 ( x i )]

Original: − E (cid:2)
Revised: − E (cid:2)

Original: y i · S 2 ( x i ) (cid:3) v ) (cid:11) ,
Revised: y i · S 2 ( x i ) (cid:3) v ) (cid:11) ,

Original: where || v ||
Revised: where || v ||

Original: = 1 , then E
Revised: = 1 , then E

Original: [ Gp (cid:48) i ]
Revised: [ Gp (cid:48) i ]

Original: = 0 .
Revised: = 0 .

Original: Similar to the proof of (130), (131) and (132) in Lemma 8, we have
Revised: Similar to the proof of (130), (131) and (132) in Lemma 8, we have

Original: [Equation]
Revised: [Equation]

Original: Hence, similar to the derivation of (134)
Revised: Hence, similar to the derivation of (134)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: for some constant C 17 > 0 .
Revised: for some constant C 17 > 0 .

Original: Let θ =
Revised: Let θ =

Original: [Equation]
Revised: [Equation]

Original: then we have
Revised: then we have

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − 2 n − Ω(
Revised: with probability at least 1 − 2 n − Ω(

Original: [Equation]
Revised: [Equation]

Original: d ) .
Revised: d ) .

Original: To sum up, from (187) and (194) we have
Revised: To sum up, from (187) and (194) we have

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − 2 n − Ω( δ 41 d ) .
Revised: with probability at least 1 − 2 n − Ω( δ 41 d ) .

Original: E. P ROOF OF L EMMA
Revised: E. P ROOF OF L EMMA

Original: We consider each component of y = 1 K (cid:80) Ki =1 φ ( w ∗
Revised: We consider each component of y = 1 K (cid:80) Ki =1 φ ( w ∗

Original: i
Revised: i

Original: (cid:62) x ) .
Revised: (cid:62) x ) .

Original: Deﬁne T i ( x ) : R d → R K × K × K such that
Revised: Deﬁne T i ( x ) : R d → R K × K × K such that

Original: [Equation]
Revised: [Equation]

Original: We ﬂatten T i ( x ) :
Revised: We ﬂatten T i ( x ) :

Original: R d →
Revised: R d →

Original: R K × K × K along the ﬁrst dimension to obtain function
Revised: R K × K × K along the ﬁrst dimension to obtain function

Original: B i ( x ) : R d → R K × K 2 .
Revised: B i ( x ) : R d → R K × K 2 .

Original: Similar to the derivation of the last step of Lemma E.8 in (Zhong et al., 2017b), we can obtain (cid:107) T i ( x ) (cid:107) ≤ (cid:107) B i ( x ) (cid:107) .
Revised: Similar to the derivation of the last step of Lemma E.8 in (Zhong et al., 2017b), we can obtain (cid:107) T i ( x ) (cid:107) ≤ (cid:107) B i ( x ) (cid:107) .

Original: By (179), we have
Revised: By (179), we have

Original: [Equation]
Revised: [Equation]

Original: Deﬁne Gr i = (cid:104) v , B i ( x i ))
Revised: Deﬁne Gr i = (cid:104) v , B i ( x i ))

Original: − E [ B i ( x i )] v ) (cid:105) , where || v ||
Revised: − E [ B i ( x i )] v ) (cid:105) , where || v ||

Original: = 1 , so E
Revised: = 1 , so E

Original: [ Gr i ] = 0 .
Revised: [ Gr i ] = 0 .

Original: Similar to the proof of (130), (131) and (132) in Lemma
Revised: Similar to the proof of (130), (131) and (132) in Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Hence, similar to the derivation of (134), we have
Revised: Hence, similar to the derivation of (134), we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for some constant C 18 > 0 .
Revised: for some constant C 18 > 0 .

Original: Let θ =
Revised: Let θ =

Original: t
Revised: t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: τ 6 √ D
Revised: τ 6 √ D

Original: [Equation]
Revised: [Equation]

Original: λ , M ,
Revised: λ , M ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2 and t = δ 21 · (cid:0) τ
Revised: 2 and t = δ 21 · (cid:0) τ

Original: [Equation]
Revised: [Equation]

Original: D
Revised: D

Original: [Equation]
Revised: [Equation]

Original: λ , M , σ ) (cid:1) · (cid:113) log nn , then we have
Revised: λ , M , σ ) (cid:1) · (cid:113) log nn , then we have

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − 2 n − Ω( δ 41 ) .
Revised: with probability at least 1 − 2 n − Ω( δ 41 ) .

Original: E.
Revised: E.

Original: P ROOF OF L EMMA
Revised: P ROOF OF L EMMA

Original: From the Deﬁnition 1, we have
Revised: From the Deﬁnition 1, we have

Original: [Equation]
Revised: [Equation]

Original: Based on Deﬁnition 1,
Revised: Based on Deﬁnition 1,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Deﬁne Gq
Revised: Deﬁne Gq

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: v
Revised: v

Original: [Equation]
Revised: [Equation]

Original: i ·
Revised: i ·

Original: [Equation]
Revised: [Equation]

Original: x i )] − E
Revised: x i )] − E

Original: [Equation]
Revised: [Equation]

Original: [ y
Revised: [ y

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: , where || v ||
Revised: , where || v ||

Original: = 1 , so E
Revised: = 1 , so E

Original: [ Gq i ] = 0 .
Revised: [ Gq i ] = 0 .

Original: Similar to the proof of (130), (131) and (132) in Lemma 8, we have
Revised: Similar to the proof of (130), (131) and (132) in Lemma 8, we have

Original: [Equation]
Revised: [Equation]

Original: Hence, similar to the derivation of (134), we have
Revised: Hence, similar to the derivation of (134), we have

Original: [Equation]
Revised: [Equation]

Original: for some constant C 19 > 0 .
Revised: for some constant C 19 > 0 .

Original: Let θ =
Revised: Let θ =

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: then we have
Revised: then we have

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − 2 n − Ω( d ) .
Revised: with probability at least 1 − 2 n − Ω( d ) .