Original: N EURAL
Revised: N EURAL

Original: N ETWORKS
Revised: N ETWORKS

Original: FOR
Revised: FOR

Original: P RINCIPAL
Revised: P RINCIPAL

Original: C OMPONENT
Revised: C OMPONENT

Original: A NALYSIS
Revised: A NALYSIS

Original: A
Revised: A

Original: N EW
Revised: N EW

Original: L OSS
Revised: L OSS

Original: F UNCTION
Revised: F UNCTION

Original: P ROVABLY
Revised: P ROVABLY

Original: Y IELDS
Revised: Y IELDS

Original: O RDERED
Revised: O RDERED

Original: E XACT
Revised: E XACT

Original: E
Revised: E

Original: IGENVECTORS
Revised: IGENVECTORS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper we propose a new loss function for performing principal component analysis (PCA) using linear autoencoders (LAEs).
Revised: In this paper we propose a new loss function for performing principal component analysis (PCA) using linear autoencoders (LAEs).

Original: Optimizing the standard Lloss results in a decoder matrix that spans the principal subspace of the sample covariance of the data but fails to identify the exact eigenvectors.
Revised: Optimizing the standard Lloss results in a decoder matrix that spans the principal subspace of the sample covariance of the data but fails to identify the exact eigenvectors.

Original: This downside originates from an invariance that cancels out in the global map.
Revised: This downside originates from an invariance that cancels out in the global map.

Original: Here, we prove that our loss function eliminates this issue, i.e. the decoder converges to the exact ordered unnormalized eigenvectors of the sample covariance matrix.
Revised: Here, we prove that our loss function eliminates this issue, i.e. the decoder converges to the exact ordered unnormalized eigenvectors of the sample covariance matrix.

Original: For this new loss, we establish that all local minima are global optima and also show that computing the new loss (and also its gradients) has the same order of complexity as the classical loss.
Revised: For this new loss, we establish that all local minima are global optima and also show that computing the new loss (and also its gradients) has the same order of complexity as the classical loss.

Original: We report numerical results on both synthetic simulations, and a real-data PCA experiment on MNIST (i e., a 60 , 000 × 784 matrix), demonstrating our approach to be practically applicable and rectify previous LAEs’ downsides
Revised: We report numerical results on both synthetic simulations, and a real-data PCA experiment on MNIST (i e., a 60 , 000 × 784 matrix), demonstrating our approach to be practically applicable and rectify previous LAEs’ downsides

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Ranking among the most widely-used and valuable statistical tools, Principal Component Analysis (PCA) represents a given set of data within a new orthogonal coordinate system in which the data are uncorrelated and the variance of the data along each orthogonal axis is successively ordered from the highest to lowest.
Revised: Ranking among the most widely-used and valuable statistical tools, Principal Component Analysis (PCA) represents a given set of data within a new orthogonal coordinate system in which the data are uncorrelated and the variance of the data along each orthogonal axis is successively ordered from the highest to lowest.

Original: The projection of data along each axis gives what are called principal components.
Revised: The projection of data along each axis gives what are called principal components.

Original: Theoretically, eigendecomposition of the covariance matrix provides exactly such a transformation.
Revised: Theoretically, eigendecomposition of the covariance matrix provides exactly such a transformation.

Original: For large data sets, however, classical decomposition techniques are infeasible and other numerical methods, such as least squares approximation schemes, are practically employed.
Revised: For large data sets, however, classical decomposition techniques are infeasible and other numerical methods, such as least squares approximation schemes, are practically employed.

Original: An especially notable instance is the problem of dimensionality reduction, where only the largest principal components—as the best representative of the data—are desired.
Revised: An especially notable instance is the problem of dimensionality reduction, where only the largest principal components—as the best representative of the data—are desired.

Original: Linear autoencoders (LAEs) are one such scheme for dimensionality reduction that is applicable to large data sets.
Revised: Linear autoencoders (LAEs) are one such scheme for dimensionality reduction that is applicable to large data sets.

Original: An LAE with a single fully-connected and linear hidden layer, and mean squared error loss function can discover the linear subspace spanned by the principal components.
Revised: An LAE with a single fully-connected and linear hidden layer, and Mean Squared Error (MSE) loss function can discover the linear subspace spanned by the principal components.

Original: This subspace is the same as the one spanned by the weights of the decoder.
Revised: This subspace is the same as the one spanned by the weights of the decoder.

Original: However, it faila to identify the exact principal directions.
Revised: However, it failure to identify the exact principal directions.

Original: This is due to the fact that, when the encoder is transformed by some matrix, transforming the decoder by the inverse of that matrix will yield no change in the loss.
Revised: This is due to the fact that, when the encoder is transformed by some matrix, transforming the decoder by the inverse of that matrix will yield no change in the loss.

Original: In other words, the loss possesses a symmetry under the action of a group of invertible matrices, so that directions (and orderings/permutations thereto) will not be discriminated.
Revised: In other words, the loss possesses a symmetry under the action of a group of invertible matrices, so that directions (and orderings/permutations thereto) will not be discriminated.

Original: The early work of Bourlard & Kamp (1988) and Baldi & Hornik (1989) connected LAEs and PCA and demonstrated the lack of identiﬁability of principal components.
Revised: The early work of Bourlard & Kamp (1988) and Baldi & Hornik (1989) connected LAEs and PCA and demonstrated the lack of identiﬁability of principal components.

Original: Several methods for neural networks compute the exact eigenvectors (Rubner & Tavan, 1989; Xu, 1993; Kung & Diamantaras, 1990; Oja et al., 1992), but they depend on either particular network structures or special optimization methods.
Revised: Several methods for neural networks compute the exact eigenvectors (Rubner & Tavan, 1989; Xu, 1993; Kung & Diamantaras, 1990; Oja et al., 1992), but they depend on either particular network structures or special optimization methods.

Original: It was recently observed (Plaut, 2018; Kunin et al., 2019) that regularization causes the left singular vectors of the decoder to become the exact eigenvectors, but recovering them still requires an extra decomposition step.
Revised: It was recently observed (Plaut, 2018; Kunin et al., 2019) that regularization causes the left singular vectors of the decoder to become the exact eigenvectors, but recovering them still requires an extra decomposition step.

Original: As Plaut (2018) point out, no existent method recovers the eigenvectors from an LAE in an optimization-independent way on a standard network — this work ﬁlls that void.
Revised: As Plaut (2018) point out, no existent method recovers the eigenvectors from an LAE in an optimization-independent way on a standard network — this work ﬁlls that void.

Revised: Moreover, analyzing the loss surface for various architectures of linear/non-linear neural networks is a highly active and prominent area of research (e.g. Baldi & Hornik (1989); Kunin et al.

Revised: Pretorius et al.

Revised: ; Frye et al.

Revised: Most of these works extend the results of Baldi & Hornik (1989) for shallow LAEs to more complex networks.

Original:  In  this paper, we  propose a new loss function for performing PCA using LAEs.
Revised: However, most retain the original MSE loss, and they prove the same critical point characterization for their speciﬁc architecture of interest. Most notably Zhou & Liang (2018) extends the results of Baldi & Hornik (1989) to deep linear networks and shallow RELU networks. In contrast in this work we are going after a loss with better loss surface properties. We propose a new loss function for performing PCA using LAEs.

Original: We show that with the proposed loss function, the decoder converges to the exact ordered unnormalized eigenvectors of the sample covariance matrix.
Revised: We show that with the proposed loss function, the decoder converges to the exact ordered unnormalized eigenvectors of the sample covariance matrix.

Original: The idea is simple: for identifying p principal directions we build up a total loss function as a sum of p squared error losses, where the i th loss function identiﬁes only the ﬁrst i principal directions.
Revised: The idea is simple: for identifying p principal directions we build up a total loss function as a sum of p squared error losses, where the i th loss function identiﬁes only the ﬁrst i principal directions.

Original: This approach breaks the symmetry since minimizing the ﬁrst loss results in the ﬁrst principal direction, which forces the second loss to ﬁnd the ﬁrst and the second.
Revised: This approach breaks the symmetry since minimizing the ﬁrst loss results in the ﬁrst principal direction, which forces the second loss to ﬁnd the ﬁrst and the second.

Original: This constraint is propagated through the rest of the losses, resulting in all p principal components being identiﬁed.
Revised: This constraint is propagated through the rest of the losses, resulting in all p principal components being identiﬁed.

Original: For the new loss we prove that all local minima are global minima.
Revised: For the new loss we prove that all local minima are global minima.

Original: Moreover, we show that  the  loss  and its gradients can be compactly vectorized so that their computational complexity is no different from the classic L 2 loss.
Revised: Consequently, the proposed loss function has both theoretical and practical implications. Theoretically, it provides better understanding of the loss surface. Speciﬁcally, we show that any critical point of our loss L is a critical point of the original MSE loss but not vice versa, and conclude that L eliminates those undesirable global minima of the original loss (i.e., exactly those which suffer from the invariance). Given that the set of critical points of L is a subset of critical points of MSE loss, many of the previous work on loss surfaces of more complex networks likely extend.

Revised: In light of the removal of undesirable global minima through L , examining more complex networks is certainly a very promising direction.

Original: We report the numerical results to show the advantages.
Revised: As for practical consequences, we show that the loss and its gradients can be compactly vectorized so that their computational complexity is no different from the MSE loss. Therefore, the loss L can be used to perform PCA/SVD on large datasets using any method of optimization such as Stochastic Gradient Descent (SGD). Chief among the compellingly reasons to perform PCA/SVD using this method is that, in recent years, there has been unprecedented gains in the performance of very large SGD optimizations, with autoencoders in particular successfully handling larger numbers of high-dimensional training data (e.g., images). The loss function we offer is attractive in terms of parallelizability and distributability, and does not prescribe any single speciﬁc algorithm or implementation, so stands to continue to beneﬁt from the arms race between SGD and its competitors.

Revised: More importantly, this single loss function (without an additional post hoc processing step) ﬁts seamlessly into optimization pipelines (where SGD is but one instance).

Revised: The result is that the loss allows for PCA/SVD computation as single optimization layer, akin to an instance of a fully differentiable building block in a NN pipeline Amos & Kolter (2017), potentially as part of a much larger network.

Original:  T HE P ROPOSED L OSS F UNCTION AND R EVIEW OF F INAL R ESULTS
Revised:  T HE P ROPOSED L OSS F UNCTION AND R EVIEW OF F INAL R ESULTS

Original: Let X ∈ R n × m and Y ∈ R n × m be the input and output matrices, where m centered sample points, each n -dimensional, are stacked column-wise.
Revised: Let X ∈ R n × m and Y ∈ R n × m be the input and output matrices, where m centered sample points, each n -dimensional, are stacked column-wise.

Original: Let x j ∈ R n and y j ∈ R n be the j th sample input and output (i.e. the j th column of X and Y , respectively).
Revised: Let x j ∈ R n and y j ∈ R n be the j th sample input and output (i.e. the j th column of X and Y , respectively).

Original: Deﬁne the loss function
Revised: Deﬁne the loss function

Original: L ( A , B ) as
Revised: L ( A , B ) as

Original: [Equation]
Revised: [Equation]

Original:  where (cid:104)· , ·(cid:105) F and (cid:107)·(cid:107) F are the Frobenius inner product and norm, I i ; p is a p × p matrix with all elements zero except the ﬁrst i diagonal elements being one.
Revised: [Equation] 1) where (cid:104)· , ·(cid:105) F and (cid:107)·(cid:107) F are the Frobenius inner product and norm, I i ; p is a p × p matrix with all ele- ments zero except the ﬁrst i diagonal elements being one.

Original: (Or, equivalently, the matrix obtained by setting the last p − i diagonal elements of a p × p identity matrix to zero, e.g. I 2;3 = (cid:34) 1 00 10 0(cid:35) .)
Revised: (Or, equivalently, the matrix obtained by setting the last p − i diagonal elements of a p × p identity matrix to zero, e.g. I 2;3 = (cid:34) 1 00 10 0(cid:35) .)

Original: In what follows, we shall denote the transpose of matrix M by M (cid:48) .
Revised: In what follows, we shall denote the transpose of matrix M by M (cid:48) .

Revised: Moreover, the matrices A ∈ R n × p , and B ∈ R p × n can be viewed as the weights of the decoder and encoder parts of an LAE.

Original: The results are based on the following standard assumptions that hold generically:
Revised: The results are based on the following standard assumptions that hold generically:

Original: Assumption 1.
Revised: Assumption 1.

Original: For an input X and an output Y , let Σ xx :
Revised: For an input X and an output Y , let Σ xx :

Original: = XX (cid:48) ,
Revised: = XX (cid:48) ,

Original: Σ xy :=
Revised: Σ xy :=

Original: XY (cid:48) , Σ yx := Σ (cid:48) xy and Σ
Revised: XY (cid:48) , Σ yx := Σ (cid:48) xy and Σ

Original: yy =
Revised: yy =

Original: Y Y (cid:48) be their sample covariance matrices.
Revised: Y Y (cid:48) be their sample covariance matrices.

Original: We assume
Revised: We assume

Original: • The input and output data are centered (zero mean).
Revised: • The input and output data are centered (zero mean).

Original: • Σ xx , Σ xy , Σ yx and Σ yy are positive deﬁnite (of full rank and invertible).
Revised: • Σ xx , Σ xy , Σ yx and Σ yy are positive deﬁnite (of full rank and invertible).

Original: • The covariance matrix Σ := Σ yx Σ − 1 xx Σ xy is of full rank with n distinct eigenvalues λ 1 > λ 2 > · · · > λ n .
Revised: • The covariance matrix Σ := Σ yx Σ − 1 xx Σ xy is of full rank with n distinct eigenvalues λ 1 > λ 2 > · · · > λ n .

Original: • The decoder matrix
Revised: • The decoder matrix

Original: A has no zero columns.
Revised: A has no zero columns.

Original: Claim.
Revised: Claim.

Original: The main result of this work proved in Theorem 2 is as follows:
Revised: The main result of this work proved in Theorem 2 is as follows:

Original: If the above assumptions hold then all the local minima of L ( A , B ) are achieved iff A and B are of the form
Revised: If the above assumptions hold then all the local minima of L ( A , B ) are achieved iff A and B are of the form

Original: [Equation]
Revised: [Equation]

Original: where the i th column of U 1: p is the unit eigenvector of Σ := Σ yx Σ − 1 xx Σ xy corresponding to the i th largest eigenvalue and D p is a diagonal matrix with nonzero diagonal elements.
Revised: where the i th column of U 1: p is the unit eigenvector of Σ := Σ yx Σ − 1 xx Σ xy corresponding to the i th largest eigenvalue and D p is a diagonal matrix with nonzero diagonal elements.

Original: In other words,
Revised: In other words,

Original: A contains ordered unnormalized eigenvectors of Σ corresponding to the p largest eigenvalues.
Revised: A contains ordered unnormalized eigenvectors of Σ corresponding to the p largest eigenvalues.

Original: Moreover, all the local minima are global minima with the value of the loss function at those global minima being
Revised: Moreover, all the local minima are global minima with the value of the loss function at those global minima being

Original: [Equation]
Revised: [Equation]

Original: where λ i is the i th largest eigenvalue of Σ := Σ yx Σ − 1 xx Σ xy .
Revised: where λ i is the i th largest eigenvalue of Σ := Σ yx Σ − 1 xx Σ xy .

Original: In the case of autoencoder ( Y = X ): Σ = Σ xx .
Revised: In the case of autoencoder ( Y = X ): Σ = Σ xx .

Original: Finally, while L ( A , B ) in the given form contains O ( p ) matrix products, we will show that it can be evaluated with constant (less than 7) matrix products independent of the value p .
Revised: Finally, while L ( A , B ) in the given form contains O ( p ) matrix products, we will show that it can be evaluated with constant (less than 7) matrix products independent of the value p .

Original:  N OTATION
Revised:  N OTATION

Original: In this paper, the underlying ﬁeld is always R , and positive semideﬁnite matrices are symmetric by deﬁnition.
Revised: In this paper, the underlying ﬁeld is always R , and positive semideﬁnite matrices are symmetric by deﬁnition.

Original: The following constant matrices are used extensively throughout.
Revised: The following constant matrices are used extensively throughout.

Original: The matrices T p ∈
Revised: The matrices T p ∈

Original: R p × p and S p ∈ R p × p are deﬁned as
Revised: R p × p and S p ∈ R p × p are deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Another matrix that will appear in the formulation is ˆ S p :
Revised: Another matrix that will appear in the formulation is ˆ S p :

Original: = T − 1 p S p T − 1 p .
Revised: = T − 1 p S p T − 1 p .

Original: Clearly, the diagonal matrix T p is positive deﬁnite.
Revised: Clearly, the diagonal matrix T p is positive deﬁnite.

Original: As shown in Lemma 2, S p and ˆ S p are positive deﬁnite as well.
Revised: As shown in Lemma 2, S p and ˆ S p are positive deﬁnite as well.

Original:  M AIN T HEOREMS
Revised:  M AIN T HEOREMS

Revised: The general strategy to prove the above claim is as follows.

Original: Let ˜ L
Revised: First the analytical gradients of the loss is derived in a matrix form in Propositions 1 and 2. We compare the gradients with that of the original Minimum Square Error (MSE) loss. Next, we analyze the loss surface by solving the gradient equations which yields the general structure of critical points based on the rank of the decoder matrix A . Next, we delineate several interesting properties of the critical points, notably, any critical point of the loss is also a critical point for the MSE loss but not the other way around.

Revised: Finally, by performing second order analysis on the loss in Theorem 2 the exact equations for local minima are derived which is shown to be as claimed.

Original:  ( A , B ) and L ( A , B ) be the original loss, and the proposed loss function, respectively, i.e., ˜ L ( A , B ) : = m
Revised: Let ˜ L ( A , B ) and L ( A , B ) be the original loss, and the proposed loss function, respectively, i.e., ˜ L ( A , B ) :=

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst step is to calculate the gradients with respect to A and B and set them to zero to derive the implicit expressions for the critical points.
Revised: The ﬁrst step is to calculate the gradients with respect to A and B and set them to zero to derive the implicit expressions for the critical points.

Original: In order to do so, ﬁrst, in Lemma 5, for a ﬁxed A , we derive the directional (Gateaux) derivative of the loss with respect to B along an arbitrary direction
Revised: In order to do so, ﬁrst, in Lemma 5, for a ﬁxed A , we derive the directional (Gateaux) derivative of the loss with respect to B along an arbitrary direction

Original: W ∈ R p × n , denoted as d B L ( A , B ) W , i.e.
Revised: W ∈ R p × n , denoted as d B L ( A , B ) W , i.e.

Original: [Equation]
Revised: [Equation]

Original: As shown in the proof of the lemma, d B L ( A , B ) W is derived by writing the norm in the loss as an inner product, opening it up using linearity of inner product, dismiss second order terms in W (i.e.
Revised: As shown in the proof of the lemma, d B L ( A , B ) W is derived by writing the norm in the loss as an inner product, opening it up using linearity of inner product, dismiss second order terms in W (i.e.

Original: O ( (cid:107) W (cid:107) 2 ) )
Revised: O ( (cid:107) W (cid:107) 2 ) )

Original: and rearrange the result as the inner product between the gradient with respect to B , and the direction W , which yields
Revised: and rearrange the result as the inner product between the gradient with respect to B , and the direction W , which yields

Original: [Equation]
Revised: [Equation]

Original: where, ◦ is the Hadamard product and the constant matrices T p and S p , were deﬁned in the beginning.
Revised: where, ◦ is the Hadamard product and the constant matrices T p and S p , were deﬁned in the beginning.

Original: Second, the same process is done in Lemma 6, to derive d A L ( A , B ) V ; the derivative of
Revised: Second, the same process is done in Lemma 6, to derive d A L ( A , B ) V ; the derivative of

Original: L with respect to A in an arbitrary direction V ∈ R n × p , for a ﬁxed B , which is then set to zero to derive the implicit expressions for the critical points.
Revised: L with respect to A in an arbitrary direction V ∈ R n × p , for a ﬁxed B , which is then set to zero to derive the implicit expressions for the critical points.

Original: The results are formally stated in the two following propositions.
Revised: The results are formally stated in the two following propositions.

Original: Proposition 1.
Revised: Proposition 1.

Original: For any ﬁxed matrix A ∈ R n × p the function L ( A , B ) is convex in the coefﬁcients of B and attains its minimum for any B satisfying the equation
Revised: For any ﬁxed matrix A ∈ R n × p the function L ( A , B ) is convex in the coefﬁcients of B and attains its minimum for any B satisfying the equation

Original: [Equation]
Revised: [Equation]

Original: where ◦ is the Hadamard (element-wise) product operator, and S p and T p are constant matrices deﬁned in the previous section.
Revised: where ◦ is the Hadamard (element-wise) product operator, and S p and T p are constant matrices deﬁned in the previous section.

Original: Further, if A has no zero column, then L ( A , B ) is strictly convex in
Revised: Further, if A has no zero column, then L ( A , B ) is strictly convex in

Original: B and has a unique minimum when the critical B is
Revised: B and has a unique minimum when the critical B is

Original: [Equation]
Revised: [Equation]

Original: and in the autoencoder case it becomes
Revised: and in the autoencoder case it becomes

Original: [Equation]
Revised: [Equation]

Original: The proof is given in appendix A.2.
Revised: The proof is given in appendix A.2.

Original: Remark 1 .
Revised: Remark 1 .

Original: Note that as long as A has no zero column, S p ◦ ( A (cid:48)
Revised: Note that as long as A has no zero column, S p ◦ ( A (cid:48)

Original: A ) is nonsingular (we will explain the reason soon).
Revised: A ) is nonsingular (we will explain the reason soon).

Original: In practice, A with zero columns can always be avoided by nudging the zero columns of A during the gradient decent process.
Revised: In practice, A with zero columns can always be avoided by nudging the zero columns of A during the gradient decent process.

Original: Proposition 2.
Revised: Proposition 2.

Original: For any ﬁxed matrix B ∈ R p × n the function
Revised: For any ﬁxed matrix B ∈ R p × n the function

Original: L ( A , B ) is a convex function in A .
Revised: L ( A , B ) is a convex function in A .

Original: Moreover, for a ﬁxed B , the matrix A that satisﬁes
Revised: Moreover, for a ﬁxed B , the matrix A that satisﬁes

Original: [Equation]
Revised: [Equation]

Original: is a critical point of L ( A , B ) .
Revised: is a critical point of L ( A , B ) .

Original: The proof is given in appendix A.3.
Revised: The proof is given in appendix A.3.

Original: The pair ( A , B ) is a critical point of L if they make d B L ( A , B ) W and d A L ( A , B ) V zero for any pair of directions ( V , W ) .
Revised: The pair ( A , B ) is a critical point of L if they make d B L ( A , B ) W and d A L ( A , B ) V zero for any pair of directions ( V , W ) .

Original: Therefore, the implicit equations for critical points are given below, next to the ones derived by Baldi & Hornik (1989) for ˜ L ( A , B ) .
Revised: Therefore, the implicit equations for critical points are given below, next to the ones derived by Baldi & Hornik (1989) for ˜ L ( A , B ) .

Original: For ˜ L ( A , B 
Revised: For ˜ L ( A , B ) :

Original: [Equation]
Revised: [Equation]

Original: [Equation] For L ( A , B [Equation] [Equation]
Revised: Remark 2 .

Original: Notice the similarity, and the difference only being the presence of Hadamard product by S p in the left and by diagonal T p in the right.
Revised: Notice the similarity, and the difference only being the presence of Hadamard product by S p in the left and by diagonal T p in the right.

Revised: Therefore, practically, the added computational cost of evaluating the gradients is negligible compare to that of MSE loss.

Original: The next step is to determine the structure of ( A , B ) that satisﬁes the above equations, and ﬁnd the subset of those solutions that account for local minima.
Revised: The next step is to determine the structure of ( A , B ) that satisﬁes the above equations, and ﬁnd the subset of those solutions that account for local minima.

Original: For the original loss, the ﬁrst expression ( A (cid:48) AB Σ xx =
Revised: For the original loss, the ﬁrst expression ( A (cid:48) AB Σ xx =

Original: A (cid:48) Σ yx ) is used to solve for B and put it in the second to derive an expression solely based on A .
Revised: A (cid:48) Σ yx ) is used to solve for B and put it in the second to derive an expression solely based on A .

Original: Obviously, in order to solve the ﬁrst expression for B , two cases are considered separately: the case where A is of full rank p , so A (cid:48)
Revised: Obviously, in order to solve the ﬁrst expression for B , two cases are considered separately: the case where A is of full rank p , so A (cid:48)

Original: A is invertible, and the case of A being of rank r < p .
Revised: A is invertible, and the case of A being of rank r < p .

Original: Here we do the same but there is a twist; for us there is only one case.
Revised: Here we do the same but there is a twist; for us there is only one case.

Original: The reason is as long as (not necessarily full rank)
Revised: The reason is as long as (not necessarily full rank)

Original: A has no zero column, S p ◦ ( A (cid:48)
Revised: A has no zero column, S p ◦ ( A (cid:48)

Original: A ) is positive deﬁnite and hence, invertible.
Revised: A ) is positive deﬁnite and hence, invertible.

Original: This is discussed in detail in Lemma 2 and we brieﬂy explain it here.
Revised: This is discussed in detail in Lemma 2 and we brieﬂy explain it here.

Original: As shown in the lemma, S p is positive deﬁnite and by Shur product theorem for any A (of any rank), S p ◦ ( A (cid:48)
Revised: As shown in the lemma, S p is positive deﬁnite and by Shur product theorem for any A (of any rank), S p ◦ ( A (cid:48)

Original: A ) is positive semideﬁnite.
Revised: A ) is positive semideﬁnite.

Original: However, as a result of Oppenheim inequality (Horn & Johnson (2012), Thm 7.8.16), that in our case translates to det( S p ) (cid:81) i
Revised: However, as a result of Oppenheim inequality (Horn & Johnson (2012), Thm 7.8.16), that in our case translates to det( S p ) (cid:81) i

Original: ( A (cid:48)
Revised: ( A (cid:48)

Original: A ) ii ≤ det( S p ◦
Revised: A ) ii ≤ det( S p ◦

Original: ( A (cid:48)
Revised: ( A (cid:48)

Original: A )) , as long as A has no zero column, (cid:81) i
Revised: A )) , as long as A has no zero column, (cid:81) i

Original: ( A (cid:48)
Revised: ( A (cid:48)

Original: A ) ii > 0 and therefore, det( S p ◦
Revised: A ) ii > 0 and therefore, det( S p ◦

Original: ( A (cid:48)
Revised: ( A (cid:48)

Original: A )) > 0 .
Revised: A )) > 0 .

Original: Here, we assume A of any rank r ≤ p has no zero column (since this can be easily avoided in practice) and consider S p ◦ ( A (cid:48)
Revised: Here, we assume A of any rank r ≤ p has no zero column (since this can be easily avoided in practice) and consider S p ◦ ( A (cid:48)

Original: A ) to be always invertible.
Revised: A ) to be always invertible.

Original: Therefore, ( A , B ) deﬁne a critical point of losses ˜ L and L if
Revised: Therefore, ( A , B ) deﬁne a critical point of losses ˜ L and L if

Original: For ˜ L ( A , B ) and full rank A :
Revised: For ˜ L ( A , B ) and full rank A :

Original: [Equation]
Revised: [Equation]

Original: For L ( A , B ) and no zero column A :
Revised: For L ( A , B ) and no zero column A :

Original: [Equation]
Revised: [Equation]

Original: Before, we state the main theorem we need the following deﬁnitions.
Revised: Before, we state the main theorem we need the following deﬁnitions.

Original: First, a rectangular permutation matrix Π r ∈ R r × p is a matrix that each column consists of at most one nonzero element with the value 1 .
Revised: First, a rectangular permutation matrix Π r ∈ R r × p is a matrix that each column consists of at most one nonzero element with the value 1 .

Original: If the rank of Π r is r with r < p then clearly, Π r has p − r zero columns.
Revised: If the rank of Π r is r with r < p then clearly, Π r has p − r zero columns.

Original: Also, by taking away those zero columns the resultant r × r submatrix of Π r is a standard square permutation matrix.
Revised: Also, by taking away those zero columns the resultant r × r submatrix of Π r is a standard square permutation matrix.

Original: Second, under the conditions provided in Assumption 1, the matrix Σ := Σ yx Σ − 1 xx Σ xy has an eigenvalue decomposition Σ = U Λ U (cid:48) , where the i th column of U , denoted as u i , is an eigenvector of Σ corresponding to the i th largest eigenvalue of Σ , denoted as λ i .
Revised: Second, under the conditions provided in Assumption 1, the matrix Σ := Σ yx Σ − 1 xx Σ xy has an eigenvalue decomposition Σ = U Λ U (cid:48) , where the i th column of U , denoted as u i , is an eigenvector of Σ corresponding to the i th largest eigenvalue of Σ , denoted as λ i .

Original: Also, Λ = diag( λ 1 , · · · , λ n ) is the diagonal vector of ordered eigenvalues of Σ , with λ 1 > λ 2 > · · · > λ n > 0 .
Revised: Also, Λ = diag( λ 1 , · · · , λ n ) is the diagonal vector of ordered eigenvalues of Σ , with λ 1 > λ 2 > · · · > λ n > 0 .

Original: We use the following notation to organize a subset of eigenvectors of Σ into a rectangular matrix.
Revised: We use the following notation to organize a subset of eigenvectors of Σ into a rectangular matrix.

Original: Let for any r ≤ p , I r
Revised: Let for any r ≤ p , I r

Original: = { i 1 , · · · , i r } (1 ≤ i 1 < · · · < i r < n ) be any ordered r − index set.
Revised: = { i 1 , · · · , i r } (1 ≤ i 1 < · · · < i r < n ) be any ordered r − index set.

Original: Deﬁne U
Revised: Deﬁne U

Original: I r ∈ R n × p as U I r
Revised: I r ∈ R n × p as U I r

Original: =
Revised: =

Original: [ u i 1 , · · · , u i r ] .
Revised: [ u i 1 , · · · , u i r ] .

Original: That is the columns of U I r are the ordered orthonormal eigenvectors of Σ associated with eigenvalues
Revised: That is the columns of U I r are the ordered orthonormal eigenvectors of Σ associated with eigenvalues

Original: λ i 1 < · · · < λ i r .
Revised: λ i 1 < · · · < λ i r .

Original: Clearly, when r = p , we have U I r
Revised: Clearly, when r = p , we have U I r

Original: =
Revised: =

Original: [ u i 1 , · · · , u i p ] corresponding to an p − index set I p =
Revised: [ u i 1 , · · · , u i p ] corresponding to an p − index set I p =

Original: { i 1 , · · · , i p } (1 ≤ i 1 < · · · < i p < n ) .
Revised: { i 1 , · · · , i p } (1 ≤ i 1 < · · · < i p < n ) .

Original: Similarly, we deﬁne Λ I r ∈ R p × p as Λ I r = diag( λ i 1 , · · · , λ i r ) .
Revised: Similarly, we deﬁne Λ I r ∈ R p × p as Λ I r = diag( λ i 1 , · · · , λ i r ) .

Original: Theorem 1.
Revised: Theorem 1.

Original: Let A ∈ R n × p and B ∈ R p × n such that A is of rank r ≤ p .
Revised: Let A ∈ R n × p and B ∈ R p × n such that A is of rank r ≤ p .

Original: Under the conditions provided in Assumption 1 and the above notation, The matrices A and B deﬁne a critical point of
Revised: Under the conditions provided in Assumption 1 and the above notation, The matrices A and B deﬁne a critical point of

Original: L ( A , B ) if and only if for any given r − index set I r , and a nonsingular diagonal matrix D ∈ R r × [Equation]
Revised: L ( A , B ) if and only if for any given r − index set I r , and a nonsingular diagonal matrix D ∈ R r × r ,

Original: A and B are of the form
Revised: A and B are of the form

Original: [Equation]
Revised: [Equation]

Original:  where, C ∈ R r × p is of of full rank r with nonzero and normalized columns such that Π C :
Revised: [Equation] (9) where, C ∈ R r × p is of of full rank r with nonzero and normalized columns such that Π C :

Original: = ( S p ◦ ( C (cid:48) C ))
Revised: = ( S p ◦ ( C (cid:48) C ))

Original: − 1 T p C (cid:48) is a rectangular permutation matrix of rank r and C Π C
Revised: − 1 T p C (cid:48) is a rectangular permutation matrix of rank r and C Π C

Original: = I r .
Revised: = I r .

Original: For all 1 ≤ r ≤ p, such C always exists.
Revised: For all 1 ≤ r ≤ p, such C always exists.

Original: In particular, if matrix A is of full rank p , i.e. r = p , the two given conditions on Π C are satisﬁed iff the invertible matrix
Revised: In particular, if matrix A is of full rank p , i.e. r = p , the two given conditions on Π C are satisﬁed iff the invertible matrix

Original: C is any squared p × p permutation matrix Π .
Revised: C is any squared p × p permutation matrix Π .

Original: In this case ( A , B ) deﬁne a critical point of L ( A , B ) iff they are of the form 
Revised: In this case ( A , B ) deﬁne a critical point of L ( A , B ) iff they are of the form [Equation]

Original: [Equation]
Revised: [Equation]

Original: The proof is given in appendix A.4. 
Revised: The proof is given in appendix A.4. Remark 3 .

Revised: The above theorem provides explicit equations for the critical points of the loss surface in terms of the rank of the decoder matrix A and the eigenvectors of Σ .

Revised: This explicit structure allows us to further analyze the loss surface and its local/global minima.

Original: Here, we provide a proof sketch for the above theorem to make the claims more clear.
Revised: Here, we provide a proof sketch for the above theorem to make the claims more clear.

Original: Again as a reminder, the EVD of Σ := Σ yx Σ − 1 xx Σ xy is Σ = U Λ U (cid:48) .
Revised: Again as a reminder, the EVD of Σ := Σ yx Σ − 1 xx Σ xy is Σ = U Λ U (cid:48) .

Original: For both ˜ L and L , the corresponding ˆ B ( A ) is replaced by B on the RHS of critical point equations.
Revised: For both ˜ L and L , the corresponding ˆ B ( A ) is replaced by B on the RHS of critical point equations.

Original: For the loss L ( A, B ) , as shown in the proof of the theorem, results in the following identity
Revised: For the loss L ( A, B ) , as shown in the proof of the theorem, results in the following identity

Original: [Equation]
Revised: [Equation]

Original: where ∆ := U (cid:48)
Revised: where ∆ := U (cid:48)

Original: AT p ( S p ◦
Revised: AT p ( S p ◦

Original: ( A (cid:48)
Revised: ( A (cid:48)

Original: A ))
Revised: A ))

Original: − 1 T p A (cid:48)
Revised: − 1 T p A (cid:48)

Original: U is symmetric and positive semideﬁnite.
Revised: U is symmetric and positive semideﬁnite.

Original: The LHS of eq.
Revised: The LHS of eq.

Original: (12) is symmetric so the RHS is symmetric too, so Λ∆
Revised: (12) is symmetric so the RHS is symmetric too, so Λ∆

Original: = ( Λ∆ ) (cid:48) =
Revised: = ( Λ∆ ) (cid:48) =

Original: ∆ (cid:48) Λ (cid:48)
Revised: ∆ (cid:48) Λ (cid:48)

Original: =
Revised: =

Original: ∆Λ .
Revised: ∆Λ .

Original: Therefore, ∆ commutes with the diagonal matrix of eigenvalues Λ .
Revised: Therefore, ∆ commutes with the diagonal matrix of eigenvalues Λ .

Original: Since eigenvalues are assumed to be distinct, ∆ has to be diagonal as well.
Revised: Since eigenvalues are assumed to be distinct, ∆ has to be diagonal as well.

Original: By Lemma 2 T p ( S p ◦
Revised: By Lemma 2 T p ( S p ◦

Original: ( A (cid:48)
Revised: ( A (cid:48)

Original: A ))
Revised: A ))

Original: − 1 T p is positive deﬁnite and U is an orthogonal matrix.
Revised: − 1 T p is positive deﬁnite and U is an orthogonal matrix.

Original: Therefore, r = rank ( A ) = rank ( ∆ ) =
Revised: Therefore, r = rank ( A ) = rank ( ∆ ) =

Original: rank ( U (cid:48)
Revised: rank ( U (cid:48)

Original: ∆ U ) , which implies that the diagonal matrix ∆ , has r nonzero and positive diagonal entries.
Revised: ∆ U ) , which implies that the diagonal matrix ∆ , has r nonzero and positive diagonal entries.

Original: There exists an r − index set I r corresponding to the nonzero diagonal elements of ∆ .
Revised: There exists an r − index set I r corresponding to the nonzero diagonal elements of ∆ .

Original: Forming a diagonal matrix ∆ I r ∈ R r × r by ﬁlling its diagonal entries (in order) by the nonzero diagonal elements of ∆ , we have
Revised: Forming a diagonal matrix ∆ I r ∈ R r × r by ﬁlling its diagonal entries (in order) by the nonzero diagonal elements of ∆ , we have

Original: [Equation]
Revised: [Equation]

Original: which indicates that the matrix A has the same column space as U I r .
Revised: which indicates that the matrix A has the same column space as U I r .

Original: Therefore, there exists a full rank matrix ¯ C ∈ R r × p such that A
Revised: Therefore, there exists a full rank matrix ¯ C ∈ R r × p such that A

Original: = U
Revised: = U

Original: I r ¯ C .
Revised: I r ¯ C .

Original: Since A has no zero column, ¯ C has no zero column.
Revised: Since A has no zero column, ¯ C has no zero column.

Original: Further, by normalizing the columns of ¯ C we can write A
Revised: Further, by normalizing the columns of ¯ C we can write A

Original: = U
Revised: = U

Original: I r CD , where D ∈ R p × p is diagonal that contains the norms of columns of ¯ C .
Revised: I r CD , where D ∈ R p × p is diagonal that contains the norms of columns of ¯ C .

Original: Baldi & Hornik (1989) did something similar for full rank A for the loss ˜ L to derive (
Revised: Baldi & Hornik (1989) did something similar for full rank A for the loss ˜ L to derive (

Original: A ˜ L
Revised: A ˜ L

Original: = U
Revised: = U

Original: I p ˜ C ).
Revised: I p ˜ C ).

Original: But their ˜ C can be any invertible p × p matrix.
Revised: But their ˜ C can be any invertible p × p matrix.

Original: However, in our case, the matrix C ∈ R r × p corresponding to rank r ≤ p matrix A , has to satisfy eq.
Revised: However, in our case, the matrix C ∈ R r × p corresponding to rank r ≤ p matrix A , has to satisfy eq.

Original: (13) by replacing A by U
Revised: (13) by replacing A by U

Original: I r CD and eq.
Revised: I r CD and eq.

Original: (12) by replacing ˆ B ( A ) by ˆ B ( U I r CD ) .
Revised: (12) by replacing ˆ B ( A ) by ˆ B ( U I r CD ) .

Original: In the case of Baldi & Hornik (1989), for the original loss ˜ L , equations similar to eq.
Revised: In the case of Baldi & Hornik (1989), for the original loss ˜ L , equations similar to eq.

Original: (13) and eq.
Revised: (13) and eq.

Original: (12) appear but they are are satisﬁed trivially by any invertible matrix ˜ C .
Revised: (12) appear but they are are satisﬁed trivially by any invertible matrix ˜ C .

Original: Simplifying those equations by using A
Revised: Simplifying those equations by using A

Original: = U
Revised: = U

Original: I r CD after some algebraic manipulation results in the following two conditions for C :
Revised: I r CD after some algebraic manipulation results in the following two conditions for C :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: As detailed in proof of Theorem 1, solving for C leads to its speciﬁc structure as laid out in the theorem.
Revised: As detailed in proof of Theorem 1, solving for C leads to its speciﬁc structure as laid out in the theorem.

Original: Remark 2 .
Revised: Remark 4 .

Original: Note that when A is of rank r < p with no zero columns then the invariant matrix C is not necessarily a rectangular permutation matrix but Π C :
Revised: Note that when A is of rank r < p with no zero columns then the invariant matrix C is not necessarily a rectangular permutation matrix but Π C :

Original: = ( S p ◦ ( C (cid:48) C ))
Revised: = ( S p ◦ ( C (cid:48) C ))

Original: − 1 T p C (cid:48) is a rectangular permutation matrix with C Π C
Revised: − 1 T p C (cid:48) is a rectangular permutation matrix with C Π C

Original: = I r .
Revised: = I r .

Original: It is only when r = p that the invariant matrix C becomes a permutation matrix.
Revised: It is only when r = p that the invariant matrix C becomes a permutation matrix.

Original: Nevertheless, as we show in the following corollary, the global map is always ∀ r ≤ p : G = AB =
Revised: Nevertheless, as we show in the following corollary, the global map is always ∀ r ≤ p : G = AB =

Original: U I r U (cid:48)
Revised: [Equation]

Original: I r Σ yx Σ − 1 xx .
Revised: I r Σ yx Σ − 1 xx .

Original: It is possible to ﬁnd further structure (in terms of block matrices) for the invariant matrix C when r < p .
Revised: It is possible to ﬁnd further structure (in terms of block matrices) for the invariant matrix C when r < p .

Original: However, this is not necessary as we soon show that all rank deﬁcient matrix A s are saddle points for the loss and ideally should be passed by during the gradient decent process.
Revised: However, this is not necessary as we soon show that all rank deﬁcient matrix A s are saddle points for the loss and ideally should be passed by during the gradient decent process.

Original: Based on some numerical results our conjecture is that when r < p the matrix C can only start with a r × k rectangular permutation matrix of rank r with r ≤ k ≤ p and the rest of p − k columns of C is arbitrary as long as none of the columns are identically zero.
Revised: Based on some numerical results our conjecture is that when r < p the matrix C can only start with a r × k rectangular permutation matrix of rank r with r ≤ k ≤ p and the rest of p − k columns of C is arbitrary as long as none of the columns are identically zero.

Original: Corollary 1.
Revised: Corollary 1.

Original: Let ( A , B ) be a critical point of L ( A , B ) under the conditions provided in Assumption 1 and rank A = r ≤ p .
Revised: Let ( A , B ) be a critical point of L ( A , B ) under the conditions provided in Assumption 1 and rank A = r ≤ p .

Original: Then the following are true
Revised: Then the following are true

Original: The matrix B Σ xx B (cid:48) is a p × p diagonal matrix of rank r .2.
Revised: The matrix B Σ xx B (cid:48) is a p × p diagonal matrix of rank r .

Original: For all 1 ≤ r ≤ p , for any critical pair ( A , B ) , the global map G := AB becomes
Revised: For all 1 ≤ r ≤ p , for any critical pair ( A , B ) , the global map G := AB becomes

Original: [Equation]
Revised: [Equation]

Original: For the autoencoder case ( Y = X ) the global map is simply G
Revised: For the autoencoder case ( Y = X ) the global map is simply G

Original: = U
Revised: = U

Original: I r U [Equation] I r
Revised: I r U (cid:48)

Original: [Equation]
Revised: [Equation]

Original: ( A , B ) is also the critical point of the classical loss ˜
Revised: ( A , B ) is also the critical point of the classical loss ˜

Original: L ( A , B ) 
Revised: L ( A [Equation] B ) =

Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: The proof is given in appendix A.5.
Revised: The proof is given in appendix A.5.

Original: Remark 3 .
Revised: Remark 5 .

Original: The above corollary implies that L ( A , B ) not only does not add any extra critical point compare to the original loss ˜
Revised: The above corollary implies that L ( A , B ) not only does not add any extra critical point compare to the original loss ˜

Original: L ( A , B ) , it provides the same global map G := AB .
Revised: L ( A , B ) , it provides the same global map G := AB .

Original: It only limits the structure of the invariance matrix C as described in Theorem 1 so that the decoder matrix A can recover the exact eigenvectors of Σ .
Revised: It only limits the structure of the invariance matrix C as described in Theorem 1 so that the decoder matrix A can recover the exact eigenvectors of Σ .

Original: Lemma 1.
Revised: Lemma 1.

Original: The loss function L ( A , B ) can be written as
Revised: The loss function L ( A , B ) can be written as

Original: [Equation]
Revised: [Equation]

Original: The above identity shows that the number of matrix operations required for computing the loss
Revised: The above identity shows that the number of matrix operations required for computing the loss

Original: L ( A , B ) is constant and thereby independent of the value of p .
Revised: L ( A , B ) is constant and thereby independent of the value of p .

Original: The proof is given in appendix A.6.
Revised: The proof is given in appendix A.6.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let A ∗ ∈ R n × p and B ∗ ∈
Revised: Let A ∗ ∈ R n × p and B ∗ ∈

Original: R p × n such that A  is of rank r ≤ p .
Revised: R p × n such that A ∗ is of rank r ≤ p .

Original: Under the conditions provided in Assumption 1, ( A ∗ , B ∗ ) deﬁne a local minima of the proposed loss function iff they are of the form
Revised: Under the conditions provided in Assumption 1, ( A ∗ , B ∗ ) deﬁne a local minima of the proposed loss function iff they are of the form

Original: [Equation]
Revised: [Equation]

Original:  where the i th column of U 1: p is a unit eigenvector of Σ := Σ yx Σ − 1 xx Σ xy corresponding the i th largest eigenvalue and D p is a diagonal matrix with nonzero diagonal elements.
Revised: [Equation] (19) where the i th column of U 1: p is a unit eigenvector of Σ := Σ yx Σ − 1 xx Σ xy corresponding the i th largest eigenvalue and D p is a diagonal matrix with nonzero diagonal elements.

Original: In other words,
Revised: In other words,

Original: A ∗ contains ordered unnormalized eigenvectors of Σ corresponding to the p largest eigenvalues.
Revised: A ∗ contains ordered unnormalized eigenvectors of Σ corresponding to the p largest eigenvalues.

Original: Moreover, all the local minima are global minima with the value of the loss function at those global minima being
Revised: Moreover, all the local minima are global minima with the value of the loss function at those global minima being

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where λ i is the i th largest eigenvalue of Σ .
Revised: where λ i is the i th largest eigenvalue of Σ .

Original: The proof is given in appendix A.7. 
Revised: The proof is given in appendix A.7. Remark 6 .

Revised: Finally, the second and third assumptions we made in the beginning in Assumption 1 can be relaxed by requiring only Σ xx to be full rank.

Revised: The output data can have a different dimension than the input.

Original: E XPERIMENTS 5.
Revised: That is Y ∈ R n × m and X ∈ R n (cid:48) × m , where n (cid:54) = n (cid:48) . The reason is that the given loss function structurally is very similar to MSE loss and can be represented as a Frobenius norm on the space of n × m matrices. In this case the covariance matrix Σ := Σ yx Σ − 1 xx Σ xy is still n × n . Clearly, for under-constrained systems with n < n (cid:48)

Revised: the full rank assumption of Σ holds.

Original: E
Revised: For the overdetermined case, where n (cid:48) > n the second and third assumptions in Assumption 1 can be relaxed: we only require Σ xx to be full rank since this is the only matrix that is inverted in the theorems. Note that if p > min( n (cid:48) , n ) then Λ I p : the p × p diagonal matrix of eigenvalues of Σ for a p -index-set I p bounds to have some zeros and will be say rank r < p , which in turn, results in the encoder A with rank r . However, the Theorem 1 is proved for encoder of any rank r ≤ p . Finally, following theorem 2 then the ﬁrst r columns of the encoder A converges to ordered eigenvectors of Σ while the p − r remaining columns span the kernel space of Σ .

Original: XPERIMENTAL
Revised: Moreover, Σ need not to have distinct eigenvectors. In this case ∆ I r becomes a block diagonal matrix, where the blocks correspond to identical eigenvalues Σ I r . In this case, the corresponding eigenvectors in A ∗ are not unique but they span the respective eigenspace.

Original:   S ETUP
Revised:  E XPERIMENTS 5. E XPERIMENTAL S ETUP

Original: LAEs with Two Loss functions We will verify the loss function
Revised: LAEs with Two Loss functions We will verify the loss function

Original: L ( A , B ) deﬁned in eq.
Revised: L ( A , B ) deﬁned in eq.

Original: (1) by setting the input matrix X ∈ R n × m equal to the output matrix Y ∈ R n × m ( Y = X ), where the linear autodecoder (LAE) becomes a solution to PCA.
Revised: (1) by setting the input matrix X ∈ R n × m equal to the output matrix Y ∈ R n × m ( Y = X ), where the linear autodecoder (LAE) becomes a solution to PCA.

Original: In order for comparison, we train another
Revised: In order for comparison, we train another

Original: LAE using the MSE loss ˜ L ( ˜ A , ˜ B ) deﬁned as ˜ L ( ˜ A , ˜ B ) =
Revised: LAE using the MSE loss ˜ L ( ˜ A , ˜ B ) deﬁned as ˜ L ( ˜ A , ˜ B ) =

Original: (cid:13)(cid:13)(cid:13) Y − ˜
Revised: (cid:13)(cid:13)(cid:13) Y − ˜

Original: A ˜ BX (cid:13)(cid:13)(cid:13) 2 F , where Y =
Revised: A ˜ BX (cid:13)(cid:13)(cid:13) 2 F , where Y =

Original: X is also applied in our experiments.
Revised: X is also applied in our experiments.

Original: The weights of networks are initialized to random numbers with a small enough standard deviation ( 10 − 7 in our case).
Revised: The weights of networks are initialized to random numbers with a small enough standard deviation ( 10 − 7 in our case).

Original: We choose to use the Adam optimizer with a scheduled learning rate (starting from 10 − 3 and ending with 10 − 6 in our case), which empirically beneﬁts the optimization process.
Revised: We choose to use the Adam optimizer with a scheduled learning rate (starting from 10 − 3 and ending with 10 − 6 in our case), which empirically beneﬁts the optimization process.

Original: The two training processes are stopped at the same iteration at which one of the models ﬁrstly ﬁnds all of the principal directions.
Revised: The two training processes are stopped at the same iteration at which one of the models ﬁrstly ﬁnds all of the principal directions.

Original: As a side note, we feed all data samples to the network at one time with batch size equal to m , although mini-batch implementations are apparently amendable.
Revised: As a side note, we feed all data samples to the network at one time with batch size equal to m , although mini-batch implementations are apparently amendable.

Original: Evaluation Metrics We use the classical PCA approach to get the ground truth principal direction matrix A ∗ ∈ R n × p , by conducting Eigen Value Decomposition (EVD) to XX (cid:

Revised: Evaluation Metrics We use the classical PCA approach to get the ground truth principal direction matrix A ∗ ∈ R n × p , by conducting Eigen Value Decomposition (EVD) to XX (cid:

Original: ) ∈ R n × n or Singular
Revised: ) ∈ R n × n or Singular

Original: Value Decomposition (SVD) to X ∈ R n × m .
Revised: Value Decomposition (SVD) to X ∈ R n × m .

Original: As a reminder, A ∈ R n × p stands for the decoder weight matrix of an trained LAE given a loss function L .
Revised: As a reminder, A ∈ R n × p stands for the decoder weight matrix of an trained LAE given a loss function L .

Original: To measure the distance between A ∗ and
Revised: To measure the distance between A ∗ and

Original: A , we propose an absolute cosine similarity ( ACS ) matrix inspired by mutual coherence (Donoho et al., 2005), which is deﬁned as:
Revised: A , we propose an absolute cosine similarity ( ACS ) matrix inspired by mutual coherence (Donoho et al., 2005), which is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where A ∗ i ∈ R n × 1 denotes the i th ground truth principal direction, and A j ∈ R n × 1 denotes the j th column of the decoder A , i, j = 1 , 2 , .
Revised: where A ∗ i ∈ R n × 1 denotes the i th ground truth principal direction, and A j ∈ R n × 1 denotes the j th column of the decoder A , i, j = 1 , 2 , .

Original: p .
Revised: p .

Original: The elements of ACS ∈
Revised: The elements of ACS ∈

Original: R p × p in eq.
Revised: R p × p in eq.

Original: (21) take values between [0,1], measuring pair-wise similarity across two sets of vectors.
Revised: (21) take values between [0,1], measuring pair-wise similarity across two sets of vectors.

Original: The absolute value absorbs the sign ambiguity of principal directions.
Revised: The absolute value absorbs the sign ambiguity of principal directions.

Original: The performances of LAEs are evaluated by deﬁning the following metrics:
Revised: The performances of LAEs are evaluated by deﬁning the following metrics:

Original: [Equation]
Revised: [Equation]

Original: where I is the indicator function and (cid:15) is a manual tolerance threshold ( (cid:15) = 0 .
Revised: where I is the indicator function and (cid:15) is a manual tolerance threshold ( (cid:15) = 0 .

Original: 01 in our case).
Revised: 01 in our case).

Original: If two vectors have absolute cosine similarity over 1 − (cid:15) , they are deemed equal.
Revised: If two vectors have absolute cosine similarity over 1 − (cid:15) , they are deemed equal.

Original: Considering some columns of decoder may be correct principal directions but not in the right order, we introduce
Revised: Considering some columns of decoder may be correct principal directions but not in the right order, we introduce

Original: Ratio TP and Ratio FP in eqs.
Revised: Ratio TP and Ratio FP in eqs.

Original: (22) and (23) to check the ratio of correct in-place and out-of-place principal directions respectively.
Revised: (22) and (23) to check the ratio of correct in-place and out-of-place principal directions respectively.

Original: Then Ratio Total in eq.
Revised: Then Ratio Total in eq.

Original: (24) measures the total ratio of the correctly obtained principal directions by the LAE regardless of the order.
Revised: (24) measures the total ratio of the correctly obtained principal directions by the LAE regardless of the order.

Original: Datasets As a proof-of-concept, both synthetic data and real data are used.
Revised: Datasets As a proof-of-concept, both synthetic data and real data are used.

Original: For the synthetic data, 2000 zero-centered data samples are generated from a 1000 -dimension zero mean multivariate normal distribution with the covariance matrix being diag( N p ) .
Revised: For the synthetic data, 2000 zero-centered data samples are generated from a 1000 -dimension zero mean multivariate normal distribution with the covariance matrix being diag( N p ) .

Original: For the real data, we choose to use
Revised: For the real data, we choose to use

Original: MNIST dataset (LeCun et al., 1998), which includes 60,000 grayscale handwritten digits images, each of dimension 28 × 28 = 784 .
Revised: MNIST dataset (LeCun et al., 1998), which includes 60,000 grayscale handwritten digits images, each of dimension 28 × 28 = 784 .

Original: 5. E VALUATION AND A NALYSIS
Revised: 5. E VALUATION AND A NALYSIS

Original: Synthetic Data Experiments In our experiment, p , the number of desired principal components (PCs), is set to 100 , i.e. the dimension is to be reduced from 1000 to 100 .
Revised: Synthetic Data Experiments In our experiment, p , the number of desired principal components (PCs), is set to 100 , i.e. the dimension is to be reduced from 1000 to 100 .

Original: Figures 1 and 2 demonstrate a few conclusions.
Revised: Figures 1 and 2 demonstrate a few conclusions.

Original: First, during the training process, the loss ratio of both losses continuously decreases to 1, i.e. they both converge to the optimal loss value.
Revised: First, during the training process, the loss ratio of both losses continuously decreases to 1, i.e. they both converge to the optimal loss value.

Original: However, when both get close enough, L require more iterations since the optimizer is forced to ﬁnd the right directions: it fully converges only after it has found all the principal directions in the right order.
Revised: However, when both get close enough, L require more iterations since the optimizer is forced to ﬁnd the right directions: it fully converges only after it has found all the principal directions in the right order.

Original: Second, using the loss L results in ﬁnding more and more correct principal directions, with
Revised: Second, using the loss L results in ﬁnding more and more correct principal directions, with

Original: Ratio TP continuously rising; and ultimately affords all correct and ordered principal directions,
Revised: Ratio TP continuously rising; and ultimately affords all correct and ordered principal directions,

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: with Ratio TP ending with 100%.
Revised: with Ratio TP ending with 100%.

Original: Notice that occasionally and temporarily, some of the principal directions is found but not at their correct position, which is indicated by the rise of Ratio FP in the ﬁgure.
Revised: Notice that occasionally and temporarily, some of the principal directions is found but not at their correct position, which is indicated by the rise of Ratio FP in the ﬁgure.

Original: However, as optimization continues they are shifted to the right column, which results in
Revised: However, as optimization continues they are shifted to the right column, which results in

Original: Ratio FP going back to zero, and Ratio TP reaching one.
Revised: Ratio FP going back to zero, and Ratio TP reaching one.

Original: As for ˜ L , it fails to identify any principal directions; both Ratio TP and Ratio FP for ˜ L stay at 0, which indicates that none of the columns of the decoder ˜ A , aligns with any principal direction.
Revised: As for ˜ L , it fails to identify any principal directions; both Ratio TP and Ratio FP for ˜ L stay at 0, which indicates that none of the columns of the decoder ˜ A , aligns with any principal direction.

Original: Third, as shown in the ﬁgure, while the optimizer ﬁnds almost all the principal directions rather quickly, it requires much more iterations to ﬁnd some ﬁnal ones.
Revised: Third, as shown in the ﬁgure, while the optimizer ﬁnds almost all the principal directions rather quickly, it requires much more iterations to ﬁnd some ﬁnal ones.

Original: This is because some eigenvalues in the empirical covariance matrix of the ﬁnite 2000 samples become very close (the difference becomes less than 1).
Revised: This is because some eigenvalues in the empirical covariance matrix of the ﬁnite 2000 samples become very close (the difference becomes less than 1).

Original: Therefore, the loss has to get very close to the optimal loss, making the gradient of the loss hard to distinguish between the two.
Revised: Therefore, the loss has to get very close to the optimal loss, making the gradient of the loss hard to distinguish between the two.

Original: [Figure]
Revised: [Figure]

Original: Real Data: MNIST Experiments We set the number of principal components (PCs) as 100, i.e., the dimension is to be reduced from 784 to 100.
Revised: Real Data: MNIST Experiments We set the number of principal components (PCs) as 100, i.e., the dimension is to be reduced from 784 to 100.

Original: We also try to reconstruct with the topcolumns found in this case.
Revised: We also try to reconstruct with the topcolumns found in this case.

Original: As in Fig. 3, the reconstruction performance of L is consistently better than ˜ L .
Revised: As in Fig. 3, the reconstruction performance of L is consistently better than ˜ L .

Original: That also reﬂects that ˜ L does not identify PCs, while L is directly applicable to performing PCA without bells and whistles.
Revised: That also reﬂects that ˜ L does not identify PCs, while L is directly applicable to performing PCA without bells and whistles.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have introduced a loss function for performing principal component analysis and linear regression using linear autoencoders.
Revised: In this paper, we have introduced a loss function for performing principal component analysis and linear regression using linear autoencoders.

Original: We have proved that the optimizing with the given loss results in the decoder matrix converges to the exact ordered unnormalized eigenvectors of the sample covariance matrix.
Revised: We have proved that the optimizing with the given loss results in the decoder matrix converges to the exact ordered unnormalized eigenvectors of the sample covariance matrix.

Original: We have also demonstrated the claims on a synthetic data set of random samples drawn from a multivariate normal distribution and on
Revised: We have also demonstrated the claims on a synthetic data set of random samples drawn from a multivariate normal distribution and on

Original: MNIST data set.
Revised: MNIST data set.

Original: There are several possible generalizations of this approach we are corently working on.
Revised: There are several possible generalizations of this approach we are currently working on.

Original: One is improving performance when the corresponding eigenvalues of two principal directions are very close and another is generalization of the loss for tensor decomposition.
Revised: One is improving performance when the corresponding eigenvalues of two principal directions are very close and another is generalization of the loss for tensor decomposition.

Original: A PPENDIX A P ROOFS A. P RELIMINARIES
Revised: A PPENDIX A P ROOFS A. P RELIMINARIES

Original: Before we present the proof for the main theorems, the following two lemmas introduce some notations and basic relations that are required for the proofs.
Revised: Before we present the proof for the main theorems, the following two lemmas introduce some notations and basic relations that are required for the proofs.

Original: Lemma 2.
Revised: Lemma 2.

Original: The constant matrices T p ∈
Revised: The constant matrices T p ∈

Original: R p × p and S p ∈ R p × p are deﬁned as
Revised: R p × p and S p ∈ R p × p are deﬁned as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2   1
Revised: 2   1

Original: [Equation]
Revised: [Equation]

Original:   4 3 23 3 22 2 21 1 1  .
Revised:   4 3 23 3 22 2 21 1 1  .

Original: Clearly, the diagonal matrix T p is positive deﬁnite.
Revised: Clearly, the diagonal matrix T p is positive deﬁnite.

Original: Another matrix that will appear in the formulation is ˆ S p :
Revised: Another matrix that will appear in the formulation is ˆ S p :

Original: = T − 1 p S p T − 1 p
Revised: = T − 1 p S p T − 1 p

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: −· · ·  
Revised: −· · ·  

Original:  , e
Revised:  , e

Original: [Equation]
Revised: [Equation]

Original: =                  
Revised: =                  

Original: [Equation]
Revised: [Equation]

Original: The following properties of Hadamard product and matrices T p and S p are used throughout:
Revised: The following properties of Hadamard product and matrices T p and S p are used throughout:

Original: For any arbitrary matrix A ∈ R n × p
Revised: For any arbitrary matrix A ∈ R n × p

Original: [Equation]
Revised: [Equation]

Original: where, ◦ is the Hadamard (element-wise) product.
Revised: where, ◦ is the Hadamard (element-wise) product.

Original: For any matrices M 1 , M 2 ∈ R p × p and diagonal matrices
Revised: For any matrices M 1 , M 2 ∈ R p × p and diagonal matrices

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: R p × p ,
Revised: R p × p ,

Original: [Equation]
Revised: [Equation]

Original: Moreover, if Π 1 , Π 2 ∈ R p × p are permutation matrices then
Revised: Moreover, if Π 1 , Π 2 ∈ R p × p are permutation matrices then

Original: [Equation]
Revised: [Equation]

Original: S p is invertible and its inverse is a symmetric tridiagonal matrix
Revised: S p is invertible and its inverse is a symmetric tridiagonal matrix

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: j =
Revised: j =

Original: [Equation]
Revised: [Equation]

Original: (cid:54)
Revised: (cid:54)

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: otherwise
Revised: otherwise

Original: , i.e. S
Revised: , i.e. S

Original: [Equation]
Revised: [Equation]

Original:   −· · ·  − − 0 ...
Revised:   −· · ·  − − 0 ...

Original: − − 1  −   .
Revised: − − 1  −   .

Original: S p is positive deﬁnite.
Revised: S p is positive deﬁnite.

Original: For any matrix A ∈ R n × p , S p ◦ ( A (cid:48)
Revised: For any matrix A ∈ R n × p , S p ◦ ( A (cid:48)

Original: A ) is positive semideﬁnite.
Revised: A ) is positive semideﬁnite.

Original: If (not necessarily full rank)
Revised: If (not necessarily full rank)

Original: A has no zero column then S p ◦ ( A (cid:48)
Revised: A has no zero column then S p ◦ ( A (cid:48)

Original: A ) is positive deﬁnite.
Revised: A ) is positive deﬁnite.

Original: For any diagonal matrix DDD ∈ R p
Revised: For any diagonal matrix DDD ∈ R p

Original: ×
Revised: ×

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: Let DDD , EEE ∈ R p
Revised: Let DDD , EEE ∈ R p

Original: ×
Revised: ×

Original: p be positive semideﬁnite matrices, where EEE has no zero diagonal element, and DDD is of rank r ≤ p .
Revised: p be positive semideﬁnite matrices, where EEE has no zero diagonal element, and DDD is of rank r ≤ p .

Original: Also, let for any r ≤ p , J r
Revised: Also, let for any r ≤ p , J r

Original: =
Revised: =

Original: { i 1 , · · · , i r } (1 ≤ i 1 < · · · < i r < n ) be any ordered r − index set.
Revised: { i 1 , · · · , i r } (1 ≤ i 1 < · · · < i r < n ) be any ordered r − index set.

Original: Then DDD and EEE satisfy
Revised: Then DDD and EEE satisfy

Original: [Equation]
Revised: [Equation]

Original: if and only if, the following two conditions are satisﬁed:
Revised: if and only if, the following two conditions are satisﬁed:

Original: (a) The matrix D D D is diagonal with p − r zero diagonal elements and r positive diagonal elements indexed by the set J r .
Revised: (a) The matrix D D D is diagonal with p − r zero diagonal elements and r positive diagonal elements indexed by the set J r .

Original: That is for any i ∈
Revised: That is for any i ∈

Original: J r : ( DDD ) ii > 0 and the rest of elements of DDD are zero.
Revised: J r : ( DDD ) ii > 0 and the rest of elements of DDD are zero.

Original: (b) For any i, j ∈ J r
Revised: (b) For any i, j ∈ J r

Original: and i (cid:54) =
Revised: and i (cid:54) =

Original: j we have ( EEE ) i,j = 0 .
Revised: j we have ( EEE ) i,j = 0 .

Original: Clearly, if DDD is positive deﬁnite then J r = N p and hence, both DDD and EEE are diagonal.
Revised: Clearly, if DDD is positive deﬁnite then J r = N p and hence, both DDD and EEE are diagonal.

Original: Proof. .
Revised: Proof. .

Original: The proof of the properties are as follows.
Revised: The proof of the properties are as follows.

Original: 1. eq.
Revised: 1. eq.

Original: (25) is trivial.
Revised: (25) is trivial.

Original: For eq.
Revised: For eq.

Original: (26) note that AI i ; p selects the ﬁrst i columns of A (zeros out the rest), and similarly, I i ; p A (cid:48) selects the ﬁrst i rows of A (zeros out the rest).
Revised: (26) note that AI i ; p selects the ﬁrst i columns of A (zeros out the rest), and similarly, I i ; p A (cid:48) selects the ﬁrst i rows of A (zeros out the rest).

Original: Therefore, I i ; p A (cid:48)
Revised: Therefore, I i ; p A (cid:48)

Original: AI
Revised: AI

Original: i ; p is a p × p matrix that its Leading Principal Submatrix of order i (LPS i ) 1 is the same as the LPS i of A (cid:48)
Revised: i ; p is a p × p matrix that its Leading Principal Submatrix of order i (LPS i ) 1 is the same as the LPS i of A (cid:48)

Original: A (and the rest of the elements are zero).
Revised: A (and the rest of the elements are zero).

Original: Hence, (cid:80) pi
Revised: Hence, (cid:80) pi

Original: =

Revised: =

Original: I i ; p A (cid:48)
Revised: I i ; p A (cid:48)

Original: AI i ; p (counting backwards) adds LPS p of A (cid:48)
Revised: AI i ; p (counting backwards) adds LPS p of A (cid:48)

Original: A (i.e. A (cid:48)
Revised: A (i.e. A (cid:48)

Original: A itself) with LPS p − 1 that doubles
Revised: A itself) with LPS p − 1 that doubles

Original: LPS p − 1 part of the result and then adds LPS p − 2 that triples the LPS p − 2 part of result, the process continues until by the last addition LPS 1 is added to the result for the p th times.
Revised: LPS p − 1 part of the result and then adds LPS p − 2 that triples the LPS p − 2 part of result, the process continues until by the last addition LPS 1 is added to the result for the p th times.

Original: This is exactly the same as evaluating S p ◦ ( A (cid:48) A ) .
Revised: This is exactly the same as evaluating S p ◦ ( A (cid:48) A ) .

Original: This
Revised: This

Original: is
Revised: is

Original: a
Revised: a

Original: standard
Revised: standard

Original: result
Revised: result

Original: (Horn & Johnson, 2012), and no
Revised: (Horn & Johnson, 2012), and no

Original: proof
Revised: proof

Original: is needed.
Revised: is needed.

Original: Directly compute S p S − 1 p
Revised: Directly compute S p S − 1 p

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Firstly, note that S − 1 p is symmetric and nonsingular so all the eigenvalues are real and nonzero.
Revised: Firstly, note that S − 1 p is symmetric and nonsingular so all the eigenvalues are real and nonzero.

Original: It is also a diagonally dominant matrix (Horn & Johnson (2012), Def 6.1.9) since
Revised: It is also a diagonally dominant matrix (Horn & Johnson (2012), Def 6.1.9) since

Original: [Equation]
Revised: [Equation]

Original: where the inequality is strict for the ﬁrst and the last row and it is equal for the rows in the middle.
Revised: where the inequality is strict for the ﬁrst and the last row and it is equal for the rows in the middle.

Original: Moreover, by Gersgorin circle theorem (Horn & Johnson (2012), Thm 6.1.1) for every eigenvalue l i of S − 1 p there exists i such that l i ∈
Revised: Moreover, by Gersgorin circle theorem (Horn & Johnson (2012), Thm 6.1.1) for every eigenvalue l i of S − 1 p there exists i such that l i ∈

Original: [ C i − R i , C i
Revised: [ C i − R i , C i

Original: + R
Revised: + R

Original: i ] .
Revised: i ] .

Original: Since ∀ i : C i ≥
Revised: Since ∀ i : C i ≥

Original: R i we have all the eigenvalues are non-negative.
Revised: R i we have all the eigenvalues are non-negative.

Original: They are also nonzero, hence, S − 1 p is positive deﬁnite, which implies S p is also positive deﬁnite.
Revised: They are also nonzero, hence, S − 1 p is positive deﬁnite, which implies S p is also positive deﬁnite.

Original: For any matrix A ∈ R n × p , A (cid:48)
Revised: For any matrix A ∈ R n × p , A (cid:48)

Original: A is positive semideﬁnite.
Revised: A is positive semideﬁnite.

Original: Also, S p is positive deﬁnite so by Schur product theorem (Horn & Johnson (2012), Thm 7.5.3(a)), S p ◦ ( A (cid:48)
Revised: Also, S p is positive deﬁnite so by Schur product theorem (Horn & Johnson (2012), Thm 7.5.3(a)), S p ◦ ( A (cid:48)

Original: A ) is positive
Revised: A ) is positive

Original: semideﬁnite.
Revised: semideﬁnite.

Original: Moreover, if all diagonal elements of A (cid:48)
Revised: Moreover, if all diagonal elements of A (cid:48)

Original: A are positive (i.e. A has no zero column) by the extension of Schur product theorem (Horn & Johnson (2012), Thm 7.5.3(b))
Revised: A are positive (i.e. A has no zero column) by the extension of Schur product theorem (Horn & Johnson (2012), Thm 7.5.3(b))

Original: it is positive deﬁnite.
Revised: it is positive deﬁnite.

Original: This can also be easily deduced using the Oppenheim inequality (Horn & Johnson (2012), Thm 7.8.16); that is for positive semideﬁnite matrices S p and A (cid:48)
Revised: This can also be easily deduced using the Oppenheim inequality (Horn & Johnson (2012), Thm 7.8.16); that is for positive semideﬁnite matrices S p and A (cid:48)

Original: A : det( S p ) (cid:81) i
Revised: A : det( S p ) (cid:81) i

Original: ( A (cid:48)
Revised: ( A (cid:48)

Original: A ) ii ≤ det( S p ◦
Revised: A ) ii ≤ det( S p ◦

Original: ( A (cid:48)
Revised: ( A (cid:48)

Original: A )) .
Revised: A )) .

Original: Since, S p is positive deﬁnite, det( S p ) >
Revised: Since, S p is positive deﬁnite, det( S p ) >

Original: (in fact it is 1 for any p ) and if A (cid:48)
Revised: (in fact it is 1 for any p ) and if A (cid:48)

Original: A has no zero diagonal then det( S p ◦ ( A (cid:48) A )) > 0 and therefore, S p ◦ ( A (cid:48)
Revised: A has no zero diagonal then det( S p ◦ ( A (cid:48) A )) > 0 and therefore, S p ◦ ( A (cid:48)

Original: A ) is positive deﬁnite.
Revised: A ) is positive deﬁnite.

Original: Clearly, the matrix T p is achieved by setting the off-diagonal elements of S p to zero.
Revised: Clearly, the matrix T p is achieved by setting the off-diagonal elements of S p to zero.

Original: Hence, for any diagonal matrix DDD ∈ R p × p : S p ◦ DDD = T p ◦ DDD .
Revised: Hence, for any diagonal matrix DDD ∈ R p × p : S p ◦ DDD = T p ◦ DDD .

Original: For the diagonal matrices Hadamard product and matrix product are interchangeable so the latter may also be written as T p DDD .
Revised: For the diagonal matrices Hadamard product and matrix product are interchangeable so the latter may also be written as T p DDD .

Original: The same argument applies for the second identity.
Revised: The same argument applies for the second identity.

Original: This property can easily be proved by induction on p and careful bookkeeping of indices.
Revised: This property can easily be proved by induction on p and careful bookkeeping of indices.

Original: Lemma 3 (Simultaneous diagonalization by congruence) .
Revised: Lemma 3 (Simultaneous diagonalization by congruence) .

Original: Let M 1 , M 2 ∈ R p × p , where M 1 is positive deﬁnite and M 2 is positive semideﬁnite.
Revised: Let M 1 , M 2 ∈ R p × p , where M 1 is positive deﬁnite and M 2 is positive semideﬁnite.

Original: Also, let DDD , EEE ∈ R
Revised: Also, let DDD , EEE ∈ R

Original: r
Revised: r

Original: ×
Revised: ×

Original: r be positive deﬁnite diagonal matrices with r ≤ p .
Revised: r be positive deﬁnite diagonal matrices with r ≤ p .

Original: Further, assume there is a C ∈ R r × p of rank r ≤ p such that
Revised: Further, assume there is a C ∈ R r × p of rank r ≤ p such that

Original: [Equation]
Revised: [Equation]

Original: Then there exists a nonsingular ¯ C ∈ R p × p that its ﬁrst r rows are the matrix C and
Revised: Then there exists a nonsingular ¯ C ∈ R p × p that its ﬁrst r rows are the matrix C and

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where, ¯ D D D = ¯ D D D ⊕ I r − p is a p × p diagonal matrix and ¯ E E E =
Revised: where, ¯ D D D = ¯ D D D ⊕ I r − p is a p × p diagonal matrix and ¯ E E E =

Original: E E
Revised: E E

Original: E ⊕ E E E is another p × p diagonal matrix,
Revised: E ⊕ E E E is another p × p diagonal matrix,

Original: in which EEE ∈ R p − r × p − r is a nonnegative diagonal matrix.
Revised: in which EEE ∈ R p − r × p − r is a nonnegative diagonal matrix.

Original: Clearly, the rank of M 2 is r plus the number of nonzero diagonal elements of EEE .
Revised: Clearly, the rank of M 2 is r plus the number of nonzero diagonal elements of EEE .

Original: Proof.
Revised: Proof.

Original: The proof is rather straightforward since this lemma is the direct consequence of Theorem 7.6.4 in Horn & Johnson (2012).
Revised: The proof is rather straightforward since this lemma is the direct consequence of Theorem 7.6.4 in Horn & Johnson (2012).

Original: The theorem basically states that if M 1 , M 2 ∈ R p × p is symmetric and M 1 is positive deﬁnite then there exists an invertible S ∈ R p × p such that S M 1 S (cid:48)
Revised: The theorem basically states that if M 1 , M 2 ∈ R p × p is symmetric and M 1 is positive deﬁnite then there exists an invertible S ∈ R p × p such that S M 1 S (cid:48)

Original: =
Revised: =

Original: I p and
Revised: I p and

Original: S M 2 S (cid:48) is a diagonal matrix with the same inertia as M 2 .
Revised: S M 2 S (cid:48) is a diagonal matrix with the same inertia as M 2 .

Original: Here, we have M 2 that is positive semideﬁnite and C ∈ R r × p of rank r ≤ p such that
Revised: Here, we have M 2 that is positive semideﬁnite and C ∈ R r × p of rank r ≤ p such that

Original: [Equation]
Revised: [Equation]

Original: Therefore, since S is of full rank p and DDD − 12 C is of rank r ≤ p , there exists p − r rows in S that are linearly independent of rows of DDD − 12 C .
Revised: Therefore, since S is of full rank p and DDD − 12 C is of rank r ≤ p , there exists p − r rows in S that are linearly independent of rows of DDD − 12 C .

Original: Establish ¯ C ∈ R p × p by adding those p − r rows to C .
Revised: Establish ¯ C ∈ R p × p by adding those p − r rows to C .

Original: Then ¯ C has p linearly independent rows so it is nonsingular, and fulﬁlls the lemma’s proposition that is
Revised: Then ¯ C has p linearly independent rows so it is nonsingular, and fulﬁlls the lemma’s proposition that is

Original: [Equation]
Revised: [Equation]

Original: where, ¯ DDD = ¯ DDD ⊕ I r − p is a p × p diagonal matrix and ¯ EEE = EEE ⊕ EEE is another p × p diagonal matrix, in which E E E ∈ R p − r × p − r is a nonnegative diagonal matrix.
Revised: where, ¯ DDD = ¯ DDD ⊕ I r − p is a p × p diagonal matrix and ¯ EEE = EEE ⊕ EEE is another p × p diagonal matrix, in which E E E ∈ R p − r × p − r is a nonnegative diagonal matrix.

Original: Lemma 4.
Revised: Lemma 4.

Original: Let A and B deﬁne a critical point of L .
Revised: Let A and B deﬁne a critical point of L .

Original: Further, let V ∈ R n × p and W ∈ R p × n are such that (cid:107) V (cid:107) F ,
Revised: Further, let V ∈ R n × p and W ∈ R p × n are such that (cid:107) V (cid:107) F ,

Original: (cid:107) W (cid:107) F = O ( ε ) for some ε > 0 .
Revised: (cid:107) W (cid:107) F = O ( ε ) for some ε > 0 .

Original: Then
Revised: Then

Original: L ( A + V , B + W ) − L ( A , B ) =
Revised: L ( A + V , B + W ) − L ( A , B ) =

Original: (cid:104) V T p B Σ xx B (cid:48) , V (cid:105) F
Revised: (cid:104) V T p B Σ xx B (cid:48) , V (cid:105) F

Original: [Equation]
Revised: [Equation]

Original: Further
Revised: Further

Original: [Equation]
Revised: [Equation]

Original: for W = ¯ W :=
Revised: for W = ¯ W :=

Original: ( S p ◦ ( A (cid:48)
Revised: ( S p ◦ ( A (cid:48)

Original: A ))
Revised: A ))

Original: − 1 T p V (cid:48)
Revised: − 1 T p V (cid:48)

Original: Σ yx Σ − 1 xx , the above equation becomes
Revised: Σ yx Σ − 1 xx , the above equation becomes

Original: [Equation]
Revised: [Equation]

Original: V
Revised: V

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: in case the critical A is of full rank p and so, ( A , B ) =
Revised: in case the critical A is of full rank p and so, ( A , B ) =

Original: ( U I p Π D , ˆ B ( U I p Π D )) , for the encoder direction V with (cid:107) V (cid:107) F
Revised: ( U I p Π D , ˆ B ( U I p Π D )) , for the encoder direction V with (cid:107) V (cid:107) F

Original: = O ( ε ) and W =
Revised: = O ( ε ) and W =

Original: ¯ W we have,
Revised: ¯ W we have,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: As described in appendix B.1, the second order Taylor expansion for the loss L ( A , B ) is then given by eq.
Revised: As described in appendix B.1, the second order Taylor expansion for the loss L ( A , B ) is then given by eq.

Original: (63), i.e.
Revised: (63), i.e.

Original: [Equation]
Revised: [Equation]

Original: If (cid:107) V (cid:107) F , (cid:107) W (cid:107) F = O ( ε ) then (cid:107) R ( V , W ) (cid:107) =
Revised: If (cid:107) V (cid:107) F , (cid:107) W (cid:107) F = O ( ε ) then (cid:107) R ( V , W ) (cid:107) =

Original: O ( ε 3 ) .
Revised: O ( ε 3 ) .

Original: Moreover, when A and B deﬁne a critical point of L we have d
Revised: Moreover, when A and B deﬁne a critical point of L we have d

Original: A L ( A , B ) V =
Revised: A L ( A , B ) V =

Original: d B L ( A , B ) W = 0 .
Revised: d B L ( A , B ) W = 0 .

Original: By setting the derivatives d 2 A L ( A , B ) V 2 , d AB L ( A , B ) V W , d 2 B L ( A , B ) W 2 that are given by eq.
Revised: By setting the derivatives d 2 A L ( A , B ) V 2 , d AB L ( A , B ) V W , d 2 B L ( A , B ) W 2 that are given by eq.

Original: (69), eq.
Revised: (69), eq.

Original: (68), and eq.
Revised: (68), and eq.

Original: (66) respectively, the above equation simpliﬁes to
Revised: (66) respectively, the above equation simpliﬁes to

Original: [Equation]
Revised: [Equation]

Original: Now, based on the ﬁrst item in Corollary 1, B Σ xx B (cid:48) is a p × p diagonal matrix, so based on eq.
Revised: Now, based on the ﬁrst item in Corollary 1, B Σ xx B (cid:48) is a p × p diagonal matrix, so based on eq.

Original: S p ◦ ( B Σ xx B (cid:48) )
Revised: S p ◦ ( B Σ xx B (cid:48) )

Original: =
Revised: =

Original: T p B Σ xx B (cid:48) .
Revised: T p B Σ xx B (cid:48) .

Original: The substitution then yields eq.
Revised: The substitution then yields eq.

Original: Finally, in the above equation replace W with ¯ W =
Revised: Finally, in the above equation replace W with ¯ W =

Original: ( S p ◦ ( A (cid:48)
Revised: ( S p ◦ ( A (cid:48)

Original: A ))
Revised: A ))

Original: − 1 T p V (cid:48)
Revised: − 1 T p V (cid:48)

Original: Σ yx Σ − 1 xx .
Revised: Σ yx Σ − 1 xx .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: which is eq.
Revised: which is eq.

Original: For the ﬁnal equation
Revised: For the ﬁnal equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Replace the above in eq.
Revised: Replace the above in eq.

Original: (30) and simplify:
Revised: (30) and simplify:

Original: [Equation]
Revised: [Equation]

Original: ( A + V
Revised: ( A + V

Original: [Equation]
Revised: [Equation]

Original: B + W ) − L
Revised: B + W ) − L

Original: [Equation]
Revised: [Equation]

Original: which ﬁnalizes the proof.
Revised: which ﬁnalizes the proof.

Original: A. P ROOF OF P ROPOSITION
Revised: A. P ROOF OF P ROPOSITION

Original: For this proof we use the ﬁrst and second order derivatives for L ( A , B ) wrt B derived in Lemma 5.
Revised: For this proof we use the ﬁrst and second order derivatives for L ( A , B ) wrt B derived in Lemma 5.

Original: From eq.
Revised: From eq.

Original: (66), we have that for a given A the second derivative wrt to B of the cost L ( A , B ) at B , and in the direction W is the quadratic form
Revised: (66), we have that for a given A the second derivative wrt to B of the cost L ( A , B ) at B , and in the direction W is the quadratic form

Original: [Equation]
Revised: [Equation]

Original: The matrix Σ xx is positive-deﬁnite and by Lemma 2, S p ◦
Revised: The matrix Σ xx is positive-deﬁnite and by Lemma 2, S p ◦

Original: A (cid:48)
Revised: A (cid:48)

Original: A is positive-semideﬁnite.
Revised: A is positive-semideﬁnite.

Original: Hence, d 2 B 2 L ( A , B ) W 2 is clearly non-negative for all W ∈ R p × n .
Revised: Hence, d 2 B 2 L ( A , B ) W 2 is clearly non-negative for all W ∈ R p × n .

Original: Therefore, L ( A , B ) is convex in coefﬁcients of B for a ﬁxed matrix A .
Revised: Therefore, L ( A , B ) is convex in coefﬁcients of B for a ﬁxed matrix A .

Original: Also the critical points of L ( A , B ) for a ﬁxed A is a matrix
Revised: Also the critical points of L ( A , B ) for a ﬁxed A is a matrix

Original: B that satisﬁes ∀ W ∈ R p × n : d B L ( A , B ) W = 0 and hence, from eq.
Revised: B that satisﬁes ∀ W ∈ R p × n : d B L ( A , B ) W = 0 and hence, from eq.

Original: (64) we have
Revised: (64) we have

Original: [Equation]
Revised: [Equation]

Original: Setting W =
Revised: Setting W =

Original: T p A (cid:48)
Revised: T p A (cid:48)

Original: Σ yx − ( S p ◦
Revised: Σ yx − ( S p ◦

Original: ( A (cid:48)
Revised: ( A (cid:48)

Original: A ))
Revised: A ))

Original: B Σ xx we have
Revised: B Σ xx we have

Original: [Equation]
Revised: [Equation]

Original: For a ﬁxed A , the cost L ( A , B ) is convex in B , so any matrix B that satisﬁes the above equation corresponds to a minimum of L ( A , B ) .
Revised: For a ﬁxed A , the cost L ( A , B ) is convex in B , so any matrix B that satisﬁes the above equation corresponds to a minimum of L ( A , B ) .

Original: Further, if A has no zero column then by Lemma 2, S p ◦
Revised: Further, if A has no zero column then by Lemma 2, S p ◦

Original: A (cid:48)
Revised: A (cid:48)

Original: A is positive deﬁnite.
Revised: A is positive deﬁnite.

Original: Hence, ∀ W ∈ R p × n : d 2 B 2 L ( A , B ) W 2 = 

Revised: Hence, ∀ W ∈ R p × n : d 2 B 2 L ( A , B ) W 2 =

Original: Tr ( W (cid:48) ( S p ◦
Revised: Tr ( W (cid:48) ( S p ◦

Original: A (cid:48)
Revised: A (cid:48)

Original: A ) W Σ xx ) is positive.
Revised: A ) W Σ xx ) is positive.

Original: Therefore, the cost L ( A , B ) becomes strictly convex and the unique global minimum is achieved at B
Revised: Therefore, the cost L ( A , B ) becomes strictly convex and the unique global minimum is achieved at B

Original: = ˆ B ( A ) as deﬁned in eq.
Revised: = ˆ B ( A ) as deﬁned in eq.

Original: A. P ROOF OF P ROPOSITION
Revised: A. P ROOF OF P ROPOSITION

Original: For this proof we use the ﬁrst and second order derivatives for L ( A , B ) wrt A derived in Lemma 6.
Revised: For this proof we use the ﬁrst and second order derivatives for L ( A , B ) wrt A derived in Lemma 6.

Original: For a ﬁxed B , based on eq.
Revised: For a ﬁxed B , based on eq.

Original: (69) the second derivative wrt to A of L ( A , B ) at A , and in the direction
Revised: (69) the second derivative wrt to A of L ( A , B ) at A , and in the direction

Original: V is the quadratic form
Revised: V is the quadratic form

Original: [Equation]
Revised: [Equation]

Original: The matrix Σ xx is positive-deﬁnite and by Lemma 2, S p ◦ ( B Σ xx B (cid:48) ) is positive-semideﬁnite.
Revised: The matrix Σ xx is positive-deﬁnite and by Lemma 2, S p ◦ ( B Σ xx B (cid:48) ) is positive-semideﬁnite.

Original: Hence, d 2 A 2 L 1 ( A , B ) V 2 is non-negative for all V ∈ R n × p .
Revised: Hence, d 2 A 2 L 1 ( A , B ) V 2 is non-negative for all V ∈ R n × p .

Original: Therefore, L ( A , B ) is convex in coefﬁcients of A for a ﬁxed matrix B .
Revised: Therefore, L ( A , B ) is convex in coefﬁcients of A for a ﬁxed matrix B .

Original: Based on eq.
Revised: Based on eq.

Original: (67) the critical point of L ( A , B ) for a ﬁxed
Revised: (67) the critical point of L ( A , B ) for a ﬁxed

Original: B is a matrix
Revised: B is a matrix

Original: A that satisﬁes for all V ∈ R n × p
Revised: A that satisﬁes for all V ∈ R n × p

Original: [Equation]
Revised: [Equation]

Original: which is eq.
Revised: which is eq.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Before we start, a reminder on notation and some useful identities that are used throughout the proof.
Revised: Before we start, a reminder on notation and some useful identities that are used throughout the proof.

Original: The matrix Σ := Σ yx Σ − 1 xx Σ xy has an eigenvalue decomposition Σ
Revised: The matrix Σ := Σ yx Σ − 1 xx Σ xy has an eigenvalue decomposition Σ

Original: = U Λ U (cid:48) , where the i th column of U , denoted as u i , is an eigenvector of Σ corresponding to the i th largest eigenvalue of Σ , denoted as λ i .
Revised: = U Λ U (cid:48) , where the i th column of U , denoted as u i , is an eigenvector of Σ corresponding to the i th largest eigenvalue of Σ , denoted as λ i .

Original: Also, Λ = diag( λ 1 , · · · , λ n ) is the diagonal vector of ordered eigenvalues of Σ , with λ 1 > λ 2 > · · · > λ n > 0 .
Revised: Also, Λ = diag( λ 1 , · · · , λ n ) is the diagonal vector of ordered eigenvalues of Σ , with λ 1 > λ 2 > · · · > λ n > 0 .

Original: We use the following notation to organize a subset of eigenvectors of Σ into a rectangular matrix.
Revised: We use the following notation to organize a subset of eigenvectors of Σ into a rectangular matrix.

Original: Let for any r ≤ p , I r
Revised: Let for any r ≤ p , I r

Original: = { i 1 , · · · , i r } (1 ≤ i 1 < · · · < i r < n ) be any ordered r − index set.
Revised: = { i 1 , · · · , i r } (1 ≤ i 1 < · · · < i r < n ) be any ordered r − index set.

Original: Deﬁne U
Revised: Deﬁne U

Original: I r ∈ R n × p as U I r
Revised: I r ∈ R n × p as U I r

Original: =
Revised: =

Original: [ u i 1 , · · · , u i r ] .
Revised: [ u i 1 , · · · , u i r ] .

Original: That is the columns of U I r are the ordered orthonormal eigenvectors of Σ associated with eigenvalues
Revised: That is the columns of U I r are the ordered orthonormal eigenvectors of Σ associated with eigenvalues

Original: λ i 1 < · · · < λ i r .
Revised: λ i 1 < · · · < λ i r .

Original: The following identities are then easy to verify:
Revised: The following identities are then easy to verify:

Original: [Equation]
Revised: [Equation]

Original: The sufﬁcient condition:
Revised: The sufﬁcient condition:

Original: Let A ∈ R n × p of rank r ≤ p and no zero column be given by eq.
Revised: Let A ∈ R n × p of rank r ≤ p and no zero column be given by eq.

Original: (8), B ∈ R p × n given by eq.
Revised: (8), B ∈ R p × n given by eq.

Original: (9), and the accompanying conditions are met.
Revised: (9), and the accompanying conditions are met.

Original: Notice that U (cid:48)
Revised: Notice that U (cid:48)

Original: I r U
Revised: I r U

Original: I r =
Revised: I r =

Original: I r implies that DC (cid:48) CD =
Revised: I r implies that DC (cid:48) CD =

Original: DC (cid:48) U (cid:48)
Revised: DC (cid:48) U (cid:48)

Original: I r U
Revised: I r U

Original: I r CD =
Revised: I r CD =

Original: A (cid:48)
Revised: A (cid:48)

Original: A , so
Revised: A , so

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: which is eq
Revised: which is eq

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: based on Proposition
Revised: based on Proposition

Original: [Equation]
Revised: [Equation]

Original: for the given A
Revised: for the given A

Original: [Equation]
Revised: [Equation]

Original: the matrix B deﬁnes a critical point of L ( A , B ) .
Revised: the matrix B deﬁnes a critical point of L ( A , B ) .

Original: For the gradient wrt to A
Revised: For the gradient wrt to A

Original: [Equation]
Revised: [Equation]

Original: ﬁrst note that with B given by eq.
Revised: ﬁrst note that with B given by eq.

Original: (9) we have
Revised: (9) we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The matrix Π C is a rectangular permutation matrix so Π C Λ
Revised: The matrix Π C is a rectangular permutation matrix so Π C Λ

Original: [Equation]
Revised: [Equation]

Original: Π
Revised: Π

Original: [Equation]
Revised: [Equation]

Original: C is diagonal so as
Revised: C is diagonal so as

Original: D − 1 Π C Λ I r Π (cid:48)
Revised: D − 1 Π C Λ I r Π (cid:48)

Original: C D − 1 .
Revised: C D − 1 .

Original: Therefore, B Σ xx B (cid:48) is diagonal and by eq.
Revised: Therefore, B Σ xx B (cid:48) is diagonal and by eq.

Original: (27) in Lemma 2-6 we have
Revised: (27) in Lemma 2-6 we have

Original: [Equation]
Revised: [Equation]

Original: which is eq.
Revised: which is eq.

Original: Therefore, based on Proposition Proposition 2, for the given B , the matrix A deﬁne a critical point of L ( A , B ) .
Revised: Therefore, based on Proposition Proposition 2, for the given B , the matrix A deﬁne a critical point of L ( A , B ) .

Original: Hence, A and B together deﬁne a critical point of L ( A , B ) .
Revised: Hence, A and B together deﬁne a critical point of L ( A , B ) .

Original: The necessary condition:
Revised: The necessary condition:

Original: Based on Proposition 1 and Proposition 2, for A (with no zero column) and B , to deﬁne a critical point of L ( A , B ) , B has to be ˆ B ( A ) given by eq.
Revised: Based on Proposition 1 and Proposition 2, for A (with no zero column) and B , to deﬁne a critical point of L ( A , B ) , B has to be ˆ B ( A ) given by eq.

Original: (6), and A has to satisfy eq.
Revised: (6), and A has to satisfy eq.

Original: That is
Revised: That is

Original: [Equation]
Revised: [Equation]

Original: (37) where, ∆ := U (cid:48)
Revised: (37) where, ∆ := U (cid:48)

Original: AT p ( S p ◦
Revised: AT p ( S p ◦

Original: ( A (cid:48)
Revised: ( A (cid:48)

Original: A ))
Revised: A ))

Original: − 1 T p A (cid:48)
Revised: − 1 T p A (cid:48)

Original: U is symmetric and positive semideﬁnite.
Revised: U is symmetric and positive semideﬁnite.

Original: The LHS of the above equation is symmetric so the RHS is symmetric too, so Λ∆
Revised: The LHS of the above equation is symmetric so the RHS is symmetric too, so Λ∆

Original: = ( Λ∆ ) (cid:48) =
Revised: = ( Λ∆ ) (cid:48) =

Original: ∆ (cid:48) Λ (cid:48)
Revised: ∆ (cid:48) Λ (cid:48)

Original: =
Revised: =

Original: ∆Λ .
Revised: ∆Λ .

Original: Therefore, ∆ commutes with the diagonal matrix of eigenvalues Λ .
Revised: Therefore, ∆ commutes with the diagonal matrix of eigenvalues Λ .

Original: Since, eigenvalues are assumed to be distinct, ∆ has to be diagonal as well.
Revised: Since, eigenvalues are assumed to be distinct, ∆ has to be diagonal as well.

Original: By Lemma 2 T p ( S p ◦
Revised: By Lemma 2 T p ( S p ◦

Original: ( A (cid:48)
Revised: ( A (cid:48)

Original: A ))
Revised: A ))

Original: − 1 T p is positive deﬁnite and U is an orthogonal matrix.
Revised: − 1 T p is positive deﬁnite and U is an orthogonal matrix.

Original: Therefore, r = rank ( A ) = rank ( ∆ ) =
Revised: Therefore, r = rank ( A ) = rank ( ∆ ) =

Original: rank ( U (cid:48)
Revised: rank ( U (cid:48)

Original: ∆ U ) , which implies that the diagonal matrix ∆ , has r nonzero and positive diagonal entries.
Revised: ∆ U ) , which implies that the diagonal matrix ∆ , has r nonzero and positive diagonal entries.

Original: There exists an r − index set I r corresponding to the nonzero diagonal elements of ∆ .
Revised: There exists an r − index set I r corresponding to the nonzero diagonal elements of ∆ .

Original: Forming a diagonal matrix ∆ I r ∈ R r × r by ﬁlling its diagonal entries (in order) by the nonzero diagonal elements of ∆ we have
Revised: Forming a diagonal matrix ∆ I r ∈ R r × r by ﬁlling its diagonal entries (in order) by the nonzero diagonal elements of ∆ we have

Original: [Equation]
Revised: [Equation]

Original: which indicates that the matrix A has the same column space as U I r .
Revised: which indicates that the matrix A has the same column space as U I r .

Original: Therefore, there exists a full rank matrix ˜ C ∈ R r × p such that A
Revised: Therefore, there exists a full rank matrix ˜ C ∈ R r × p such that A

Original: = U
Revised: = U

Original: I r ˜ C .
Revised: I r ˜ C .

Original: Since A has no zero column, ˜ C has no zero column.
Revised: Since A has no zero column, ˜ C has no zero column.

Original: Further, by normalizing the columns of ˜ C we can write A
Revised: Further, by normalizing the columns of ˜ C we can write A

Original: = U
Revised: = U

Original: I r CD , where D ∈ R p × p is diagonal that contains the norms of columns of ˜ C .
Revised: I r CD , where D ∈ R p × p is diagonal that contains the norms of columns of ˜ C .

Original: Therefore, A is exactly in the form given by eq.
Revised: Therefore, A is exactly in the form given by eq.

Original: The matrix C has to satisfy eq.
Revised: The matrix C has to satisfy eq.

Original: (38) that is
Revised: (38) that is

Original: [Equation]
Revised: [Equation]

Original: Now that the structure of A has been identiﬁed
Revised: Now that the structure of A has been identiﬁed

Original: [Equation]
Revised: [Equation]

Original: evaluate ˆ B ( A ) of eq.
Revised: evaluate ˆ B ( A ) of eq.

Original: (6) by setting A
Revised: (6) by setting A

Original: = U
Revised: = U

Original: I r CD
Revised: I r CD

Original: [Equation]
Revised: [Equation]

Original: that is
Revised: that is

Original: [Equation]
Revised: [Equation]

Original: which by deﬁning Π C :
Revised: which by deﬁning Π C :

Original: = ( S p ◦ ( C (cid:48) C ))
Revised: = ( S p ◦ ( C (cid:48) C ))

Original: − 1 T p C (cid:48) gives eq.
Revised: − 1 T p C (cid:48) gives eq.

Original: (34) for B as claimed.
Revised: (34) for B as claimed.

Original: While C has to satisfy eq.
Revised: While C has to satisfy eq.

Original: [Equation]
Revised: [Equation]

Original: A and B in the given form have to satisfy eq.
Revised: A and B in the given form have to satisfy eq.

Original: (37) that provides another condition for C as follows.
Revised: (37) that provides another condition for C as follows.

Original: First, note that
Revised: First, note that

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: replace A and B in eq
Revised: replace A and B in eq

Original: [Equation]
Revised: [Equation]

Original: (37) by their respective identities that we just derived.
Revised: (37) by their respective identities that we just derived.

Original: Performing the same process for eq
Revised: Performing the same process for eq

Original: [Equation]
Revised: [Equation]

Original: (37) we have
Revised: (37) we have

Original: [Equation]
Revised: [Equation]

Original: Now we have to ﬁnd C such that it satisﬁes eq.
Revised: Now we have to ﬁnd C such that it satisﬁes eq.

Original: (39) and eq.
Revised: (39) and eq.

Original: To make the process easier to follow, lets have them in one place.
Revised: To make the process easier to follow, lets have them in one place.

Original: The matrix C ∈ R r × p have to satisfy
Revised: The matrix C ∈ R r × p have to satisfy

Original: [Equation]
Revised: [Equation]

Original: Since C is a rectangular matrix, solving above equations for C in this form seems intractable.
Revised: Since C is a rectangular matrix, solving above equations for C in this form seems intractable.

Original: We use a trick to temporarily extend C into an invertible square matrix as follows.
Revised: We use a trick to temporarily extend C into an invertible square matrix as follows.

Original: • Temporarily, let M
Revised: • Temporarily, let M

Original: [Equation]
Revised: [Equation]

Original: Then M 1 is positive deﬁnite and M 2 is positive semideﬁnite, so they are simultaneously diagonalizable by congruence that is based on Lemma 3 and eq.
Revised: Then M 1 is positive deﬁnite and M 2 is positive semideﬁnite, so they are simultaneously diagonalizable by congruence that is based on Lemma 3 and eq.

Original: (41) and eq.
Revised: (41) and eq.

Original: (42), there exists a nonsingular ¯ C ∈ R p × p such that C consists of the ﬁrst r rows of ¯ C and
Revised: (42), there exists a nonsingular ¯ C ∈ R p × p such that C consists of the ﬁrst r rows of ¯ C and

Original: [Equation]
Revised: [Equation]

Original: where, ¯ ∆ I r = ∆ I r ⊕
Revised: where, ¯ ∆ I r = ∆ I r ⊕

Original: I r − p is a p × p diagonal matrix and ¯ Λ I r =
Revised: I r − p is a p × p diagonal matrix and ¯ Λ I r =

Original: Λ I r ⊕ Λ is another p × p diagonal matrix, in which Λ
Revised: Λ I r ⊕ Λ is another p × p diagonal matrix, in which Λ

Original: [Equation]
Revised: [Equation]

Original: − p ×
Revised: − p ×

Original: r − p is a nonnegative diagonal matrix.
Revised: r − p is a nonnegative diagonal matrix.

Original: • Substitute ¯ ∆ I r from eq.
Revised: • Substitute ¯ ∆ I r from eq.

Original: (43) in eq.
Revised: (43) in eq.

Original: (44), then left multiply by ¯ C (cid:48)− 1 , and right multiply by ¯
Revised: (44), then left multiply by ¯ C (cid:48)− 1 , and right multiply by ¯

Original: [Equation]
Revised: [Equation]

Original: • Now we can revert back everything to C again.
Revised: • Now we can revert back everything to C again.

Original: Since C consists of the ﬁrst r rows of ¯ C we have ¯ C (cid:48)
Revised: Since C consists of the ﬁrst r rows of ¯ C we have ¯ C (cid:48)

Original: I r ; p ¯ C = C (cid:48) C , and ¯ C (cid:48)
Revised: I r ; p ¯ C = C (cid:48) C , and ¯ C (cid:48)

Original: I r ; p ¯ Λ I r ¯ C = C (cid:48) Λ I r C , which turns the above equation into
Revised: I r ; p ¯ Λ I r ¯ C = C (cid:48) Λ I r C , which turns the above equation into

Original: [Equation]
Revised: [Equation]

Original: • In the above equation, replace I p by T
Revised: • In the above equation, replace I p by T

Original: [Equation]
Revised: [Equation]

Original: 1 p
Revised: 1 p

Original: [Equation]
Revised: [Equation]

Original: in
Revised: in

Original: LHS
Revised: LHS

Original: and by T
Revised: and by T

Original: [Equation]
Revised: [Equation]

Original: in
Revised: in

Original: the
Revised: the

Original: RHS.
Revised: RHS.

Original: Use
Revised: Use

Original: [Equation]
Revised: [Equation]

Original: to shrink it into :
Revised: to shrink it into :

Original: [Equation]
Revised: [Equation]

Original: • By the second property of Lemma 2 we can collect diagonal matrices T − 1 p ’s around S p to arrive at
Revised: • By the second property of Lemma 2 we can collect diagonal matrices T − 1 p ’s around S p to arrive at

Original: [Equation]
Revised: [Equation]

Original: where, ˆ S p :
Revised: where, ˆ S p :

Original: = T − 1 p S p T − 1 p .
Revised: = T − 1 p S p T − 1 p .

Original: • Deﬁne p × p matrices EEE r :=
Revised: • Deﬁne p × p matrices EEE r :=

Original: C (cid:48) C and DDD r :=
Revised: C (cid:48) C and DDD r :=

Original: T p Π C Λ I r Π (cid:48)
Revised: T p Π C Λ I r Π (cid:48)

Original: C T p .
Revised: C T p .

Original: Substitute in the above to arrive at:
Revised: Substitute in the above to arrive at:

Original: [Equation]
Revised: [Equation]

Original: Both DDD r and EEE r in the above identity are positive semideﬁnite.
Revised: Both DDD r and EEE r in the above identity are positive semideﬁnite.

Original: Moreover, since by assumption C has no zero columns, EEE r has no zero diagonal element.
Revised: Moreover, since by assumption C has no zero columns, EEE r has no zero diagonal element.

Original: Then the 7 th property of Lemma 2 implies the following two conclusions:
Revised: Then the 7 th property of Lemma 2 implies the following two conclusions:

Original: The matrix DDD r is diagonal.
Revised: The matrix DDD r is diagonal.

Original: The rank of DDD r is r so it has exactly r positive diagonal elements and the rest is zero.
Revised: The rank of DDD r is r so it has exactly r positive diagonal elements and the rest is zero.

Original: This argument is true for T − 1 p DDD r T − 1 p =
Revised: This argument is true for T − 1 p DDD r T − 1 p =

Original: Π C Λ I r Π (cid:48) C .
Revised: Π C Λ I r Π (cid:48) C .

Original: Since Λ I r is a diagonal positive deﬁnite matrix, the p × r matrix Π C :
Revised: Since Λ I r is a diagonal positive deﬁnite matrix, the p × r matrix Π C :

Original: =
Revised: =

Original: ( S p ◦ ( C (cid:48) C ))
Revised: ( S p ◦ ( C (cid:48) C ))

Original: − 1 T p C (cid:48)
Revised: − 1 T p C (cid:48)

Original: of rank r
Revised: of rank r

Original: should have p − r zero rows.
Revised: should have p − r zero rows.

Original: Let J r be an r − index set corresponding to nonzero diagonal elements of Π C Λ I r Π (cid:48) C .
Revised: Let J r be an r − index set corresponding to nonzero diagonal elements of Π C Λ I r Π (cid:48) C .

Original: Then the matrix Π C
Revised: Then the matrix Π C

Original: [ J r , N r ] ( r × r submatrix of Π C consist of its J r rows) is nonsingular.
Revised: [ J r , N r ] ( r × r submatrix of Π C consist of its J r rows) is nonsingular.

Original: For every i, j ∈ J r
Revised: For every i, j ∈ J r

Original: and i (cid:54) =
Revised: and i (cid:54) =

Original: j , ( EEE r ) i,j = 0 .
Revised: j , ( EEE r ) i,j = 0 .

Original: Since EEE r := C (cid:48) C and so ( EEE r ) i,j is the inner product of i th and j th columns of C , we conclude that the columns of
Revised: Since EEE r := C (cid:48) C and so ( EEE r ) i,j is the inner product of i th and j th columns of C , we conclude that the columns of

Original: C [ N r , J r ] ( r × r submatrix of C consist of its J r columns) are orthogonal or in other words
Revised: C [ N r , J r ] ( r × r submatrix of C consist of its J r columns) are orthogonal or in other words

Original: C [ N r , J r ] (cid:48) C [ N r , J r ] is diagonal.
Revised: C [ N r , J r ] (cid:48) C [ N r , J r ] is diagonal.

Original: The columns of C are normalized.
Revised: The columns of C are normalized.

Original: Therefore,
Revised: Therefore,

Original: C [ N r , J r ] (cid:48) C
Revised: C [ N r , J r ] (cid:48) C

Original: [ N r , J r ] = I r
Revised: [ N r , J r ] = I r

Original: and hence, C [ N r , J r ] is an orthogonal matrix.
Revised: and hence, C [ N r , J r ] is an orthogonal matrix.

Original: • We use the two conclusions to solve the original eq.
Revised: • We use the two conclusions to solve the original eq.

Original: (41) and eq.
Revised: (41) and eq.

Original: First use Π C
Revised: First use Π C

Original: :=
Revised: :=

Original: [Equation]
Revised: [Equation]

Original: ◦
Revised: ◦

Original: [Equation]
Revised: [Equation]

Original: to shrink them into :
Revised: to shrink them into :

Original: [Equation]
Revised: [Equation]

Original: Next, by the ﬁrst conclusion, the matrix T − 1 p DDD r T − 1 p
Revised: Next, by the ﬁrst conclusion, the matrix T − 1 p DDD r T − 1 p

Original: [Equation]
Revised: [Equation]

Original: is diagonal and so eq.
Revised: is diagonal and so eq.

Original: (46) becomes
Revised: (46) becomes

Original: [Equation]
Revised: [Equation]

Original: which is one of the two claimed conditions.
Revised: which is one of the two claimed conditions.

Original: What is left is to show that Π C is a rectangular permutation matrix.
Revised: What is left is to show that Π C is a rectangular permutation matrix.

Original: From the ﬁrst conclusion we also have Π C has exactly r nonzero columns indexed by J r so
Revised: From the ﬁrst conclusion we also have Π C has exactly r nonzero columns indexed by J r so

Original: [Equation]
Revised: [Equation]

Original: By the second conclusion C [ N r , J r ] is an orthogonal matrix therefore, Π C
Revised: By the second conclusion C [ N r , J r ] is an orthogonal matrix therefore, Π C

Original: [ J r , N r ] is the orthogonal matrix C
Revised: [ J r , N r ] is the orthogonal matrix C

Original: [ N r , J r ] (cid:48) .
Revised: [ N r , J r ] (cid:48) .

Original: Moreover, we had T − 1 p DDD r T − 1 p =
Revised: Moreover, we had T − 1 p DDD r T − 1 p =

Original: Π C Λ
Revised: Π C Λ

Original: I r Π (cid:48)
Revised: I r Π (cid:48)

Original: C is a p × p diagonal matrix with exactly r nonzero diagonal elements.
Revised: C is a p × p diagonal matrix with exactly r nonzero diagonal elements.

Original: Hence, Π C [ N r , J r ] Λ I r Π (cid:48)
Revised: Hence, Π C [ N r , J r ] Λ I r Π (cid:48)

Original: C [ N r , J r ] is an r × r positive deﬁnite diagonal matrix with Λ I r having distinct diagonal elements, and Π C
Revised: C [ N r , J r ] is an r × r positive deﬁnite diagonal matrix with Λ I r having distinct diagonal elements, and Π C

Original: [ N r , J r ] being orthogonal.
Revised: [ N r , J r ] being orthogonal.

Original: Therefore, Π C [ J r , N r ] (as well as C [ N r , J r ] ) should be a square permutation matrix.
Revised: Therefore, Π C [ J r , N r ] (as well as C [ N r , J r ] ) should be a square permutation matrix.

Original: Putting back the zero columns, we conclude that C should be such that Π C :=
Revised: Putting back the zero columns, we conclude that C should be such that Π C :=

Original: ( S p ◦ ( C (cid:48) C ))
Revised: ( S p ◦ ( C (cid:48) C ))

Original: − 1 T p C (cid:48) is a rectangular permutation matrix and C Π C
Revised: − 1 T p C (cid:48) is a rectangular permutation matrix and C Π C

Original: = I r .
Revised: = I r .

Original: Note that it is possible to further analyze these conditions and determine the exact structure of C .
Revised: Note that it is possible to further analyze these conditions and determine the exact structure of C .

Original: However, this is not needed in general for the critical point analysis of the next theorem except for the case where r = p and C is a square invertible matrix.
Revised: However, this is not needed in general for the critical point analysis of the next theorem except for the case where r = p and C is a square invertible matrix.

Original: In this case, square matrix Π C is of full rank p , J r = N p and therefore, C [ N r , J r ]
Revised: In this case, square matrix Π C is of full rank p , J r = N p and therefore, C [ N r , J r ]

Original: =
Revised: =

Original: C
Revised: C

Original: [ N p , N p ] = C .
Revised: [ N p , N p ] = C .

Original: Hence, C is any square permutation matrix Π ,
Revised: Hence, C is any square permutation matrix Π ,

Original: C (cid:48)
Revised: C (cid:48)

Original: C =
Revised: C =

Original: Π (cid:48)
Revised: Π (cid:48)

Original: Π
Revised: Π

Original: = I p and Π C :=
Revised: = I p and Π C :=

Original: ( S p ◦ ( C (cid:48) C ))
Revised: ( S p ◦ ( C (cid:48) C ))

Original: − 1 T p C (cid:48)
Revised: − 1 T p C (cid:48)

Original: =
Revised: =

Original: T − 1 p T p Π (cid:48)
Revised: T − 1 p T p Π (cid:48)

Original: =
Revised: =

Original: Π (cid:48) , which veriﬁes eq.
Revised: Π (cid:48) , which veriﬁes eq.

Original: (10) and eq.
Revised: (10) and eq.

Original: (11) for A and B when A is of full rank
Revised: (11) for A and B when A is of full rank

Original: p.
Revised: p.

Original: A. P ROOF OF C OROLLARY
Revised: A. P ROOF OF C OROLLARY

Original: We already show in the proof Theorem 1 that for critical ( A , B ) the matrix B Σ xx B (cid:48) is given by eq.
Revised: We already show in the proof Theorem 1 that for critical ( A , B ) the matrix B Σ xx B (cid:48) is given by eq.

Original: (36) that is
Revised: (36) that is

Original: [Equation]
Revised: [Equation]

Original: The matrix Π C is a p × r rectangular permutation matrix so Π C Λ
Revised: The matrix Π C is a p × r rectangular permutation matrix so Π C Λ

Original: I r Π (cid:48)
Revised: I r Π (cid:48)

Original: C is diagonal as well as D − 1 Π C Λ
Revised: C is diagonal as well as D − 1 Π C Λ

Original: I r Π (cid:48)
Revised: I r Π (cid:48)

Original: C D − 1 .
Revised: C D − 1 .

Original: Therefore, B Σ xx B (cid:48) is diagonal.
Revised: Therefore, B Σ xx B (cid:48) is diagonal.

Original: The diagonal matrix Λ I r is of rank r therefore, B Σ xx B (cid:48) is of rank r .
Revised: The diagonal matrix Λ I r is of rank r therefore, B Σ xx B (cid:48) is of rank r .

Original: Again by Theorem 1 critical ( A , B ) is of the form given by eq.
Revised: Again by Theorem 1 critical ( A , B ) is of the form given by eq.

Original: (8) and eq.
Revised: (8) and eq.

Original: (9) with the proceeding conditions on the invariance C .
Revised: (9) with the proceeding conditions on the invariance C .

Original: Therefore, the global map is
Revised: Therefore, the global map is

Original: [Equation]
Revised: [Equation]

Original: Based on Baldi & Hornik (1989) ( A , B ) deﬁne a critical point of ˜ L ( A , B ) =
Revised: Based on Baldi & Hornik (1989) ( A , B ) deﬁne a critical point of ˜ L ( A , B ) =

Original: (cid:80) pi
Revised: (cid:80) pi

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Y − ABX (cid:107) 2 F iff they satisfy
Revised: Y − ABX (cid:107) 2 F iff they satisfy

Original: [Equation]
Revised: [Equation]

Original: Again by assumption ( A , B ) deﬁne a critical point of L ( A , B ) so by Theorem 1 they are of the form given by eq.
Revised: Again by assumption ( A , B ) deﬁne a critical point of L ( A , B ) so by Theorem 1 they are of the form given by eq.

Original: (8) and eq.
Revised: (8) and eq.

Original: (9) with the proceeding conditions on the invariance
Revised: (9) with the proceeding conditions on the invariance

Original: C .
Revised: C .

Original: Hence,
Revised: Hence,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Hence, eq. (48) is satisﬁed.
Revised: Hence, eq. (48) is satisﬁed.

Original: For the second equation we use the ﬁrst property of this corollary that is B Σ xx B (cid:48) is diagonal and satisfy eq.
Revised: For the second equation we use the ﬁrst property of this corollary that is B Σ xx B (cid:48) is diagonal and satisfy eq.

Original: (7) of Proposition 2 that is
Revised: (7) of Proposition 2 that is

Original: [Equation]
Revised: [Equation]

Original: Hence, the second condition, eq.
Revised: Hence, the second condition, eq.

Original: (49) is also satisﬁed.
Revised: (49) is also satisﬁed.

Original: Therefore, any critical point of
Revised: Therefore, any critical point of

Original: L ( A , B ) is a critical point of ˜ L ( A , B ) .
Revised: L ( A , B ) is a critical point of ˜ L ( A , B ) .

Original: A. P ROOF OF L EMMA
Revised: A. P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: We have
Revised: We have

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: A , B
Revised: A , B

Original: [Equation]
Revised: [Equation]

Original: which is eq.
Revised: which is eq.

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: The full rank matrices A ∗ and B ∗ given by eq.
Revised: The full rank matrices A ∗ and B ∗ given by eq.

Original: (18) and eq.
Revised: (18) and eq.

Original: (19) are clearly of the form given by Theorem 1 with I p =
Revised: (19) are clearly of the form given by Theorem 1 with I p =

Original: N p := { 1 , 2 , · · · , p } , and Π p = I p .
Revised: N p := { 1 , 2 , · · · , p } , and Π p = I p .

Original: Hence, they deﬁne a critical point of
Revised: Hence, they deﬁne a critical point of

Original: L ( A , B ) .
Revised: L ( A , B ) .

Original: We want to show that these are the only local minima, that is any other critical ( A , B ) is a saddle points.
Revised: We want to show that these are the only local minima, that is any other critical ( A , B ) is a saddle points.

Original: The proof is similar to the second partial derivative test.
Revised: The proof is similar to the second partial derivative test.

Original: However, in this case the Hessian is a forth order tensor.
Revised: However, in this case the Hessian is a forth order tensor.

Original: Therefore, the second order Taylor approximation of the loss, derived in Lemma 4, is used directly.
Revised: Therefore, the second order Taylor approximation of the loss, derived in Lemma 4, is used directly.

Original: To prove the necessary condition, we show that at any other critical point ( A , B ) , where the ﬁrst order derivatives are zero, there exists inﬁnitesimal direction along which the second derivative of loss is negative.
Revised: To prove the necessary condition, we show that at any other critical point ( A , B ) , where the ﬁrst order derivatives are zero, there exists inﬁnitesimal direction along which the second derivative of loss is negative.

Original: Next, for the sufﬁcient condition we show that the any critical point of the form ( A ∗ , B ∗ ) is a local and global minima.
Revised: Next, for the sufﬁcient condition we show that the any critical point of the form ( A ∗ , B ∗ ) is a local and global minima.

Original: The necessary condition:
Revised: The necessary condition:

Original: Recall that U
Revised: Recall that U

Original: I p is the matrix of eigenvectors indexed by the p − index set I p and Π is a p × p permutation matrix.
Revised: I p is the matrix of eigenvectors indexed by the p − index set I p and Π is a p × p permutation matrix.

Original: Since all the index sets I r , r ≤ p are assumed to be ordered, the only way to have U N p =
Revised: Since all the index sets I r , r ≤ p are assumed to be ordered, the only way to have U N p =

Original: U
Revised: U

Original: I p Π is by having I p =
Revised: I p Π is by having I p =

Original: N p and Π
Revised: N p and Π

Original: = I p .
Revised: = I p .

Original: Let A (with no zero column) and B deﬁne an arbitrary critical point of L ( A , B ) .
Revised: Let A (with no zero column) and B deﬁne an arbitrary critical point of L ( A , B ) .

Original: Then Based on the previous theorem, either A
Revised: Then Based on the previous theorem, either A

Original: = U
Revised: = U

Original: I r C with r < p or A
Revised: I r C with r < p or A

Original: = U
Revised: = U

Original: I p Π D
Revised: I p Π D

Original: while in both cases B = ˆ B ( A ) given by eq.
Revised: while in both cases B = ˆ B ( A ) given by eq.

Original: If ( A , B ) is not of the form of ( A ∗ , B ∗ ) then there are three possibilities either 1)
Revised: If ( A , B ) is not of the form of ( A ∗ , B ∗ ) then there are three possibilities either 1)

Original: A
Revised: A

Original: =
Revised: =

Original: U
Revised: U

Original: I r CD with r < p , or 2)
Revised: I r CD with r < p , or 2)

Original: A
Revised: A

Original: =
Revised: =

Original: U
Revised: U

Original: I p Π D with I p (cid:54) =
Revised: I p Π D with I p (cid:54) =

Original: N p or 2)
Revised: N p or 2)

Original: A =
Revised: A =

Original: U N p Π D
Revised: U N p Π D

Original: but Π (cid:54)
Revised: but Π (cid:54)

Original: =
Revised: =

Original: I p .
Revised: I p .

Original: The ﬁrst two cases corresponds to not having the “right” and/or “enough” eigenvectors, and the third corresponds to not having the “right” ordering.
Revised: The ﬁrst two cases corresponds to not having the “right” and/or “enough” eigenvectors, and the third corresponds to not having the “right” ordering.

Original: We introduce the following notation and investigate each case separately.
Revised: We introduce the following notation and investigate each case separately.

Original: Let ε > 0 and
Revised: Let ε > 0 and

Original: U i ; j ∈ R n × p be a matrix of all zeros except the i th column, which contains u j ; the eigenvector of Σ corresponding to the j th largest eigenvalue.
Revised: U i ; j ∈ R n × p be a matrix of all zeros except the i th column, which contains u j ; the eigenvector of Σ corresponding to the j th largest eigenvalue.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: where, E i ∈ R p × p is matrix of zeros except the i th diagonal element that contains 1 .
Revised: where, E i ∈ R p × p is matrix of zeros except the i th diagonal element that contains 1 .

Original: In what follows, for each case we deﬁne a encoder direction V ∈ R n × p with (cid:107) V (cid:107) F = O ( ε ) , and set the decoder direction W ∈ R p × n as W =
Revised: In what follows, for each case we deﬁne a encoder direction V ∈ R n × p with (cid:107) V (cid:107) F = O ( ε ) , and set the decoder direction W ∈ R p × n as W =

Original: ¯ W :=
Revised: ¯ W :=

Original: ( S p ◦ ( A (cid:48)
Revised: ( S p ◦ ( A (cid:48)

Original: A ))
Revised: A ))

Original: − 1 T p V (cid:48)
Revised: − 1 T p V (cid:48)

Original: Σ yx Σ − 1 xx .
Revised: Σ yx Σ − 1 xx .

Original: Then we use eq.
Revised: Then we use eq.

Original: (30) and eq.
Revised: (30) and eq.

Original: (31) of Lemma 4, to show that the given direction ( V , W ) inﬁnitesimally reduces the loss and hence, in every case the corresponding critical ( A , B ) is a saddle point.
Revised: (31) of Lemma 4, to show that the given direction ( V , W ) inﬁnitesimally reduces the loss and hence, in every case the corresponding critical ( A , B ) is a saddle point.

Original: For the case
Revised: For the case

Original: A
Revised: A

Original: = U
Revised: = U

Original: I r CD , with r < p , note that based on the ﬁrst item in Corollary 1,
Revised: I r CD , with r < p , note that based on the ﬁrst item in Corollary 1,

Original: B Σ xx B (cid:48) is a p × p diagonal matrix of rank r so it has p − r zero diagonal elements.
Revised: B Σ xx B (cid:48) is a p × p diagonal matrix of rank r so it has p − r zero diagonal elements.

Original: Pick an i ∈ N p such that ( B Σ xx B (cid:48) ) ii is zero and a j ∈ N p \ I r .
Revised: Pick an i ∈ N p such that ( B Σ xx B (cid:48) ) ii is zero and a j ∈ N p \ I r .

Original: Set V =
Revised: Set V =

Original: ε U
Revised: ε U

Original: i ; j D and W = ¯ W .
Revised: i ; j D and W = ¯ W .

Original: Clearly,
Revised: Clearly,

Original: [Equation]
Revised: [Equation]

Original: Notice, (cid:107) V (cid:107) F
Revised: Notice, (cid:107) V (cid:107) F

Original: [Equation]
Revised: [Equation]

Original: (cid:107) W (cid:107) F = O ( ε ) , so based on eq.
Revised: (cid:107) W (cid:107) F = O ( ε ) , so based on eq.

Original: (30) of Lemma 4, we have
Revised: (30) of Lemma 4, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, since (cid:16) ˆ S p ◦ ( C (cid:48) C ) (cid:17) − 1 is a positive deﬁnite matrix, as ε → 0 , we have L ( A +
Revised: Therefore, since (cid:16) ˆ S p ◦ ( C (cid:48) C ) (cid:17) − 1 is a positive deﬁnite matrix, as ε → 0 , we have L ( A +

Original: V , B + W ) ≤
Revised: V , B + W ) ≤

Original: L ( A , B ) .
Revised: L ( A , B ) .

Original: Hence, any ( A , B ) =
Revised: Hence, any ( A , B ) =

Original: ( U I r CD , ˆ B ( U I r CD )) with r < p is a saddle point.
Revised: ( U I r CD , ˆ B ( U I r CD )) with r < p is a saddle point.

Original: Next, consider the case where A = U
Revised: Next, consider the case where A = U

Original: I p Π D with I p (cid:54) =
Revised: I p Π D with I p (cid:54) =

Original: N p .
Revised: N p .

Original: Then there exists at least one j ∈ I p \ N p and i ∈
Revised: Then there exists at least one j ∈ I p \ N p and i ∈

Original: N p \ I p such that
Revised: N p \ I p such that

Original: i < j (so λ i > λ j ).
Revised: i < j (so λ i > λ j ).

Original: Let σ be the permutation corresponding to the permutation matrix Π .
Revised: Let σ be the permutation corresponding to the permutation matrix Π .

Original: Also, let ε > 

Revised: Also, let ε >

Original: and U σ ( j ); i ∈ R n × p be a matrix of all zeros except the σ ( j ) th column, which contains u i ; the eigenvector of Σ corresponding to the i th largest eigenvalue.
Revised: and U σ ( j ); i ∈ R n × p be a matrix of all zeros except the σ ( j ) th column, which contains u i ; the eigenvector of Σ corresponding to the i th largest eigenvalue.

Original: Set V =
Revised: Set V =

Original: ε U σ ( j ); i D and W = ¯ W .
Revised: ε U σ ( j ); i D and W = ¯ W .

Original: Then, since i / ∈
Revised: Then, since i / ∈

Original: I p we have
Revised: I p we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: DU (cid:48)
Revised: DU (cid:48)

Original: σ ( j )
Revised: σ ( j )

Original: ; i U σ ( j ); i D = ε 2 D 2 E σ ( j ) , and (55)
Revised: ; i U σ ( j ); i D = ε 2 D 2 E σ ( j ) , and (55)

Original: [Equation]
Revised: [Equation]

Original: Since (cid:107) V (cid:107) F , (cid:107) W (cid:107) F
Revised: Since (cid:107) V (cid:107) F , (cid:107) W (cid:107) F

Original: = O ( ε ) , based on eq.
Revised: = O ( ε ) , based on eq.

Original: (31) of Lemma
Revised: (31) of Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Note that in the above, the diagonal matrix Π (cid:48) Λ I p Π has the same diagonal elements as Λ I p
Revised: Note that in the above, the diagonal matrix Π (cid:48) Λ I p Π has the same diagonal elements as Λ I p

Original: but they are permuted by σ .
Revised: but they are permuted by σ .

Original: So E σ ( j ) Π (cid:48)
Revised: So E σ ( j ) Π (cid:48)

Original: Λ I p Π selects σ ( j ) th diagonal element of Π (cid:48)
Revised: Λ I p Π selects σ ( j ) th diagonal element of Π (cid:48)

Original: Λ I p Π that is the j th diagonal element of Λ I p , which is nothing but λ j .
Revised: Λ I p Π that is the j th diagonal element of Λ I p , which is nothing but λ j .

Original: Now, since i <
Revised: Now, since i <

Original: j
Revised: j

Original: so λ i > λ j and σ ( j ) ≤ p , as ε → 0 , we have L ( A + V , B + W )
Revised: so λ i > λ j and σ ( j ) ≤ p , as ε → 0 , we have L ( A + V , B + W )

Original: ≤ L ( A , B ) .
Revised: ≤ L ( A , B ) .

Original: Hence, any ( A , B ) =
Revised: Hence, any ( A , B ) =

Original: ( U I p Π D , ˆ B ( U I p Π D )) is a saddle point.
Revised: ( U I p Π D , ˆ B ( U I p Π D )) is a saddle point.

Original: Finally consider the case where
Revised: Finally consider the case where

Original: A = U N p Π D with Π (cid:54) =
Revised: A = U N p Π D with Π (cid:54) =

Original: I p .
Revised: I p .

Original: Since Π (cid:54) =
Revised: Since Π (cid:54) =

Original: I p , the permutation σ of the set N p , corresponding to the permutation matrix Π , has at least a cycle ( i 1 i 2 · · · i k ) , where 1 < i 1 < i 2 · · · < i k < p and 2 ≤ k ≤ p .
Revised: I p , the permutation σ of the set N p , corresponding to the permutation matrix Π , has at least a cycle ( i 1 i 2 · · · i k ) , where 1 < i 1 < i 2 · · · < i k < p and 2 ≤ k ≤ p .

Original: Hence, Π can be decomposed as Π = Π
Revised: Hence, Π can be decomposed as Π = Π

Original: ( i 1 i 2 ··· i k )
Revised: ( i 1 i 2 ··· i k )

Original: ˆ Π , where ˆ Π is the permutation matrix corresponding to other cycles of σ .
Revised: ˆ Π , where ˆ Π is the permutation matrix corresponding to other cycles of σ .

Original: The cycle ( i 1 i 2 · · · i k ) can be decomposed into transpositions as ( i 1 i 2 · · · i k ) =
Revised: The cycle ( i 1 i 2 · · · i k ) can be decomposed into transpositions as ( i 1 i 2 · · · i k ) =

Original: ( i k
Revised: ( i k

Original: i k − 1 ) · · · ( i k i 1 ) , which in matrix form is Π ( i 1 i 2 ··· i k ) =
Revised: i k − 1 ) · · · ( i k i 1 ) , which in matrix form is Π ( i 1 i 2 ··· i k ) =

Original: Π ( i k i 1 ) Π
Revised: Π ( i k i 1 ) Π

Original: ( i k i 2 ) · · · Π ( i k
Revised: ( i k i 2 ) · · · Π ( i k

Original: i k − 1 ) .
Revised: i k − 1 ) .

Original: Therefore, Π can be decomposed as Π = Π ( i k i 1 ) ˜ Π , where ˜ Π
Revised: Therefore, Π can be decomposed as Π = Π ( i k i 1 ) ˜ Π , where ˜ Π

Original: = Π
Revised: = Π

Original: ( i k i 2 ) · · · Π ( i k
Revised: ( i k i 2 ) · · · Π ( i k

Original: i k − 1 ) ˆ Π .
Revised: i k − 1 ) ˆ Π .

Original: Note that Π ( i k i 1 ) , the permutation matrix corresponding to transposition ( i k i 1 ) is a symmetric involutory matrix, i.e. Π 2( i k i 1 )
Revised: Note that Π ( i k i 1 ) , the permutation matrix corresponding to transposition ( i k i 1 ) is a symmetric involutory matrix, i.e. Π 2( i k i 1 )

Original: =
Revised: =

Original: I p .
Revised: I p .

Original: Set V =
Revised: Set V =

Original: ε ( U i 1 ; i 1 − U i k ; i k )
Revised: ε ( U i 1 ; i 1 − U i k ; i k )

Original: ˜
Revised: ˜

Original: Π D and W = ¯ W .
Revised: Π D and W = ¯ W .

Original: Again we replace V and W in eq.
Revised: Again we replace V and W in eq.

Original: (31) of Lemma 4.
Revised: (31) of Lemma 4.

Original: There are some tedious steps to simplify the equation, which is given in appendix A.7.1.
Revised: There are some tedious steps to simplify the equation, which is given in appendix A.7.1.

Original: The ﬁnal result is as follows.
Revised: The ﬁnal result is as follows.

Original: With the given V and W , the third and forth terms of the RHS of eq.
Revised: With the given V and W , the third and forth terms of the RHS of eq.

Original: (31) are canceled and the ﬁrst two terms are simpliﬁed to
Revised: (31) are canceled and the ﬁrst two terms are simpliﬁed to

Original: [Equation]
Revised: [Equation]

Original: in which, m = max { k − 1 , 2 } .
Revised: in which, m = max { k − 1 , 2 } .

Original: This means that If the selected cycle is just a transposition ( i 1 i 2 ) then i m =
Revised: This means that If the selected cycle is just a transposition ( i 1 i 2 ) then i m =

Original: i 2 .
Revised: i 2 .

Original: But if for the selected cycle ( i 1 i 2 · · · i k ) , k is greater than 2 then i m =
Revised: But if for the selected cycle ( i 1 i 2 · · · i k ) , k is greater than 2 then i m =

Original: i k − 1 .
Revised: i k − 1 .

Original: Using above equations, eq. (31) yields
Revised: Using above equations, eq. (31) yields

Original: [Equation]
Revised: [Equation]

Original: By the above deﬁnition of i m , we have i m − i 1 > 0 and since i 1 <
Revised: By the above deﬁnition of i m , we have i m − i 1 > 0 and since i 1 <

Original: i k , λ i 1 − λ i k > 0 .
Revised: i k , λ i 1 − λ i k > 0 .

Original: Hence, the ﬁrst term in the above equation is negative and as ε → 0 , we have L ( A +
Revised: Hence, the ﬁrst term in the above equation is negative and as ε → 0 , we have L ( A +

Original: V , B + W ) − L ( A , B ) < 0 .
Revised: V , B + W ) − L ( A , B ) < 0 .

Original: Therefore, any any ( A , B ) =
Revised: Therefore, any any ( A , B ) =

Original: ( U I p Π D , ˆ B ( U I p Π D )) with Π (cid:54) =
Revised: ( U I p Π D , ˆ B ( U I p Π D )) with Π (cid:54) =

Original: I p is a saddle point.
Revised: I p is a saddle point.

Original: The Sufﬁcient condition:
Revised: The Sufﬁcient condition:

Original: From Lemma 1 we know that the loss L ( A , B ) can be written in the form of eq.
Revised: From Lemma 1 we know that the loss L ( A , B ) can be written in the form of eq.

Original: Use this equation to evaluate loss at ( A ∗ , B ∗ ) = (cid:16) U N p D p , D − 1 p U (cid:48)
Revised: Use this equation to evaluate loss at ( A ∗ , B ∗ ) = (cid:16) U N p D p , D − 1 p U (cid:48)

Original: N p Σ yx Σ − 1 xx (cid:17) as follows
Revised: N p Σ yx Σ − 1 xx (cid:17) as follows

Original: [Equation]
Revised: [Equation]

Original: + Tr (cid:18)(cid:18)
Revised: + Tr (cid:18)(cid:18)

Original: S
Revised: S

Original: [Equation]
Revised: [Equation]

Original: (cid:123)(cid:122) (cid:125) (cid:19)
Revised: (cid:123)(cid:122) (cid:125) (cid:19)

Original: D p D
Revised: D p D

Original: [Equation]
Revised: [Equation]

Original: (cid:124) (cid:123)(cid:122) (cid:125)
Revised: (cid:124) (cid:123)(cid:122) (cid:125)

Original: U (cid:48)
Revised: U (cid:48)

Original: N p Σ
Revised: N p Σ

Original: U N p
Revised: U N p

Original: (cid:124)
Revised: (cid:124)

Original: (cid:123)(cid:122) (cid:125)
Revised: (cid:123)(cid:122) (cid:125)

Original: D − 1 p D p (cid:124) (cid:123)(cid:122) (cid:125) (cid:19)
Revised: D − 1 p D p (cid:124) (cid:123)(cid:122) (cid:125) (cid:19)

Original: [Equation]
Revised: [Equation]

Original: which is eq.
Revised: which is eq.

Original: (20), as claimed.
Revised: (20), as claimed.

Original: Notice that the above value is independent of the diagonal matrix D p .
Revised: Notice that the above value is independent of the diagonal matrix D p .

Original: From the necessary condition we know that any critical point not in the form of ( A ∗ , B ∗ ) is a saddle point.
Revised: From the necessary condition we know that any critical point not in the form of ( A ∗ , B ∗ ) is a saddle point.

Original: Hence, due to the convexity of the loss at least one ( A ∗ , B ∗ ) is a global minimum but since the value of the loss at ( A ∗ , B ∗ ) is independent of D p all these critical points yield the same value for the loss.
Revised: Hence, due to the convexity of the loss at least one ( A ∗ , B ∗ ) is a global minimum but since the value of the loss at ( A ∗ , B ∗ ) is independent of D p all these critical points yield the same value for the loss.

Original: Therefore, any critical point in the form of ( A ∗ , B ∗ ) is a local and global minima.
Revised: Therefore, any critical point in the form of ( A ∗ , B ∗ ) is a local and global minima.

Original: A.7. S UPPLEMENTARY DETAILS OF THE PROOF OF T HEOREM
Revised: A.7. S UPPLEMENTARY DETAILS OF THE PROOF OF T HEOREM

Original: To verify eq.
Revised: To verify eq.

Original: (57), eq.
Revised: (57), eq.

Original: (58), and eq.
Revised: (58), and eq.

Original: (59) in the proof of Theorem 2, we want to replace V and W in eq.
Revised: (59) in the proof of Theorem 2, we want to replace V and W in eq.

Original: (31) of Lemma 4 with V =
Revised: (31) of Lemma 4 with V =

Original: ε ( U i 1 ; i 1 −
Revised: ε ( U i 1 ; i 1 −

Original: U
Revised: U

Original: i
Revised: i

Original: k ; i k ) ˜
Revised: k ; i k ) ˜

Original: Π D and W = ¯ W and simplify.
Revised: Π D and W = ¯ W and simplify.

Original: eq.
Revised: eq.

Original: (31) is
Revised: (31) is

Original: [Equation]
Revised: [Equation]

Original: We investigate each term on the RHS separately.
Revised: We investigate each term on the RHS separately.

Original: but before note that
Revised: but before note that

Original: [Equation]
Revised: [Equation]

Original: where, ˜ σ and its function inverse ˜ σ − 1 are permutations corresponding to ˜ Π and ˜ Π (cid:48) respectively.
Revised: where, ˜ σ and its function inverse ˜ σ − 1 are permutations corresponding to ˜ Π and ˜ Π (cid:48) respectively.

Original: ˜ Π T p ˜ Π (cid:48) is a diagonal matrix where diagonal elements of T p are ordered based on ˜ σ − 1 .
Revised: ˜ Π T p ˜ Π (cid:48) is a diagonal matrix where diagonal elements of T p are ordered based on ˜ σ − 1 .

Original: Moreover, recall that we decomposed the permutation matrix Π in A with a cycle ( i 1 i 2 · · · i k ) as Π =
Revised: Moreover, recall that we decomposed the permutation matrix Π in A with a cycle ( i 1 i 2 · · · i k ) as Π =

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: Π ( i 1 i k )
Revised: Π ( i 1 i k )

Original: ˜
Revised: ˜

Original: Π , where i 1 , i 2 , · · · i k are ﬁxed points of ˆ Π .
Revised: Π , where i 1 , i 2 , · · · i k are ﬁxed points of ˆ Π .

Original: Therefore, with ˜ σ being the permutation corresponding to ˜ Π we have
Revised: Therefore, with ˜ σ being the permutation corresponding to ˜ Π we have

Original: [Equation]
Revised: [Equation]

Original: where, m = max { k − 1 , 2 } .
Revised: where, m = max { k − 1 , 2 } .

Original: This means that If the selected cycle is just a transposition ( i 1 i 2 ) then i m =
Revised: This means that If the selected cycle is just a transposition ( i 1 i 2 ) then i m =

Original: i 2 .
Revised: i 2 .

Original: But if for the selected cycle ( i 1 i 2 · · · i k ) , k is greater than 2 then i m =
Revised: But if for the selected cycle ( i 1 i 2 · · · i k ) , k is greater than 2 then i m =

Original: i k − 1 .
Revised: i k − 1 .

Original: For the ﬁrst term we have
Revised: For the ﬁrst term we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: which is eq.
Revised: which is eq.

Original: (57) as claimed
Revised: (57) as claimed

Original: [Equation]
Revised: [Equation]

Original: For the second term we have
Revised: For the second term we have

Original: [Equation]
Revised: [Equation]

Original: which is eq.
Revised: which is eq.

Original: (58) as claimed
Revised: (58) as claimed

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: we have to show that the third and the forth terms of the eq.
Revised: we have to show that the third and the forth terms of the eq.

Original: (31) are canceled.
Revised: (31) are canceled.

Original: First, observe that
Revised: First, observe that

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Equation]
Revised: [Equation]

Original: Now, note that in both cases the matrices that are multiplied elementwise with Π S p ˜ Π (cid:48) are diagonal and hence, we only need to look at diagonal elements of Π S p ˜ Π (cid:48) .
Revised: Now, note that in both cases the matrices that are multiplied elementwise with Π S p ˜ Π (cid:48) are diagonal and hence, we only need to look at diagonal elements of Π S p ˜ Π (cid:48) .

Original: Moreover,
Revised: Moreover,

Original: [Equation]
Revised: [Equation]

Original: where, i 1 · · · i k are ﬁxed points of permutation corresponding to ˆ Π
Revised: where, i 1 · · · i k are ﬁxed points of permutation corresponding to ˆ Π

Original: so ˆ Π S p
Revised: so ˆ Π S p

Original: ˆ Π (cid:48) has the same values at diagonal positions
Revised: ˆ Π (cid:48) has the same values at diagonal positions

Original: i 

Revised: i

Original: and i k as the original matrix S p .
Revised: and i k as the original matrix S p .

Original: The only permutation that is only on the left side is Π ( i 1 i k ) which exchanges the i 

Revised: The only permutation that is only on the left side is Π ( i 1 i k ) which exchanges the i

Original: and i k rows of S p .
Revised: and i k rows of S p .

Original: Since S p is such that the elements at each row before the diagonal element are the same and i k > i 1 , we have the i 1 and i k diagonal elements of Π S p ˜ Π (cid:48) have the same value.
Revised: Since S p is such that the elements at each row before the diagonal element are the same and i k > i 1 , we have the i 1 and i k diagonal elements of Π S p ˜ Π (cid:48) have the same value.

Original: Let that value be denoted as s .
Revised: Let that value be denoted as s .

Original: Then the sum of the above two equations yields m ( λ i 

Revised: Then the sum of the above two equations yields m ( λ i

Original: + λ i k ) − m ( λ i 

Revised: + λ i k ) − m ( λ i

Original: + λ i k )
Revised: + λ i k )

Original: = 0 , as claimed.
Revised: = 0 , as claimed.

Original: B D ERIVATIVES OF THE L OSS FUNCTION B. F IRST AND S ECOND O RDER F RÉCHET D ERIVATIVE
Revised: B D ERIVATIVES OF THE L OSS FUNCTION B. F IRST AND S ECOND O RDER F RÉCHET D ERIVATIVE

Original: In order to derive and analyze the critical points of the cost function which is a real-valued function of matrices we use the ﬁrst and second order Fréchet derivatives as described in chapter 4 of Zeidler (1995).
Revised: In order to derive and analyze the critical points of the cost function which is a real-valued function of matrices we use the ﬁrst and second order Fréchet derivatives as described in chapter 4 of Zeidler (1995).

Original: For a function f : R n × m → R the ﬁrst order Fréchet derivative at the point A ∈ R n × m is a linear functional df ( A ) :
Revised: For a function f : R n × m → R the ﬁrst order Fréchet derivative at the point A ∈ R n × m is a linear functional df ( A ) :

Original: R n × m → R such that
Revised: R n × m → R such that

Original: [Equation]
Revised: [Equation]

Original: where we used the shorthand df ( A ) V ≡ ( df ( A ))
Revised: where we used the shorthand df ( A ) V ≡ ( df ( A ))

Original: ( V ) .
Revised: ( V ) .

Original: Similarly, the 2nd derivative is a bilinear functional d 2 f ( A ) : R n × m × R n × m → R such that
Revised: Similarly, the 2nd derivative is a bilinear functional d 2 f ( A ) : R n × m × R n × m → R such that

Original: [Equation]
Revised: [Equation]

Original: for all (cid:107) K (cid:107) F ≤ 1 , where again d 2 f ( A ) V K ≡ ( d 2 f ( A ))
Revised: for all (cid:107) K (cid:107) F ≤ 1 , where again d 2 f ( A ) V K ≡ ( d 2 f ( A ))

Original: ( V , K ) .
Revised: ( V , K ) .

Original: The generalized Taylor formula then becomes:
Revised: The generalized Taylor formula then becomes:

Original: [Equation]
Revised: [Equation]

Original: Moreover, we derive functions ∇ f : R n × m → R n × m and H ( A ) : R n × m → R n × m such that df ( A ) V
Revised: Moreover, we derive functions ∇ f : R n × m → R n × m and H ( A ) : R n × m → R n × m such that df ( A ) V

Original: =
Revised: =

Original: (cid:104)∇ f ( A ) , V (cid:105) F and d 2 f ( A ) V 

Revised: (cid:104)∇ f ( A ) , V (cid:105) F and d 2 f ( A ) V

Original: = (cid:104) H ( A ) V , V (cid:105) F , where again H ( A ) V ≡
Revised: = (cid:104) H ( A ) V , V (cid:105) F , where again H ( A ) V ≡

Original: H ( A )( V ) .
Revised: H ( A )( V ) .

Original: Then clearly, A ∈ R n × m is a critical point of f iff ∇ f ( A )
Revised: Then clearly, A ∈ R n × m is a critical point of f iff ∇ f ( A )

Original: = 

Revised: =

Original: and for such A s the sign of the bilinear form (cid:104) H ( A ) V , V (cid:105) over directions V determines the type of the critical point.
Revised: and for such A s the sign of the bilinear form (cid:104) H ( A ) V , V (cid:105) over directions V determines the type of the critical point.

Original: Extending the generalized Taylor theorem of Zeidler (1995), the second order Taylor expansion for the loss L ( A , B ) is then given by
Revised: Extending the generalized Taylor theorem of Zeidler (1995), the second order Taylor expansion for the loss L ( A , B ) is then given by

Original: [Equation]
Revised: [Equation]

Original: where, if (cid:107) V (cid:107) F , (cid:107) W (cid:107) F = O ( ε ) then (cid:107) R ( V , W ) (cid:107) =
Revised: where, if (cid:107) V (cid:107) F , (cid:107) W (cid:107) F = O ( ε ) then (cid:107) R ( V , W ) (cid:107) =

Original: O ( ε 3 ) .
Revised: O ( ε 3 ) .

Original: Clearly, as at critical points where d
Revised: Clearly, as at critical points where d

Original: A L ( A , B ) V + d B L ( A , B ) W = 0 , as ε → 0 we have R V , W ( A , B ) → 0 and the sign of the sum of the second order partial Fréchet derivatives determines the type of the critical point very much similar to second partial derivative test for two variable functions.
Revised: A L ( A , B ) V + d B L ( A , B ) W = 0 , as ε → 0 we have R V , W ( A , B ) → 0 and the sign of the sum of the second order partial Fréchet derivatives determines the type of the critical point very much similar to second partial derivative test for two variable functions.

Original: However, here for local minima we have to show the sign is positive in all directions and for saddle points have to show the sign is positive in some directions and negative at least in on direction.
Revised: However, here for local minima we have to show the sign is positive in all directions and for saddle points have to show the sign is positive in some directions and negative at least in on direction.

Original: Finally, note that the smoothness of the loss entails that Fréchet derivative and directional derivative
Revised: Finally, note that the smoothness of the loss entails that Fréchet derivative and directional derivative

Original: Gateaux) both exist and (foregoing some subtleties in deﬁnition) are the same.
Revised: Gateaux) both exist and (foregoing some subtleties in deﬁnition) are the same.

Original: B. F IRST AND S ECOND O RDER D ERIVATIVE OF THE L OSS WRT TO B
Revised: B. F IRST AND S ECOND O RDER D ERIVATIVE OF THE L OSS WRT TO B

Original: Lemma 5.
Revised: Lemma 5.

Original: The ﬁrst and second (partial Fréchet ) derivative of the loss L ( A , B ) wrt to B is derived as follows.
Revised: The ﬁrst and second (partial Fréchet ) derivative of the loss L ( A , B ) wrt to B is derived as follows.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Directly compute
Revised: Directly compute

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which can be written as the given form.
Revised: which can be written as the given form.

Original: For the second derivative wrt B we have
Revised: For the second derivative wrt B we have

Original: [Equation]
Revised: [Equation]

Original: which by having ¯ W → 0 results in the second order partial derivative
Revised: which by having ¯ W → 0 results in the second order partial derivative

Original: [Equation]
Revised: [Equation]

Original: B. F IRST AND S ECOND O RDER D ERIVATIVE OF THE L OSS WRT TO A
Revised: B. F IRST AND S ECOND O RDER D ERIVATIVE OF THE L OSS WRT TO A

Original: Lemma 6.
Revised: Lemma 6.

Original: The ﬁrst and second (partial Fréchet ) derivative of the loss L ( A , B ) wrt to A is derived as follows.
Revised: The ﬁrst and second (partial Fréchet ) derivative of the loss L ( A , B ) wrt to A is derived as follows.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: Directly compute
Revised: Directly compute

Original: [Equation] [Table]
Revised: [Equation] [Table]