C RITICAL P OINTS OF L INEAR N EURAL N ETWORKS : A NALYTICAL F ORMS AND L ANDSCAPE P ROPERTIES
Anonymous Authors
A BSTRACT
Due to the success of deep learning to solving a variety of challenging machine learning tasks, there is a rising interest in understanding loss functions for training neural networks from a theoretical aspect.
Particularly, the properties of critical points and the landscape around them are of importance to determine the convergence performance of optimization algorithms.
In this paper, we provide a necessary and sufﬁcient characterization of the analytical forms for the critical points (as well as global minimizers) of the square loss functions for linear neural networks.
We show that the analytical forms of the critical points characterize the values of the corresponding loss functions as well as the necessary and sufﬁcient conditions to achieve global minimum.
Furthermore, we exploit the analytical forms of the critical points to characterize the landscape properties for the loss functions of linear neural networks and shallow ReLU networks.
One particular conclusion is that: While the loss function of linear networks has no spurious local minimum, the loss function of one-hidden-layer nonlinear networks with ReLU activation function does have local minimum that is not global minimum.
I NTRODUCTION
In the past decade, deep neural networks Goodfellow et al.
(2016) have become a popular tool that has successfully solved many challenging tasks in a variety of areas such as machine learning, artiﬁcial intelligence, computer vision, and natural language processing, etc.
As the understandings of deep neural networks from different aspects are mostly based on empirical studies, there is a rising need and interest to develop understandings of neural networks from theoretical aspects such as generalization error, representation power, and landscape (also referred to as geometry) properties, etc.
In particular, the landscape properties of loss functions (that are typically nonconex for neural networks) play a central role to determine the iteration path and convergence performance of optimization algorithms.
One major landscape property is the nature of critical points, which can possibly be global minima, local minima, saddle points.
There have been intensive efforts in the past into understanding such an issue for various neural networks.
For example, it has been shown that every local minimum of the loss function is also a global minimum for shallow linear networks under the autoencoder setting and invertibility assumptions Baldi & Hornik (1989) and for deep linear networks Kawaguchi (2016);
Lu & Kawaguchi (2017); Yun et al.
(2017) respectively under different assumptions.
The conditions on the equivalence between local minimum or critical point and global minimum has also been established for various nonlinear neural networks Yu & Chen (1995); Gori & Tesi (1992); Nguyen & Hein (2017); Soudry & Carmon (2016); Feizi et al.
(2017) under respective assumptions.
However, most previous studies did not provide characterization of analytical forms for critical points of loss functions for neural networks with only very few exceptions.
In Baldi & Hornik (1989), the authors provided an analytical form for the critical points of the square loss function of shallow linear networks under certain conditions.
Such an analytical form further helps to establish the landscape properties around the critical points.
Further in Li et al. (2016b), the authors characterized certain sufﬁcient form of critical points for the square loss function of matrix factorization problems and deep linear networks. The focus of this paper is on characterizing the sufﬁcient and necessary forms of critical points for broader scenarios, i.e., shallow and deep linear networks with no assumptions on data matrices and network dimensions, and shallow ReLU networks over certain parameter space.
In particular, such analytical forms of critical points capture the corresponding loss function values and the necessary and sufﬁcient conditions to achieve global minimum.
This further enables us to establish new landscape properties around these critical points for the loss function of these networks under general settings, and provides alternative (yet simpler and more intuitive) proofs for existing understanding of the landscape properties.
O UR C ONTRIBUTION
We summarize our contributions in detail as follows
For the square loss function of linear networks with one hidden layer, we provide a full (necessary and sufﬁcient) characterization of the analytical forms for its critical points and global minimizers.
These results generalize the characterization in Baldi & Hornik (1989) to arbitrary network parameter dimensions and any data matrices.
Such a generalization further enables us to establish the landscape property, i.e., every local minimum is also a global minimum and all other critical points are saddle points, under no assumptions on parameter dimensions and data matrices.
From a technical standpoint, we exploit the analytical forms of critical points to provide a new proof for characterizing the landscape around the critical points under full relaxation of assumptions, where the corresponding approaches in Baldi & Hornik (1989) are not applicable.
As a special case of linear networks, the matrix factorization problem satisﬁes all these landscape properties.
For the square loss function of deep linear networks, we establish a full (necessary and sufﬁcient) characterization of the analytical forms for its critical points and global minimizers.
Such characterizations are new and have not been established in the existing art.
Furthermore, such analytical form divides the set of non-global-minimum critical points into different categories.
We identify the directions along which the loss function value decreases for two categories of the critical points, for which our result directly implies the equivalence between the local minimum and the global minimum.
For these cases, our proof generalizes the result in Kawaguchi (2016) under no assumptions on the network parameter dimensions and data matrices.
For the square loss function of one-hidden-layer nonlinear neural networks with ReLU activation function, we provide a full characterization of both the existence and the analytical forms of the critical points in certain types of regions in the parameter space.
Particularly, in the case where there is one hidden unit
, our results fully characterize the existence and the analytical forms of the critical points in the entire parameter space.
Such characterization were not provided in previous work on nonlinear neural networks.
Moreover, we apply our results to a concrete example to demonstrate that both local minimum that is not a global minimum and local maximum do exist in such a case.
R ELATED W ORK
Analytical forms of critical points :
Characterizing the analytical form of critical points for loss functions of neural networks dates back to Baldi & Hornik (1989), where the authors provided an analytical form of the critical points for the square loss function of linear networks with one hidden layer. In Li et al.
(2016b), the authors provided a sufﬁcient condition of critical points of a generic function, i.e., the ﬁxed point of invariant groups.
They then characterized certain sufﬁcient forms of critical points for the square loss function of matrix factorization problems and deep linear networks, whereas our results provide sufﬁcient and necessary forms of critical points for deep linear networks via a different approach.
Properties of critical points : Baldi & Hornik (1989); Baldi (1989) studied the linear autoencoder with one hidden layer and showed the equivalence between the local minimum and the global minimum.
Moreover, Baldi & Lu (2012) generalized these results to the complex-valued autoencoder setting.
The deep linear networks were studied by some recent work Kawaguchi (2016);
Lu &
Kawaguchi (2017); Yun et al.
(2017), in which the equivalence between the local minimum and the global minimum was established respectively under different assumptions.
Particularly, Yun et al.
(2017) established a necessary and sufﬁcient condition for a critical point of the deep linear network to be a global minimum.
A similar result was established in Freeman & Bruna (2017) for deep linear networks under the setting that the widths of intermediate layers are larger than those of the input and output layers.
The effect of regularization on the critical points for a two-layer linear network was studied in Taghvaei et al.
For nonlinear neural networks, Yu & Chen (1995) studied a nonlinear neural network with one hidden layer and sigmoid activation function, and showed that every local minimum is also a global minimum provided that the number of input units equals the number of data samples.
Gori & Tesi (1992) considered a class of multi-layer nonlinear networks with a pyramidal structure, and showed that all critical points of full column rank achieve the zero loss when the sample size is less than the input dimension.
These results were further generalized to a larger class of nonlinear networks in
Nguyen & Hein (2017), in which they also showed that critical points with non-degenerate Hessian are global minimum.
Choromanska et al.
(2015a;b) connected the loss surface of deep nonlinear networks with the Hamiltonian of the spin-glass model under certain assumptions and characterized the distribution of the local minimum.
Kawaguchi (2016) further eliminated some of the assumptions in Choromanska et al.
(2015a), and established the equivalence between the local minimum and the global minimum by reducing the loss function of the deep nonlinear network to that of the deep linear network.
Soudry & Carmon (2016) showed that a two-layer nonlinear network has no bad differentiable local minimum.
Feizi et al.
(2017) studied a one-hidden-layer nonlinear neural network with the parameters restricted in a set of directions of lines, and showed that most local minima are global minima.
Tian (2017) considered a two-layer ReLU network with Gaussian input data, and showed that critical points in certain region are non-isolated and characterized the critical-point-free regions.
Geometric curvature : Hardt & Ma (2017) established the gradient dominance condition of deep linear residual networks, and Zhou & Liang (2017) further established the gradient dominance condition and regularity condition around the global minimizers for deep linear, deep linear residual and shallow nonlinear networks.
Li et al.
(2016a) studied the property of the Hessian matrix for deep linear residual networks.
The local strong convexity property was established in Soltanolkotabi et al.
(2017) for overparameterized nonlinear networks with one hidden layer and quadratic activation functions, and was established in Zhong et al.
(2017) for a class of nonlinear networks with one hidden layer and Gaussian input data.
Zhong et al.
(2017) further established the local linear convergence of gradient descent method with tensor initialization.
Soudry & Hoffer (2017) studied a one-hidden-layer nonlinear network with a single output, and showed that the volume of sub-optimal differentiable local minima is exponentially vanishing in comparison with the volume of global minima. Dauphin et al. (2014) investigated the saddle points in deep neural networks using the results from statistical physics and random matrix theory.
Notation: The pseudoinverse, column space and null space of a matrix M are denoted by
M † , col( M ) and ker( M ) , respectively.
For any index sets I, J ⊂ N , M I,J denotes the submatrix of M formed by the entries with the row indices in I and the column indices in J .
For positive integers i ≤ j , we deﬁne i :
j = { i, i + 1 , . . .
, j − 1 , j } .
The projection operator onto a linear subspace V is denoted by P V .
L INEAR N EURAL N ETWORKS WITH O NE H IDDEN L AYER
In this section, we study linear neural networks with one hidden layer.
Suppose we have an input data matrix X ∈ R d 0 × m and a corresponding output data matrix Y ∈ R
d 2 × m , where there are in total m data samples.
We are interested in learning a model that maps from X to Y via a linear network with one hidden layer.
Speciﬁcally, we denote the weight parameters between the output layer and the hidden layer of the network as A 2 ∈ R d 2 × d 1 , and denote the weight parameters between the hidden layer and the input layer of the network as A 1 ∈ R d 1 × d 0 .
We are interested in the square loss function of this linear network, which is given by
[Equation]
Note that in a special case where X =
I , L reduces to a loss function for the matrix factorization problem, to which all our results apply.
The loss function L has been studied in Baldi & Hornik (1989) under the assumptions that d 2 = d 0 ≥ d 1 and the matrices XX (cid:62) , Y X (cid:62) ( XX (cid:62) ) − 1 XY (cid:62) are invertible.
In our study, no assumption is made on either the parameter dimensions or the invertibility of the data matrices.
Such full generalization of the results in Baldi & Hornik (1989) turns out to be critical for our study of nonlinear shallow neural networks in Section 4.
We further deﬁne Σ :=
Y X † XY (cid:62) and denote its full singular value decomposition as U Λ U (cid:62) .
Suppose that Σ has r distinct positive singular values σ 1 > · · · > σ r > 0 with multiplicities m 1 , . . .
, m r , respectively, and has ¯ m zero singular values.
Recall that ( A 1 , A 2 ) is deﬁned to be a critical point of L if ∇ A 1 L = 0 , ∇ A 2 L = 0 .
Our ﬁrst result provides a full characterization of all critical points of L .
Theorem 1 (Characterization of critical points) .
All critical points of L are necessarily and sufﬁciently characterized by a matrix L 1 ∈ R d 1 × d 0 , a block matrix V ∈ R d 2 × d 1 and an invertible matrix
C ∈ R d 1 × d 1 via
[Equation]
Speciﬁcally, V =
[diag( V 1 , . .
, V r , V ) , 0 d 2 × ( d 1 − rank( A 2 )) ] , where both V i ∈
R m i × p
i and V ∈ R ¯ m × ¯ p consist of orthonormal columns with the number of columns p i ≤ m i , i = 1 , . . .
, r, ¯ p ≤ ¯ m such that (cid:80) ri =1 p i +
¯ p = rank
( A 2 ) , and L 1 , V , C satisfy
[Equation]
Theorem 1 characterizes the necessary and sufﬁcient forms for all critical points of L .
In general, the set of critical points is uncountable and cannot be fully listed out.
Intuitively, the matrix C captures the invariance of the product
A 2 A 1 under an invertible transform, and L 1 captures the degree of freedom of the solution set for linear systems.
However, the analytical forms in eqs.
(1) and (2) do allow one to construct some critical points of L by specifying choices of L 1 , V , C that fulﬁll the condition in eq.
For example, choosing L 1 =guarantees eq.
(3), in which case eqs.
(1) and (2) yield a critical point ( C − 1 V (cid:62) U (cid:62) Y X † , UV C ) for any invertible matrix C and any block matrix V that takes the form speciﬁed in Theorem 1.
For nonzero L 1 , one can ﬁx a proper V and solve the linear equation on C in eq. If a solution exists, we then obtain the form of a corresponding critical point. We further note that the analytical structures of the critical points is more important, which have direct implications on the global optimality conditions and landscape properties as we show in the remaining part of the section. Remark 1.
We note that the block pattern parameters { p i } ri =1 and ¯ p denote the number of columns of { V i } ri =1 and V , respectively, and their sum equals the rank of A 2 , i.e., (cid:80) ri =1 p i + ¯ p = rank( A 2 ) .
The parameters p i , i = 1 , . . .
, r, ¯ p of V contain all useful information of the critical points that determine the function value of L as presented in the following proposition.
Proposition 1.
Any critical point ( A 1 , A 2 ) of L satisﬁes
[Equation]
Proposition 1 evaluates the function value L at a critical point using the parameters { p i } ri =1 .
To explain further, recall that the data matrix Σ has each singular value σ i with multiplicity m i .
For each i , the critical point captures p i out of m i singular values σ i .
Hence, for a σ i with larger value (i.e., a smaller index i ), it is desirable that a critical point captures a larger number p i of them.
In this way, the critical point captures more important principle components of the data so that the value of the loss function is further reduced as suggested by Proposition 1.
In summary, the parameters { p i } ri =1 characterize how well the learned model ﬁts the data in terms of the value of the loss function.
Moreover, the parameters { p i } ri =1 also determine a full characterization of the global minimizers as given below.
Proposition 2 (Characterization of global minimizers) .
A critical point ( A 1 , A 2 ) of L is a global minimizer if and only if it falls into the following two cases.
Case 1: min { d 2 , d 1 } ≤ (cid:80) ri =
m i , A 2 is full rank, and p 1 = m 1 , . .
, p k − 1 = m k − 1 , p k = rank( A 2 ) − (cid:80) k − 1 i =
m i ≤ m k for some k ≤ r ; 2.
Case 2: min { d 2 , d 1 } > (cid:80) ri =
m i , p i
= m i for i = 1 , . . .
, r , and ¯ p ≥ 0 .
In particular, A 2 can be non-full rank with rank( A 2 ) =
(cid:80) ri =
1 m i .
The analytical form of any global minimizer can be obtained from Theorem 1 with further speciﬁcation to the above two cases.
Proposition 2 establishes the neccessary and sufﬁcient conditions for any critical point to be a global minimizer.
If the data matrix Σ has a large number of nonzero singular values, i.e., the ﬁrst case, one needs to exhaust the representation budget (i.e., rank) of A 2 and capture as many large singular values as the rank allows to achieve the global minimum; Otherwise, A 2 of a global minimizer can be non-full rank and still captures all nonzero singular values. Note that A 2 must be full rank in the case 1, and so is A 1 if we further adopt the assumptions on the network size and data matrices in Baldi & Hornik (1989).
Furthermore, the parameters { p i } ri =1 naturally divide all non-global-minimum critical points ( A 1 , A 2 ) of L into the following two categories.
• (Non-optimal order): The matrix V speciﬁed in Theorem 1 satisﬁes that there exists 1 ≤ i < j ≤ r such that p i < m i and p j > 0 .
• (Optimal order): rank( A 2 ) < min { d 2 , d 1 } and the matrix V speciﬁed in Theorem 1 satisﬁes that p 1 = m 1 , . . .
, p k − 1 = m k − 1 , p k = rank( A 2 ) − (cid:80) k − 1 i =
m i ≤ m k for some 1 ≤ k ≤ r .
To understand the above two categories, note that a critical point of L with non-optimal order captures a smaller singular value σ j (since p j > 0 ) while skipping a larger singular value σ i with a lower index i <
j (since p i < m i ), and hence cannot be a global minimizer.
On the other hand, although a critical point of L with optimal order captures the singular values in the optimal (i.e., decreasing) order, it does not fully utilize the representation budget of A 2 (because A 2 is non-full rank) to further capture nonzero singular values and reduce the function value, and hence cannot be a global minimizer either.
Next, we show that these two types of non-global-minimum critical points have different landscape properties around them.
Throughout, a matrix (cid:102) M is called the perturbation of M if it lies in an arbitrarily small neighborhood of M .
Proposition 3 (Landscape around critical points) .
The critical points of L have the following landscape properties.
A non-optimal-order critical point ( A 1 , A 2 ) has a perturbation ( (cid:102)
A 1 , (cid:102)
A 2 ) with rank( (cid:102)
A 2 ) = rank ( A 2 ) , which achieves a lower function value; 2.
An optimal-order critical point ( A 1 , A 2 ) has a perturbation ( (cid:102)
A 1 , (cid:102)
A 2 ) with rank( (cid:102)
A 2 ) = rank ( A 2 ) + 1 , which achieves a lower function value; 3.
Any point in X := { ( A 1 , A 2 ) :
A 2 A 1 X (cid:54) = 0 } has a perturbation ( A 1 , (cid:102)
A 2 ) , which achieves a higher function value;
As a consequence, items 1 and 2 imply that any non-global-minimum critical point has a descent direction, and hence cannot be a local minimizer.
Thus, any local minimizer must be a global minimizer.
Item 3 implies that any point has an ascent direction whenever the output is nonzero.
Hence, there does not exist any local/global maximizer in X .
Furthermore, item 3 together with items 1 and 2 implies that any non-global-minimum critical point in X has both descent and ascent directions, and hence must be a saddle point.
We summarize these facts in the following theorem.
Theorem 2 (Landscape of L ) .
The loss function L satisﬁes: 1) every local minimum is also a global minimum; 2) every non-global-minimum critical point in X is a saddle point.
We note that the saddle points in Theorem 2 can be non-strict when the data matrices are singular.
As an illustrative example, consider the following loss function of a shallow linear network L ( a 2 , a 1 ) =
12 ( a 2 a 1 x − y ) 2 , where a 1 , a 2 , x
and y are all scalars. Consider the case y = 0 . Then, the Hessian at the saddle point a 1 = 0 , a 2 = 1 is [ x 2 , 0; 0 , 0] , which does not have any negative eigenvalue. From a technical point of view, the proof of item 1 of Proposition 3 applies that in Baldi (1989) and generalizes it to the setting where Σ can have repeated singular values and may not be invertible.
To further understand the perturbation scheme from a high level perspective, note that non-optimalorder critical points capture a smaller singular value σ j instead of a larger one σ i with i < j .
Thus, one naturally perturbs the singular vector corresponding to σ j along the direction of the singular vector corresponding to σ i .
Such a perturbation scheme preserves the rank of A 2 and reduces the value of the loss function.
More importantly, the proof of item 2 of Proposition 3 introduces a new technique.
As a comparison,
Baldi & Hornik (1989) proves a similar result as item 2 using the strict convexity of the function, which requires the parameter dimensions to satisfy d 2 = d 0 ≥ d 1 and the data matrices to be invertible.
In contrast, our proof completely removes these restrictions by introducing a new perturbation direction and exploiting the analytical forms of critical points in eqs.
(1) and (2) and the condition in eq.
The accomplishment of the proof further requires careful choices of perturbation parameters as well as judicious manipulations of matrices.
We refer the reader to the supplemental materials for more details.
As a high level understanding, since optimal-order critical points capture the singular values in an optimal (i.e., decreasing) order, the previous perturbation scheme for non-optimal-order critical points does not apply.
Instead, we increase the rank of A 2 by one in a way that the perturbed matrix captures the next singular value beyond the ones that have already been captured so that the value of the loss function can be further reduced.
D EEP L INEAR N EURAL N ETWORKS
In this section, we study deep linear networks with (cid:96) ≥ 2 layers.
We denote the weight parameters between the layers as A k ∈ R d k × d k − 1 for k = 1 , . . .
, (cid:96) , respectively.
The input and output data are denoted by X ∈ R d 0 × m ,
Y ∈ R
d
[Equation]
× m , respectively.
We are interested in the square loss function of deep linear networks, which is given by
[Equation]
Denote Σ k :=
Y ( A ( k, 1) X ) † A ( k, 1) XY (cid:62) for k = 0 , . .
, (cid:96) with the full singular value decomposition U k Λ k U (cid:62) k .
Suppose that Σ k has r ( k ) distinct positive singular values σ 1 ( k ) > · · · > σ r ( k ) ( k ) > 0 with multiplicities m 1 ( k ) , . . .
, m r ( k ) ( k ) , respectively, and ¯ m ( k ) zero singular values.
Our ﬁrst result provides a full characterization of all critical points of L D , where we denote
A ( i,j ) :
=
A i A i − 1 · · · A j +1 A j for notational convenience 1 .
Theorem 3 (Characterization of critical points) .
All critical points of L D are necessarily and sufﬁciently characterized by matrices
L k ∈ R d k × d k − 1 , block matrices V k ∈ R d l × d k +1 and invertible matrices C k ∈ R d k +1 ×
d k +1 for k = 0 , . . .
, (cid:96) − 2 such
that A 1 , . . .
, A (cid:96) can be individually expressed out recursively via the following two equations:
[Equation]
Speciﬁcally, V k
[Equation]
, where V ( k ) i ∈
R m i ( k ) × p
i ( k ) ,
V ( k ) ∈
R ¯ m ( k ) × ¯ p ( k ) consist of orthonormal columns with p i ( k ) ≤ m i ( k ) for i = 1 , . . .
, r ( k ) , ¯ p ( k ) ≤ ¯ m ( k ) such that (cid:80) r ( k ) i =1 p
i ( k ) + ¯ p ( k ) = rank( A (cid:96),k +1 ) , and L k , V k , C k satisfy for k = 2 , . . .
, (cid:96) −
[Equation]
[Equation]
Note that the forms of the individual parameters A 1 , . . .
, A (cid:96) can be obtained as follows by recursively applying eqs.
(4) and (5).
First, eq.
(5) with k = 0 yields the form of A ( (cid:96), 2) .
Then, eq.
(4) with k = 0 and the form of A ( (cid:96), 2) yield the form of A 1 .
Next, eq.
(5) with k = 1 yields the form of A ( (cid:96), 3) , and then, eq.
(4) with k = 1 and the forms of A ( (cid:96), 3) , A 1 further yield the form of A 2 .
Inductively, one obtains the expressions of all individual parameter matrices.
Furthermore, the ﬁrst condition in eq.
(6) is a consistency condition that guarantees that the analytical form for the entire product of parameter matrices factorizes into the forms of individual parameter matrices.
Similarly to shallow linear networks, while the set of critical points here is also uncountable, Theorem 3 suggests ways to obtain some critical points.
For example, if we set L k = 0 for all k (i.e., eq.
(6) is satisﬁed), we can obtain the form of critical points for any invertible C k and proper V k with the structure speciﬁed in Theorem 3.
For nonzero L k , eq.
(6) needs to be veriﬁed for given C k and V k to determine a critical point.
Similarly to shallow linear networks, the parameters { p i (0) } r (0) i =1 , ¯ p (0) determine the value of the loss function at the critical points and further specify the analytical form for the global minimizers, as we present in the following two propositions.
Proposition 4.
Any critical point ( A 1 , . . .
, A (cid:96) ) of L D satisﬁes
[Equation]
identity matrix I .
Proposition 5 (Characterization of global minimizers) .
A critical point ( A 1 , . . .
, A (cid:96) ) of L D is a global minimizer if and only if it falls into the following two cases.
Case 1: min { d (cid:96) , . . .
, d 1 } ≤ (cid:80) r (0) i =
1 m i (0) , A ( (cid:96), 2) achieves the maximal rank, and p 1 (0)
=
m 1 (0) , . .
, p k − 1 (0)
= m k − 1 (0) , p k (0) =
rank( A ( (cid:96), 2) ) − (cid:80)
k − 1 i =
m i (0) ≤ m k (0) for some k ≤ r (0) ; 2.
Case 2: min { d (cid:96) , . . .
, d 1 } > (cid:80) r
(0) i =
1 m i (0) , p i (0)
= m i (0) for all i
= 1 , . .
, r (0) and ¯ p (0) ≥ 0 .
In particular, A
( (cid:96),
) can be non full
rank with rank( A ( (cid:96), 2) )
= (cid:80) r
(0) i =
1 m i (0) .
The analytical form of any global minimizer can be obtained from Theorem 3 with further speciﬁcation to the above two cases.
In particular for case 1, if we further adopt the invertibility assumptions on data matrices as in Baldi & Hornik (1989) and assume that all parameter matrices are square, then all global minima must correspond to full rank parameter matrices.
We next exploit the analytical forms of the critical points to further understand the landscape of the loss function L D .
It has been shown in Kawaguchi (2016) that every local minimum of L D is also a global minimum, under certain conditions on the parameter dimensions and the invertibility of the data matrices.
Here, our characterization of the analytical forms for the critical points allow us to understand such a result from an alternative viewpoint.
The proofs for certain cases (that we discuss below) are simpler and more intuitive, and no assumption is made on the data matrices and dimensions of the network.
Similarly to shallow linear networks, we want to understand the local landscape around the critical points.
However, due to the effect of depth, the critical points of L D are more complicated than those of L .
Among them, we identify the following subsets of the non-global-minimum critical points ( A 1 ,
A (cid:96) )
of L D
• (Deep-non-optimal order): There exist 0 ≤ k ≤ (cid:96) − 2 such that the matrix V k speciﬁed in Theorem 3 satisﬁes that there exist 1 ≤ i < j ≤ r ( k ) such that p i ( k ) <
m i ( k ) and p j ( k ) > 0 .
• (Deep-optimal order): ( A (cid:96) , A (cid:96) − 1 ) is not a global minimizer of L D with A ( (cid:96) − 2 , 1) being ﬁxed, rank( A (cid:96) )
< min { d (cid:96) , d (cid:96) − 1 } , and the matrix V (cid:96) − 2 speciﬁed in Theorem satisﬁes that p 1 ( l − 2) = m 1 ( l − 2) , . .
, p k − 1 ( l − 2)
= m k − 1 ( l − 2) ,
p k ( l − 2) =
rank( A l ) − (cid:80) k − 1 i =
m i ( l − 2) ≤ m k
( l − 2) for some 1 ≤ k ≤ r ( l − 2) .
The following result summarizes the landscape of L D around the above two types of critical points.
Theorem 4 (Landscape of L D ) .
The loss function L D has the following landscape properties.
A deep-non-optimal-order critical point ( A 1 , . . .
, A (cid:96) ) has a perturbation ( A 1 , . . .
, (cid:101)
A k +1 , . . .
, (cid:101)
A (cid:96) ) with rank( (cid:101)
A (cid:96) )
=
rank( A (cid:96) ) , which achieves a lower function value.
A deep-optimal-order critical point ( A 1 , . . .
, A (cid:96) ) has a perturbation ( A 1 , . . .
, (cid:101)
A (cid:96) − 1 , (cid:101)
A (cid:96) ) with rank( (cid:101)
A (cid:96) )
=
rank( A (cid:96) ) + 1 , which achieves a lower function value.
Any point in X D :=
{ ( A 1 , . . .
, A (cid:96) ) :
A ( (cid:96), 1) X (cid:54) = 0 } has a perturbation ( A 1 , . . .
, (cid:101)
A (cid:96) ) that achieves a higher function value.
Consequently, 1) every local minimum of L D is also a global minimum for the above two types of critical points; and 2) every critical point of these two types in X D is a saddle point.
Theorem 4 implies that the landscape of L D for deep linear networks is similar to that of L for shallow linear networks, i.e., the pattern of the parameters { p i ( k ) } r ( k ) i =1 implies different descent directions of the function value around the critical points.
Our approach does not handle the remaining set of non-global minimizers, i.e., there exists q ≤ (cid:96) − 1 such that
( A (cid:96) , . . .
, A q ) is a global minimum point of L D with A ( q − 1 , 1) being ﬁxed, and A ( (cid:96),q ) is of optimal order.
It is unclear how to perturb the intermediate weight parameters using their analytical forms for deep networks , and we leave this as an open problem for the future work.
N ONLINEAR N EURAL N ETWORKS WITH O NE H IDDEN L AYER
In this section, we study nonlinear neural networks with one hidden layer.
In particular, we consider nonlinear networks with ReLU activation function σ : R → R that is deﬁned as σ ( x ) :
= max { x, 0 } .
Our study focuses on the set of differentiable critical points.
The weight parameters between the layers are denoted by A 2 ∈ R d 2 × d 1 , A 1 ∈ R d 1 × d 0 , respectively, and the input and output data are denoted by X ∈ R d 0 × m ,
Y ∈ R d 2 × m , respectively.
We are interested in the square loss function which is given by
[Equation]
where σ acts on A 1 X entrywise.
Existing studies on nonlinear networks characterized the sufﬁcient conditions for critical points being global minimum Gori & Tesi (1992); Nguyen & Hein (2017), established the equivalence between local minimum and global minimum under the condition that d 0 = m Yu & Chen (1995), and provided understanding of geometric properties of the critical points
Tian (2017).
In comparison, our results below provide a full characterization of the critical points of
L N with d 1 = 1 and critical points of L N over certain parameter space with d 1 > 1 , and show that local minimum of L N that is not global minimum can exist.
Since the activation function σ is piecewise linear, the entire parameter space can be partitioned into disjoint cones.
In particular, we consider the set of cones K
I × J where I ⊂ { 1 , . . .
, d 1 } , J ⊂ { 1 , . . .
, m } that satisfy
[Equation]
[Equation]
[Equation]
where “ ≥ ” and “ < ” represent entrywise comparisons.
Within K I × J , the term σ ( A 1 X ) activates only the entries σ ( A 1 X )
I : J , and the corresponding loss function L N is equivalent to
[Equation]
Hence, within K I × J , L N reduces to the loss of a shallow linear network with parameters (( A 2 ) : ,I , ( A 1 ) I, : ) and input & output data pair ( X : ,J , Y : ,J ) .
Note that our results on shallow linear networks in Section 2 are applicable to all parameter dimensions and data matrices.
Thus,
Theorem 1 fully characterizes the forms of critical points of L N in K
I × J .
Moreover, the existence of such critical points can be analytically examined by substituting their forms into eq.
In summary, we obtain the following result, where we denote Σ J :=
Y : ,J X † : ,J X : ,J Y (cid:62) : ,J with the full singular value decomposition U J Λ J U
(cid:62) J , and suppose that Σ J has r ( J ) distinct positive singular values σ 1 ( J ) > · · · > σ r ( J ) ( J ) with multiplicities m 1 , . . .
, m r ( J ) , respectively, and ¯ m ( J ) zero singular values.
Proposition 6 (Characterization of critical points) .
All critical points of L N in K I × J for any I ⊂ { 1 , . . .
, d 1 } , J ⊂ { 1 , . . . , m } are necessarily and sufﬁciently characterized by an L 1 ∈ R | I |× d 0 , a block matrix V ∈ R d 2 ×| I | and an invertible matrix C ∈ R | I |×| I | such that
[Equation]
Speciﬁcally, V =
[diag( V 1 , . .
, V r ( J ) , V ) , 0 d 2 × ( | I |− rank(( A 2 ) : ,I )) ] ,
where V i ∈
R m i × p
i , V ∈
R ¯ m × ¯ p consist of orthonormal columns with p i ≤ m i for i = 1 , . .
, r ( J ) , ¯ p ≤ ¯ m such that (cid:80) r
( J ) i =1 p i +
¯ p = rank(( A 2 ) : ,I ) , and L 1 , V , C satisfy
[Equation]
Moreover, a critical point in K
I × J exists if and only if there exists such C , V , L 1 that
[Equation]
Other entries of A 1 X < 0 .
[Equation]
[Equation] To further illustrate, we consider a special case where the nonlinear network has one unit in the hidden layer, i.e., d 1 = 1 , in which case A 1 and A 2 are row and column vectors, respectively.
Then, the entire parameter space can be partitioned into disjoint cones taking the form of K I × J , and
I = { 1 } is the only nontrivial choice.
We obtain the following result from Proposition 6.
Proposition 7 (Characterization of critical points) .
Consider L N with d 1 = 1 and any J ⊂ { 1 , . . .
, m } .
Then, any nonzero critical point of L N within K { 1 }× J can be necessarily and sufﬁciently characterized by an (cid:96) (cid:62) 1 ∈ R 1 × d 0 , a block unit vector v ∈ R d 2 × 1 and a scalar c ∈ R such that
[Equation]
Speciﬁcally, v is a unit vector that is supported on the entries corresponding to the same singular value of Σ J .
Moreover, a nonzero critical point in K { 1 }× J exists if and only if there exist such c, v , (cid:96)
[Equation]
1 that satisfy
[Equation]
We note that Proposition 7 characterizes both the existence and the forms of critical points of L N over the entire parameter space for nonlinear networks with a single hidden unit.
The condition in eq.
(12) is guaranteed because P ker( v ) = 0 for v (cid:54) = 0 .
To further understand Proposition 7, suppose that there exists a critical point in K { 1 }× J with v being supported on the entries that correspond to the i -th singular value of Σ J .
Then, Propositionimplies that L N =
Tr( Y Y (cid:62) ) − 12 σ i ( J ) .
In particular, the critical point achieves the local minimum 12 Tr( Y Y (cid:62) )
− 12 σ 1 ( J ) in K { 1 }× J with i = 1 .
This is because in this case the critical point is full rank with an optimal order, and hence corresponds to the global minimum of the linear network in eq.
Since the singular values of Σ J may vary with the choice of J , L N may achieve different local minima in different cones.
Thus, local minimum that is not global minimum can exist for L N .
The following proposition concludes this fact by considering a concrete example.
Proposition 8.
For one-hidden-layer nonlinear neural networks with ReLU activation function, there exists local minimum that is not global minimum, and there also exists local maximum.
The above proposition is demonstrated by the following example.
Example 1.
Consider the loss function L N of the nonlinear network with d 2 = d 0 = 2 , and d 1 = 1 .
The input and output data are set to be X =
diag(1 , 1) , Y = diag(2 , 1) .
First, consider the cone K
I × J with I = { 1 } , J = { 1 } .
Calculation yields that Σ J = diag(4 , 0) , and the conditions for existence of critical points in eqs.
(16) and (17) hold if 2 c − 1 ( v ) 1 , : ≥ 0 , ( (cid:96) 1 ) 2 , : < 0 .
Then choosing c = 1 , v = (1 , 0) (cid:62) , (cid:96) 1 = (1 , − 1) (cid:62) yields a local minimum in K
I × J , because the nonzero entry in v corresponds to the largest singular value of Σ J .
Then, calculation shows that the local minimum achieves L N = 12 .
On the other hand, consider the cone K
I × J (cid:48) with
I = { 1 } , J (cid:48) =
{ 2 } , in which Σ J (cid:48) =
diag(0 , 1) .
The conditions for existence of critical points in eqs.
(16) and (17) hold if c − 1 ( v ) 1 , : ≥ 0 , ( (cid:96) 1 ) 1 , : < 0 .
Similarly to the previous case, choosing c = 1 , v = (1 , 0) (cid:62) , (cid:96) 1 = ( − 1 , 0) (cid:62) yields a local minimum that achieves the function value L n = 2 .
Hence, local minimum that is not global minimum does exist.
Moreover, in the cone K I × J (cid:48)(cid:48) with
I = { 1 } , J (cid:48)(cid:48) = ∅ , the function L N remains to be the constant 52 , and all points in this cone are local minimum or local maximum.
Thus, the landscape of the loss function of nonlinear networks is very different from that of the loss function of linear networks.
C ONCLUSION
In this paper, we provide full characterization of the analytical forms of the critical points for the square loss function of three types of neural networks, namely, shallow linear networks, deep linear networks, and shallow ReLU nonlinear networks.
We show that such analytical forms of the critical points have direct implications on the values of the corresponding loss functions, achievement of global minimum, and various landscape properties around these critical points.
As a consequence, the loss function for linear networks has no spurious local minimum, while such point does exist for nonlinear networks with ReLU activation.
In the future, it is interesting to further explore nonlinear neural networks.
In particular, we wish to characterize the analytical form of critical points for deep nonlinear networks and over the full parameter space.
Such results will further facilitate the understanding of the landscape properties around these critical points.
Supplementary Materials P ROOF OF T HEOREM
Notations: For any matrix M , denote vec( M ) as the column vector formed by stacking its columns.
Denote the Kronecker product as “ ⊗ ”.
Then, the following useful relationships hold for any dimension compatible matrices M , U , V , W :
[Equation]
Recall that a point ( A 1 , A 2 ) is a critical point of L if it satisﬁes
[Equation]
We ﬁrst prove eqs.
(1) and (2).
Lemma 1.
Let ( A 2 , A 1 ) be a critical point of L .
Then it must satisfy, for some L 1 ∈ R d 1 × d 0 , that
[Equation] [Equation]
[Equation]
Proof of Lemma 1.
Since ( A 2 , A 1 ) is a critical point of L , eq.
(23) implies that
[Equation]
Applying the vectorizing operator on both sides of the above equation and use the property in eq.
(18), we conclude that
[Equation]
Since vec( A 1 ) is a solution of the above linear equation, it must take the form of the solution of linear systems, i.e., for some L 1 ∈ R d 1 × d 0 , we have
[Equation]
where (i) uses eqs.
(19) and (20) and (ii) uses eq.
Then, eq.
(25) follows by reshaping the vector into a matrix.
Next we prove eq.
Multiplying both sides of eq.
(25) by A 2 on the left and by X on the right and then using eq.
(22), we obtain
[Equation] [Equation]
Also, multiplying both sides of eq.
(24) by A (cid:62) 2 on the right yields that A 2 A 1 XX (cid:62) A (cid:62) 1 A (cid:62) 2 =
Y X (cid:62) A (cid:62) 1 A (cid:62) 2 .
This equation, together with the above expression of A 2 A 1 X , further implies that
[Equation]
Note that P col( A 2 ) Σ P col
( A 2 ) is symmetric.
Thus, we conclude that Σ P col( A 2 )
=
P col
( A 2 ) Σ .
Next, we derive the form of A 2 .
Recall the full singular value decomposition Σ = U Λ U (cid:62) , where Λ is a diagonal matrix with distinct singular values σ 1 > . . .
> σ r > 0 and multiplicities m 1 , . . .
, m r , respectively.
We also assume that there are ¯ m number of zero singular values in Λ .
Using the fact that P col( A 2 )
=
U P col
( U (cid:62) A 2 ) U (cid:62) , the last equality in eq.
(26) reduces to
[Equation]
By the multiplicity pattern of the singular values in Λ , P col( U (cid:62) A 2 ) must be block diagonal.
Specifically, we can write P col
( U (cid:62) A 2 ) =
diag( P 1 , . .
P r , P ) , where P i ∈ R m i ×
m i and P ∈ R
¯ m × ¯ m .
Also, since P col( U (cid:62) A 2 ) is a projection, P 1 , . .
, P r , P must all be projections.
Note that
P col
( U (cid:62) A 2 ) has rank rank( A 2 ) , and suppose that P 1 , . .
, P r , P have ranks p 1 , . . .
, p r , ¯ p , respectively.
Then, we must have p i ≤ m i for i = 1 , . . .
, r , ¯ p ≤ ¯ m and (cid:80) ri =1 p
i + ¯ p = rank( A 2 ) .
Also, note that each projection can be expressed as P i
=
V
i V (cid:62) i with V
i ∈ R m i × p
i , V ∈ R ¯ m × ¯ p consisting of orthonormal columns.
Hence, we can write P col
( U (cid:62) A 2 )
=
(cid:98) V (cid:98) V (cid:62) where (cid:98) V = diag( V 1 , . .
, V r , V ) .
We then conclude that P col
( A 2 )
=
U P col
[Equation] U [Equation]
U
[Equation] Thus, A 2 has the same column space as U (cid:98) V , and there must exist an invertible matrix C ∈ R d 1 × d
such that A 2 =
U [ (cid:98) V , 0 ] C , where 0 ∈ R d 2 × ( d 1 − rank( A 2 )) is a zero matrix.
Denoting V =
[ (cid:98) V , 0 ] , we conclude that A 2 = UV C .
Then, plugging A † 2 = C − 1 V (cid:62) U (cid:62) into eq.
(25) yields the desired form of A 1 .
We now prove eq.
Note that the above proof is based on the equations ∇ A 1 L = 0 , ( ∇ A 2 L ) A (cid:62) 2 = 0 .
Hence, the forms of A 1 , A 2 in eqs.
(1) and (2) need to further satisfy ∇ A 2 L = 0 .
By eq.
(27) and the form of A 2 , we obtain that A 2 A 1 X
= P col
( A 2 ) Y X † X = UV ( UV ) (cid:62) Y X † X .
This expression, together with the form of A 1 in eq.
(1), implies that
[Equation]
where (i) uses the fact that X † XX (cid:62)
=
X (cid:62) , (ii) uses the fact that the block pattern of V is compatible with the multiplicity pattern of the singular values in Λ , and hence V V (cid:62) Λ V = Λ V .
On the other hand, we also obtain that
[Equation]
Thus, to satisfy ∇ A 2 L = 0 in eq.
(24), we require that
[Equation]
which is equivalent to
[Equation]
Lastly, note that ( I − UV ( UV ) (cid:62) )
= P col( UV ) ⊥ , and ( I − V (cid:62) V )
=
P ker( V ) , which concludes the proof.
P ROOF OF P ROPOSITION
By expansion we obtain that L = 12 Tr( Y Y (cid:62) )
− Tr( A 2 A 1 XY (cid:62) ) + 12 Tr( A 2 A 1 XX (cid:62) A (cid:62)
A (cid:62) 2 ) .
Consider any ( A 1 , A 2 ) that satisﬁes eq.
(23), we have shown that such a point also satisﬁes eq.
(27), which further yields that
[Equation]
where (i) follows from the fact that Tr( P col( A 2 ) Σ P col( A 2 ) ) =
Tr( P col( A 2 ) Σ ) , and (ii) uses the fact that P col( A 2 )
=
U P col
( U (cid:62) A 2 ) U (cid:62) .
In particular, a critical point ( A 1 , A 2 ) satisﬁes eq.
Moreover, using the form of the critical point A 2 = UV C , eq.
(28) further becomes
[Equation]
where (i) is due to P
col
( V C )
=
P col( V ) =
V V (cid:62) , and (ii) utilizes the block pattern of V and the multiplicity pattern of Λ that are speciﬁed in Theorem 1.
P ROOF OF P ROPOSITION
(1): Consider a critical point ( A 1 , A 2 ) with the forms given by Theorem 1.
By choosing L 1 = 0 , the condition in eq.
(3) is guaranteed.
Then, we can specify a critical point with any V that satisﬁes the block pattern speciﬁed in Theorem 1, i.e., we can choose any p i , i = 1 , . . .
, r, ¯ p such that p i ≤ m i for i = 1 , . . .
, r , ¯ p ≤ ¯ m and (cid:80) ri =1 p
i + ¯ p = rank( A 2 ) .
Suppose that ( A 1 , A 2 ) is a global minimizer, Proposition 1 gives that L ( A 1 , A 2 )
=
Tr( Y Y (cid:62) ) − 12 (cid:80) ri =1 p i σ i .
Under the condition that min { d 2 , d 1 } ≤ (cid:80) ri =1 m i , the global minimum value is achieved by a full rank Awith rank( A 2 )
=
min { d 2 , d 1 } and p 1 = m 1 , . .
, p k − 1 = m k − 1 , p k = rank( A 2 ) − (cid:80) k − 1 i =
m i ≤ m k for some k ≤ r .
That is, the singular values are selected in a decreasing order to minimize the function value.
(2): If ( A 2 , A 1 ) is a global minimizer and min { d y , d } > (cid:80) ri
=
m i , the global minimum can be achieved by choosing p i
= m i for all
i = 1 , . . .
, r and ¯ p ≥ 0 .
In particular, we do not need a full rank
A 2 to achieve the global minimum.
For example, we can choose rank( A 2 ) =
(cid:80) ri =
m i < min { d y , d } with p i
= m i for all
i = 1 , . . .
, r and ¯ p = 0 .
P ROOF OF P ROPOSITION
We ﬁrst prove item 1.
Consider a non-optimal-order critical point ( A 1 , A 2 ) .
By Theorem 1, we can write A 2 = UV C where V =
[diag( V 1 , . .
, V r , V ) , 0 ] and V i , i = 1 , . . .
, r, V consist of orthonormal columns.
Deﬁne the orthonormal block diagonal matrix
S (cid:62)
[Equation]
where the matrices O 1 , · · · , O r , O are such that each diagonal block forms an orthonormal submatrix.
By construction we have S (cid:62) V =
[diag( I m 1 × p 1 , . . .
, I m r × p r , I ¯ m × ¯ p ) , 0 ] , where I m k × p k corresponds to the ﬁrst p k columns of the identity matrix I m k × m k .
Then, A 2 can be alternatively written as A 2 = USS (cid:62) V C .
Also, denote the columns of US as
[Equation]
Since ( A 1 , A 2 ) is a non-optimal-order critical point, there exists 1 ≤ i < j ≤ r such that p i <
m i and p j > 0 .
Then, consider the following perturbation of US for some (cid:15) > 0 .
[Equation]
with which we further deﬁne the perturbation matrix (cid:101)
A 2 = (cid:102) MS (cid:62) V C .
Also, let the perturbation matrix (cid:101)
A 1 be generated by eq.
(1) with U ← (cid:102) M and V ← S (cid:62) V .
Note that with this construction, ( (cid:101)
A 1 , (cid:101)
A 2 ) satisﬁes eq.
(25), which further implies eq.
(27) for ( (cid:101)
A 1 , (cid:101)
A 2 ) , i.e., (cid:101)
A 2 (cid:101)
A 1 X =
P col( (cid:101)
A 2 ) Y X † X .
Thus, eq. (28) holds for the point ( (cid:101)
A 1 , (cid:101)
A 2 ) , and we obtain that
[Equation]
where the last equality uses the fact that S (cid:62) Λ S = Λ , as can be observed from the block pattern of
S and the multiplicity pattern of Λ .
Also, by the construction of (cid:102) M and the form of S (cid:62) V , a careful calculation shows that only the i, j -th diagonal elements of P col( S (cid:62) U
[Table]
V ) have changed, i.e.,
[Table]
[Equation]
[Table]
[Equation]
As the index i, j correspond to the singular values σ i , σ j , respectively, and σ i > σ j , one obtain that
[Equation]
Thus, the construction of the point ( (cid:101)
A 2 , (cid:101)
A 1 ) achieves a lower function value for any (cid:15) > 0 .
Letting (cid:15) → 0 and noticing that (cid:102) M is a perturbation of US , the point ( (cid:101)
A 2 , (cid:101)
A 1 ) can be in an arbitrary neighborhood of ( A 2 , A 1 ) .
Lastly, note that rank( (cid:101)
A 2 ) =
rank
( A 2 ) .
This completes the proof of item 1.
Next, we prove item 2.
Consider an optimal-order critical point ( A 1 , A 2 ) .
Then, A 2 must be non-full rank, since otherwise a full rank
A 2 with optimal order corresponds to a global minimizer by Proposition 2.
Since there exists some k ≤ r such that p 1 = m 1 , . .
, p k − 1 = m k − 1 , p k = rank( A 2 ) − (cid:80) k − 1 i =
m i ≤ m k , the necessary form of A 2 gives that A 2 = UV C with V =
[diag( V 1 , . .
, V k ) , 0 ] :=
[ V diag , 0 ] .
Using this expression, eq.
(1) yields that
[Equation]
We now specify our perturbation scheme.
Recalling the orthonormal matrix S deﬁned in eq.
[Equation]
Then, we consider the following matrices for some (cid:15) 1 , (cid:15) 2 >
[Equation]
, where
[Equation]
Our goal is to show that L ( (cid:101)
A 1 , (cid:101)
A 2 )
<
L ( A 1 , A 2 ) for (cid:15) 1 , (cid:15) 2 → 0 .
For this purpose, we need to utilize the condition of critical points in eq.
(3), which can be equivalently expressed as
[Equation]
[Equation] [Table]
[Equation]
where (i) follows by taking the transpose and then simplifying, and (ii) uses the fact that V =
SS (cid:62) V = S : , 1:( q − 1) in the case of optimal-order critical point.
Calculating the function value at ( (cid:101)
A 1 , (cid:101)
A 2 ) , we obtain that
[Equation]
We next simplify the above three trace terms using eq.
For the ﬁrst trace term, observe that
[Equation]
where (i) follows from eq.
(31) as S : ,q is orthogonal to the columns of S : , 1:( q − 1) .
For the second trace term
[Equation]
we obtain that
[Equation]
where (i) follows from S (cid:62) : ,q Λ S
=
σ k e (cid:62) q , and (ii) follows from e (cid:62) q S (cid:62) V diag = 0 .
For the third trace term
[Equation]
we obtain that
[Equation]
Combining the expressions for the three trace terms above
[Equation]
we conclude that
[Equation]
diag ) (cid:62) − I ])
( i ) =
{+12 (cid:15) 
+}
Tr(( CL 1 ) (rank( A 2 )+1) , : XX (cid:62) ( CL 1 ) (cid:62) (rank( A 2 )+1) , : )
{++ ( 1 2 (cid:15) 2 1 (cid:15) 2 2 − (cid:15) 1 (cid:15) 2 ) Tr( S (cid:62) : ,q Λ S : ,q ) = 12 (cid:15) 
 Tr(( CL 1 ) (rank( A 2 )+1) , : XX (cid:62) ( CL 1 ) (cid:62) (rank( A 2 )+1) , : ) + ( 12 (cid:15) 21 (cid:15) 22 − (cid:15) 1 (cid:15) 2 ) σ k ,+} where (i) follows from eq.
Note that the ﬁrst term in the last equation is nonnegative.
Now, letting (cid:15) 2 = (cid:15) 21 → 0 , the overall perturbation of the function value becomes {+
 [Tr( P P [Equation] ++}
[Equation]
( P Q (cid:62) ) − 2Tr( P Y (cid:62) )]
= O ( (cid:15) 41 ) − O ( (cid:15) 31 ) < 0 .
Thus, the constructed perturbation ( (cid:101)
A 1 , (cid:101)
A 2 ) achieves a lower function value, and it can be in an arbitrary neighborhood of ( A 1 , A 2 ) as (cid:15)
2 = (cid:15) 21 → 0 .
Lastly, note that rank( (cid:101)
A 2 ) =
rank
( A 2 ) + 1 .
Next, we prove item 3.
We ﬁrst introduce a technical lemma.
Consider row vectors a (cid:62) , y (cid:62) with a (cid:62) (cid:54)
= 0 .
Deﬁne the scalar function f ( α ) =
12 (cid:13)(cid:13)
α a (cid:62) − y (cid:62) (cid:13)(cid:13) 2 2 .
Then, f ( α ) is strongly convex as f (cid:48)(cid:48)
( α ) = (cid:107) a (cid:107) 22 > 0 .
Thus, the following fact holds due to strong convexity.
Fact 1.
For any α ∈ R , we can identify a perturbation (cid:101) α such that f ( (cid:101) α ) >
f ( α ) .
Now consider any point ( A 1 , A 2 ) ∈ X .
Since A 2 A 1 X (cid:54) = 0 , then there exists a certain row, say, the i -th row ( A 2 ) i, : A 1 X , that is nonzero.
We then apply the above fact with α = 1 , a (cid:62) =
( A 2 ) i, : A 1 X , y (cid:62) =
Y i, : , and conclude that one can ﬁnd a perturbation (cid:101) α that achieves a higher function value.
Equivalently, one can treat the perturbation of α as the perturbation of ( A 2 ) i, : , i.e., deﬁne the perturbation ( (cid:101)
A 2 ) i, : := (cid:101)
α
( A 2 ) i, : .
P ROOF OF T HEOREM
We ﬁrst derive the forms of the parameter matrices.
Consider a critical point ( A 1 , . . A
[Equation] of L D .
By deﬁnition of the critical point, we have ∇ A k L D = 0 for all k = 1 , . . .
, (cid:96) , which implies that
[Equation]
Solving this linear system of A k , we obtain that, for some L k ∈ R d k × d k −
[Equation]
Multiplying eq.
(33) on both sides by A ( (cid:96),k +1) on the left and A ( k − 1 , 1) X on the right and then simplifying, we obtain that for all k = 1 , . . .
, (cid:96) −
[Equation]
On the other hand, applying eq.
(32) with k = (cid:96) and multiplying both sides by A (cid:62) (cid:96) on the right, one obtains that
[Equation]
which, together with eq.
(34), further implies that for
[Equation]
Following the same argument as that in the proof of Theorem 1, we conclude that A ( (cid:96),k +1) =
U k − 1 V k − 1 C k − 1 , where U k − 1 , V k − 1 , C k − 1 satisfy the conditions that are stated in the theorem.
Then, plugging the expression of A ( (cid:96),k +1) in eq.
(33), one obtains the form of A k in eq.
We next prove the conditions in eq.
Note that the ﬁrst condition is simply a consistency condition on the matrix products.
This is because eq.
(36) only provides the forms of the matrices
A ( (cid:96),k +1) , k = 1 , . . .
, l − 1 , which must factorize into the product of individual matrices.
For the other condition in eq.
(6), note that the proof of eq.
(35) uses the weaker condition ( ∇ A (cid:96) L D )
A (cid:62) (cid:96) =than the original condition ∇ A (cid:96) L D = 0 of the critical point.
Thus, the forms of the parameter matrices must also satisfy ∇ A (cid:96) L D = 0 , i.e., A ( (cid:96), 1) X
( A ( (cid:96) − 1 , 1) X ) (cid:62) =
Y X (cid:62) A (cid:62) ( (cid:96) − 1 , 1) .
Then, plugging eq.
(34) in the above condition and simplifying, one obtains eq.
P ROOF OF P ROPOSITION
Note that by expansion L D =
[Equation]
2 Tr( Y Y (cid:62) )
− Tr( A ( (cid:96), 1) XY
[Equation]
A
[Equation]
[Equation]
XX
[Equation]
A [Equation] [Equation]
For any
A 1 , . . .
, A (cid:96) that satisfy eq.
(32), we have shown that they must satisfy eq.
(34) with k = 1 , with which we further obtain that
[Equation]
Consider a critical point ( A 1 , . .
A (cid:96) )
so that eq.
(37) holds.
Using the form of critical points
A ( (cid:96), 2) =
U 0 V 0 C 0 , eq.
(37) further becomes
[Equation]
where (i) utilizes the block pattern of V 0 and the multiplicity pattern of Λ 0 that are speciﬁed in
Theorem 3.
P ROOF OF P ROPOSITION
Observe that the product matrix A ( (cid:96), 2) is equivalent to the class of matrices B 2 ∈ R min { d (cid:96) ,...,d 2 }× d 1 .
Consider a critical point ( B 2 , A 1 ) of the shallow linear network L := 12 (cid:107) B 2 A 1 X − Y (cid:107) 2 F .
By
Proposition 2, we conclude that
If min { d (cid:96) , . . . , d 1 } ≤ (cid:80) r (0) i =1 m i (0) , then ( A 1 , B 2 ) is a global minimizer if and only if Bis full rank and p
= m 1 (0) , . .
, p k − 1 = m k − 1 (0) , p k = rank( B 2 ) − (cid:80) k − 1 i =
m i (0) ≤ m k (0) for some k ≤ r (0) ; •
If min { d (cid:96) , . . .
, d 1 } > (cid:80) r
(0) i =
1 m i (0) , then ( A 1 , B 2 ) is a global minimizer if and only if p i
= m i (0) for all i = 1 , . .
, r (0) and ¯ p ( 0) ≥ 0 .
In particular, B 2 can be non-full rank with rank( B 2 ) =
(cid:80) r
(0) i =
1 m i (0) .
Note that L D achieves the same global minimum as L .
Hence Proposition 1 and Proposition 4 must match, which yields that (cid:80) r (0) i =1 p
i σ i (0) =
(cid:80) r
(0) i =1 p i (0) σ i (0) .
We then conclude that p i (0)
=
p i for i = 1 , . . .
, r (0) as σ 1 (0) > · · · > σ r (0) (0) .
This proves the proposition.
P ROOF OF T HEOREM
The proof is similar to that for shallow linear networks.
Consider a deep-non-optimal-order critical point ( A 1 , . .
A (cid:96) ) , and deﬁne the orthonormal block matrix S k using the blocks of V k in a similar way as eq.
Then, A ( l,k +2) takes the form A ( l,k +2)
=
U k S k S (cid:62) k V k C k .
Since A ( l,k +2) is of non-optimal order, there exists i < j < r ( k ) such that p i ( k )
< m i ( k ) and p j ( k ) > 0 .
Thus, we perturb the j -th column of U k S k to be u sj 1 + (cid:15) u si ( pi ( k )+1) √ 1+ (cid:15), and denote the resulting matrix as (cid:102) M k .
Then, we perturb A (cid:96) to be (cid:101)
A (cid:96) =
(cid:102)
M k
( U k S k ) (cid:62) A (cid:96) so that (cid:101)
A (cid:96) A ( (cid:96) − 1 ,k +2)
= (cid:102)
M k S (cid:62) k V k C k .
Moreover, we generate (cid:101)
A k +1 by eq.
(4) with U k ← (cid:102)
M k , V k ← S (cid:62)
k V k .
Note that such construction satisﬁes eq.
(32), and hence also satisﬁes eq.
(34), which further yields that
[Equation]
With the above equation, the function value at this perturbed point is evaluated as
[Equation]
Then, a careful calculation shows that only the i, j -th diagonal elements of P col( S (cid:62) k
U
[Equation]
k
[Equation]
M k S
[Equation]
k V k ) Λ have changed, and are (cid:15) 2 1+ (cid:15) 2 , 1 1+ (cid:15) 2 , respectively.
We then conclude that
L
[Equation]
A
[Equation]
[Equation]
[Equation]
A
[Equation]
−
[Equation]
A
[Equation] A [Equation] Now consider a deep-optimal-order critical point ( A 1 , . .
A (cid:96) ) .
Note that with A ( (cid:96) − 2 , 1) ﬁxed to be a constant, the deep linear network reduces to a shallow linear network with parameters ( A (cid:96) , A (cid:96) − 1 ) .
Since ( A (cid:96) , A (cid:96) − 1 ) is not a non-global minimum critical point of this shallow linear network and A (cid:96) is of optimal-order, we can apply the perturbation scheme in the proof of Proposition 3 to identify a perturbation ( (cid:101)
A (cid:96) , (cid:101)
A (cid:96) − 1 ) with rank( (cid:101)
A (cid:96) )
=
rank( A (cid:96) )
+ 1 that achieves a lower function value.
Consider any point in X D .
Since A ( (cid:96), 1) X (cid:54) = 0 , we can scale the nonzero row, say, the i -th row ( A (cid:96) )
i, : A ( (cid:96) − 1 , 1) X properly in the same way as that in the proof of Proposition 3 to increase the function value.
Lastly, item 1 and item 2 imply that every local minimum is a global minimum for these two types of critical points.
Moreover, combining items 1,2 and 3, we conclude that every critical point of these two types in X D is a saddle point.