S MALL NONLINEARITIES IN ACTIVATION FUNCTIONS CREATE BAD LOCAL MINIMA IN NEURAL NETWORKS
We investigate the loss surface of neural networks.
We prove that even for onehidden-layer networks with “slightest” nonlinearity, the empirical risks have spurious local minima in most cases.
Our results thus indicate that in general “ no spurious local minima ” is a property limited to deep linear networks.
and insights obtained from linear networks may not be robust.
Speciﬁcally, for ReLU(-like) networks we constructively prove that for almost all practical datasets there exist inﬁnitely many local minima.
We also present a counterexample for more general activations (sigmoid, tanh, arctan, ReLU, etc.), for which there exists a bad local minimum.
Our results make the least restrictive assumptions relative to existing results on spurious local optima in neural networks.
We complete our discussion by presenting a comprehensive characterization of global optimality for deep linear networks, which uniﬁes other results on this topic.
I NTRODUCTION
Neural network training reduces to solving nonconvex empirical risk minimization problems, a task that is in general intractable.
But success stories of deep learning suggest that local minima of the empirical risk could be close to global minima.
Choromanska et al.
(2015) use spherical spin-glass models from statistical physics to justify how the size of neural networks may result in local minima that are close to global.
However, due to the complexities introduced by nonlinearity, a rigorous understanding of optimality in deep neural networks remains elusive.
Initial steps towards understanding optimality have focused on deep linear networks.
This area has seen substantial recent progress.
In deep linear networks there is no nonlinear activation; the output is simply a multilinear function of the input.
Baldi & Hornik (1989) prove that some shallow networks have no spurious local minima and Kawaguchi (2016) extends this result to squared error deep linear networks, showing that they only have global minima and saddle points.
Several other works on linear nets have also appeared (Lu & Kawaguchi, 2017; Freeman & Bruna, 2017; Yun et al., 2018; Zhou & Liang, 2018; Laurent & Brecht, 2018a;b).
The theory of nonlinear neural networks (which is the actual setting of interest), however, is still in its infancy.
There have been attempts to extend the “local minima are global” property from linear to nonlinear networks, but recent results suggest that this property does not usually hold (Zhou &
Liang, 2018).
Although not unexpected, rigorously proving such results turns out to be non-trivial, forcing several authors (e.g., Safran & Shamir (2018);
Du et al.
(2018b)
; Wu et al.
to make somewhat unrealistic assumptions (realizability and Gaussianity) on data.
In contrast, we prove existence of spurious local minima under the least restrictive (to our knowledge) assumptions.
Since seemingly subtle changes to assumptions can greatly inﬂuence the analysis as well as the applicability of known results, let us ﬁrst summarize what is known; this will also help provide a better intuitive perspective on our results (as the technical details are somewhat involved).
1. W HAT IS KNOWN SO FAR ?
There is a large and rapidly expanding literature of optimization of neural networks.
Some works focus on the loss surface (Baldi & Hornik, 1989; Yu & Chen, 1995; Kawaguchi, 2016;
Swirszcz et al., 2016; Soudry & Carmon, 2016; Xie et al., 2016; Nguyen & Hein, 2017; 2018; Safran & Shamir,
; Laurent & Brecht, 2018a;
Yun et al., 2018; Zhou & Liang, 2018;
Wu et al., 2018; Liang et al., 2018a;b; Shamir, 2018), while others study the convergence of gradient-based methods for optimizing this loss (Tian, 2017; Brutzkus & Globerson, 2017; Zhong et al., 2017; Soltanolkotabi, 2017;
Li & Yuan, 2017; Du et al., 2018b; Zhang et al., 2018; Brutzkus et al., 2018; Wang et al., 2018; Li & Liang, 2018; Du et al., 2018a;c; Allen-Zhu et al., 2018; Zou et al., 2018;
Zhou et al., 2019).
In particular, our focus is on the loss surface itself, independent of any algorithmic concerns; this is reﬂected in the works summarized below.
For ReLU networks, the works (Swirszcz et al., 2016; Zhou & Liang, 2018) provide counterexample datasets that lead to spurious local minima, dashing hopes of “local implies global” properties.
However, these works fail to provide statements about generic datasets, and one can argue that their setups are limited to isolated pathological examples.
In comparison, our Theorem 1 shows existence of spurious local minima for almost all datasets, a much more general result.
Zhou & Liang (2018) also give characterization of critical points of shallow ReLU networks, but with more than one hidden node the characterization provided is limited to certain regions.
There are also results that study population risk of shallow ReLU networks under an assumption that input data is i.i.d.
Gaussian distributed (Safran & Shamir, 2018; Wu et al., 2018; Du et al., 2018b).
Moreover, these works also assume realizability , i.e., the output data is generated from a neural network with the same architecture as the model one trains, with unknown true parameters.
These assumptions enable one to compute the population risk in a closed form, and ensure that one can always achieve zero loss at global minima.
The authors of Safran & Shamir (2018); Wu et al.
(2018) study the population risk function of the form E
x [( (cid:80) ki =1 ReLU ( w Ti x ) − ReLU ( v Ti x )) 2 ] , where the true parameters v i ’s are orthogonal unit vectors.
Through extensive experiments and computer-assisted local minimality checks, Safran & Shamir (2018) show existence of local minima for k ≥ 6 .
However, this result is empirical and does not have constructive proofs.
Wu et al.
(2018) show that with k = 2 , there is no bad local minima on the manifold (cid:107) w 1 (cid:107) 2 = (cid:107) w 2 (cid:107) 2 = 1 .
Du et al.
(2018b) study population risk of one-hidden-layer CNN.
They show that there can be a spurious local minimum, but gradient descent converges to the global minimum with probability at least 1/4.
Our paper focuses on empirical risk instead of population risk, and does not assume either Gaussianity or realizability.
Theorem 1 1’s assumption on the dataset is that it is not linearly ﬁttable 1 , which is vastly more general and realistic than assuming that input data is Gaussian or that the output is generated from an unknown neural network.
Our results also show that Wu et al.
(2018) fails to extend to empirical risk and non-unit parameter vectors (see the discussion after Theorem 2).
Liang et al.
(2018b) showed that under assumptions on the loss function, data distribution, network structure, and activation function, all local minima of the empirical loss have zero classiﬁcation error in binary classiﬁcation tasks.
The result relies on stringent assumptions, and it is not directly comparable to ours because both “the local minimum has nonzero classiﬁcation error” and “the local minima is spurious” do not imply one another.
Liang et al.
(2018a) proved that adding a parallel network with one exponential hidden node can eliminate all bad local minima.
The result relies on the special parallel structure, whereas we analyze standard fully connected network architecture.
Laurent & Brecht (2018a) studies one-hidden-layer networks with hinge loss for classiﬁcation.
Under linear separability, the authors prove that Leaky-ReLU networks don’t have bad local minima, while ReLU networks do.
Our focus is on regression, and we only make mild assumptions on data.
For deep linear networks, the most relevant result to ours is Laurent & Brecht (2018b).
When all hidden layers are wider than the input or output layers, Laurent & Brecht (2018b) prove that any local minimum of a deep linear network under differentiable convex loss is global.
They prove this by showing a statement about relationship between linear vs. multilinear parametrization.
Our result in
Theorem 4 is strictly more general that their results, and presents a comprehensive characterization.
A different body of literature (Yu & Chen, 1995; Soudry & Carmon, 2016; Xie et al., 2016; Nguyen & Hein, 2017; 2018) considers sufﬁcient conditions for global optimality in nonlinear networks.
These results make certain architectural assumptions (and some technical restrictions) that may not usually apply to realistic networks.
There are also other works on global optimality conditions for specially designed architectures (Haeffele & Vidal, 2017; Feizi et al., 2017).
1. C ONTRIBUTIONS AND S UMMARY OF R ESULTS
We summarize our key contributions more precisely below.
Our work encompasses results for both nonlinear and linear neural networks.
First, we study whether the “local minima are global” property holds for nonlinear networks.
Unfortunately, our results here are negative.
Speciﬁcally, we prove (cid:73) For piecewise linear and nonnegative homogeneous activation functions (e.g., ReLU), we prove in Theorem 1 that if linear models cannot perfectly ﬁt the data, one can construct inﬁnitely many local minima that are not global.
In practice, most datasets are not linearly ﬁttable, hence this result gives a constructive proof of spurious local minima for generic datasets.
In contrast, several existing results either provide only one counterexample (Swirszcz et al., 2016; Zhou & Liang, 2018), or make restrictive assumptions of realizability (Safran & Shamir, 2018; Du et al., 2018b) or linear separability (Laurent & Brecht, 2018a).
This result is presented in Section 2.
(cid:73)
In Theorem 2 we tackle more general nonlinear activation functions, and provide a simple architecture (with squared loss) and dataset, for which there exists a local minimum inferior to the global minimum for a realizable dataset.
Our analysis applies to a wide range of activations, including sigmoid, tanh, arctan, ELU (Clevert et al., 2015), SELU (Klambauer et al., 2017), and
ReLU.
Considering that realizability of data simpliﬁes the analysis and ensures zero loss at global optima, our counterexample that is realizable and yet has a spurious local minimum is surprising, suggesting that the situation is likely worse for non-realizable data.
See Section 3 for details.
We complement our negative results by presenting the following positive result on linear networks:
(cid:73)
Assume that the hidden layers are as wide as either the input or the output, and that the empirical risk
(cid:96) (( W j ) H +1 j =
1 ) equals (cid:96)
( W H +1 W H · · · W 1 ) , where (cid:96) 0 is a differentiable loss function and W
i is the weight matrix for layer i .
Theorem 4 shows if ( ˆ W j ) H +1 j =1 is a critical point of (cid:96) , then its type of stationarity (local min/max, or saddle) is closely related to the behavior of (cid:96) 0 evaluated at the product
ˆ W H +1 · · · ˆ W 1 .
If we additionally assume that any critical point of (cid:96) 0 is a global minimum,
Corollary 5 shows that the empirical risk (cid:96) only has global minima and saddles, and provides a simple condition to distinguish between them.
To the best of our knowledge, this is the most general result on deep linear networks and it subsumes several previous results, e.g., (Kawaguchi, 2016;
Yun et al., 2018;
Zhou & Liang, 2018; Laurent & Brecht, 2018b).
This result is in Section 4.
Notation.
For an integer a ≥ 1 , [ a ] denotes the set of integers from 1 to a (inclusive).
For a vector v , we use [ v ] i to denote its i -th component, while [ v ] [ i ] denotes a vector comprised of the ﬁrst i components of v .
Let 1 ( · ) ( 0 ( · ) ) be the all ones (zeros) column vector or matrix with size ( · ) .
“R E LU- LIKE ” NETWORKS : BAD LOCAL MINIMA EXIST FOR MOST DATA
We study below whether nonlinear neural networks provably have spurious local minima.
We show in § 2 and § 3 that even for extremely simple nonlinear networks, one encounters spurious local minima.
We ﬁrst consider ReLU and ReLU-like networks.
Here, we prove that as long as linear models cannot perfectly ﬁt the data, there exists a local minimum strictly inferior to the global one.
Using nonnegative homogeneity, we can scale the parameters to get inﬁnitely many local minima.
Consider a training dataset that consists of m data points.
The inputs and the outputs are of dimension d x and d y , respectively.
We aggregate these items, and write X ∈ R d x × m as the data matrix and Y ∈ R d y × m as the label matrix.
Consider the 1-hidden-layer neural network ˆ Y = W 2 h ( W 1 X + b 1 1 Tm ) + b 2 1 Tm , where h is a nonlinear activation function, W 2 ∈ R d y × d 1 , b 2 ∈ R
d
y , W 1 ∈ R d 1 × d x , and b 1 ∈ R d 1 .
We analyze the empirical risk with squared loss (cid:96) ( W 1 , W 2 , b 1 , b 2 )=
12 (cid:107)
[Equation]
Next, deﬁne a class of piecewise linear nonnegative homogeneous functions
[Equation]
where s + > 0 , s − ≥
and s + (cid:54) =
s − .
Note that ReLU and Leaky-ReLU are members of this class.
2. M AIN RESULTS AND DISCUSSION
We use the shorthand ˜ X := (cid:2) X T 1 m (cid:3) T ∈ R ( d x +1) × m .
The main result of this section, Theorem 1, considers the case where linear models cannot ﬁt Y , i.e., Y (cid:54) =
R ˜ X for all matrix R .
With ReLU-like activation (1) and a few mild assumptions, Theorem 1 shows that there exist spurious local minima.
Theorem 1.
Suppose that the following conditions hold:
(C1.1) Output dimension is d y = 1 , and linear models
R ˜
X cannot perfectly ﬁt Y .
(C1.2) All the data points x i ’s are distinct.
(C1.3)
The activation function h is ¯ h s + ,s − .
(C1.4)
The hidden layer has at least width 2: d 1 ≥ 2 .
Then, there is a spurious local minimum whose risk is the same as linear least squares model.
Moreover, due to nonnegative homogeneity of ¯ h s + ,s − , there are inﬁnitely many such local minima.
Noticing that most real world datasets cannot be perfectly ﬁt with linear models, Theorem 1 shows that when we use the activation ¯ h s + ,s − , the empirical risk has bad local minima for almost all datasets that one may encounter in practice.
Although it is not very surprising that neural networks have spurious local minima, proving this rigorously is non-trivial.
We provide a constructive and deterministic proof for this problem that holds for general datasets, which is in contrast to experimental results of Safran & Shamir (2018).
We emphasize that Theorem 1 also holds even for “slightest” nonlinearities, e.g., when s + = 1+ (cid:15) and s − = 1 where (cid:15) > 0 is small.
This suggests that the “local min is global” property is limited to the simpliﬁed setting of linear neural networks.
Existing results on squared error loss either provide one counterexample (Swirszcz et al., 2016;
Zhou & Liang, 2018), or assume realizability and Gaussian input (Safran & Shamir, 2018; Du et al., 2018b).
Realizability is an assumption that the output is generated by a network with unknown parameters.
In real datasets, neither input is Gaussian nor output is generated by neural networks; in contrast, our result holds for most realistic situations, and hence delivers useful insight.
There are several results proving sufﬁcient conditions for global optimality of nonlinear neural networks (Soudry & Carmon, 2016; Xie et al., 2016; Nguyen & Hein, 2017).
But they rely on assumptions that the network width scales with the number of data points.
For instance, applying Theorem 3.4 of Nguyen & Hein (2017) to our network proves that if ˜ X has linearly independent columns and other assumptions hold, then any critical point with W 2 (cid:54) = 0 is a global minimum.
However, linearly independent columns already imply row( ˜ X ) =
R m , so even linear models R ˜
X can ﬁt any Y ; i.e., there is less merit in using a complex model to ﬁt Y .
Theorem 1 does not make any structural assumption other than d 1 ≥ 2 , and addresses the case where it is impossible to ﬁt Y with linear models, which is much more realistic.
It is worth comparing our result with Laurent & Brecht (2018a), who use hinge loss based classiﬁcation and assume linear separability to prove “no spurious local minima” for Leaky-ReLU networks.
Their result does not contradict our theorem because the losses are different and we do not assume linear separability.
One might wonder if our theorem holds even with d 1 ≥ m .
Venturi et al.
(2018) showed that onehidden-layer neural networks with d 1 ≥ m doesn’t have spurious valleys, hence there is no strict spurious local minima; however, due to nonnegative homogeneity of ¯ h s + ,s − we only have non-strict local minima.
Based on Bengio et al.
(2006), one might claim that with wide enough hidden layer and random W 1 and b 1 , one can ﬁt any Y ; however, this is not the case, by our assumption that linear models
R ˜
X cannot ﬁt Y .
Note that for any d 1 , there is a non-trivial region (measure > 0 ) in the parameter space where W 1 X + b 1 1 Tm > 0 (entry-wise).
In this region, the output of neural network ˆ Y is still a linear combination of rows of ˜ X , so ˆ Y cannot ﬁt Y ; in fact, it can only do as well as linear models.
We will see in the Step 1 of Section 2.2 that the bad local minimum that we construct “kills” d 1 − 1 neurons; however, killing many neurons is not a necessity, and it is just to simply the exposition.
In fact, any local minimum in the region W 1 X + b 1 1 Tm > 0 is a spurious local minimum.
2. A NALYSIS OF T HEOREM
The proof of the theorem is split into two steps.
First, we prove that there exist local minima ( ˆ W j , ˆ b j ) 2 j =1 whose risk value is the same as the linear least squares solution, and that there are inﬁnitely many such minima.
Second, we will construct a tuple of parameters ( ˜ W j , ˜ b j ) 2 j =1 that has strictly smaller empirical risk than ( ˆ W j , ˆ b j ) 2 j =1 .
Step 1: A local minimum as good as the linear solution.
The main idea here is to exploit the weights from the linear least squares solution, and to tune the parameters so that all inputs to hidden nodes become positive.
Doing so makes the hidden nodes “locally linear,” so that the constructed ( ˆ W j , ˆ b j ) 2 j =1 that produce linear least squares estimates at the output become locally optimal.
Recall that ˜ X
= (cid:2)
X T 1 m (cid:3) T ∈ R
( d x +1)
× m , and deﬁne a linear least squares loss (cid:96)
( R ) :
= 12 (cid:107) R ˜ X − Y (cid:107) 2F that is minimized at ¯ W , so that ∇ (cid:96) 0 ( ¯ W ) =
( ¯ W ˜ X − Y )
˜ X T = 0 .
Since d y = 1 , the solution ¯ W ∈ R d y × ( d x +1) is a row vector.
For all i ∈ [ m ] , let ¯ y i
=
¯ W (cid:2)
x
Ti 1 (cid:3) T be the output of the linear least squares model, and similarly ¯ Y = ¯ W ˜ X .
Let η :=
min
− 1 , 2 min
i ¯ y i } , a negative constant making ¯
y
i − η > 0 for all i .
Deﬁne parameters
[Equation]
where α > 0 is any arbitrary ﬁxed positive constant, [ ¯ W ] [ d x ] gives the ﬁrst d x components of ¯ W , and [ ¯ W ] d x +1 the last component.
Since ¯ y i =
[ ¯ W ] [ d x ] x i + [ ¯ W ] d x +1 , for any i , ˆ W 1 x i + ˆ b
[Equation]
> 0 d(component-wise), given our choice of η .
Thus, all hidden node inputs are positive.
Moreover, ˆ
Y = 1 αs + s + ( α ¯ Y − αη
[Equation]
Tm ) + η 1 Tm = ¯ Y , so that the loss (cid:96) (( ˆ W j , ˆ b j ) 2 j =1 ) = 12 (cid:107) ¯ Y − Y (cid:107) 2F = (cid:96) 0 ( ¯ W ) .
So far, we checked that ( ˆ W j , ˆ b j ) 2 j =1 has the same empirical risk as a linear least squares solution.
It now remains to show that this point is indeed a local minimum of (cid:96) .
To that end, we consider the perturbed parameters ( ˆ W j + ∆ j , ˆ b j + δ j ) 2 j =1 , and check their risk is always larger.
A useful point is that since ¯ W is a minimum of (cid:96)
( R ) =
12 (cid:107) R ˜ X − Y
(cid:107) 2F , we have
[Equation]
so ( ¯ Y − Y )
X T = 0 and ( ¯ Y − Y ) 1 m = 0 .
For small enough perturbations, ( ˆ W 1 +∆ 1 ) x i +(ˆ b 1 + δ 1 ) > 0 still holds for all i .
So, we can observe that
[Equation]
where ˜∆ and ˜ δ are ˜∆ :
= s + ( ˆ W 2 ∆ 1 + ∆ 2 ˆ W 1 + ∆ 2 ∆ 1 ) and ˜ δ :
=
s
+ ( ˆ W 2 δ 1 + ∆ 2 ˆ b 1 + ∆ 2 δ 1 ) + δ 2 ; they are aggregated perturbation terms.
We used (2) to obtain the last equality of (3).
Thus, (cid:96) (( ˆ W j + ∆ j , ˆ b j + δ j ) 2 j =1 ) ≥ (cid:96) (( ˆ W j , ˆ b j ) 2 j =1 ) for small perturbations, proving ( ˆ W j , ˆ b j ) 2 j =1 is indeed a local minimum of (cid:96) .
Since this is true for arbitrary α > 0 , there are inﬁnitely many such local minima.
We can also construct similar local minima by permuting hidden nodes, etc.
Step 2: A point strictly better than the local minimum.
The proof of this step is more involved.
In the previous step, we “pushed” all the input to the hidden nodes to positive side, and took advantage of “local linearity” of the hidden nodes near ( ˆ W j , ˆ b j ) 2 j =1 .
But to construct parameters ( ˜ W j , ˜ b j ) 2 j =1 that have strictly smaller risk than ( ˆ W j , ˆ b j ) 2 j =1 (to prove that ( ˆ W j , ˆ b j ) 2 j =1 is a spurious local minimum), we make the sign of inputs to the hidden nodes different depending on data.
To this end, we sort the indices of data points in increasing order of ¯ y i ; i.e., ¯ y 1 ≤ ¯ y 2 ≤ · · · ≤ ¯ y m .
Deﬁne the set J :=
{ j ∈ [ m − 1] | (cid:80) i ≤ j (¯ y
i − y i ) (cid:54) = 0 , ¯ y j < ¯ y j +1 } .
The remaining construction is divided into two cases: J (cid:54) = ∅ and J = ∅ , whose main ideas are essentially the same.
We present the proof for J (cid:54) = ∅ , and defer the other case to Appendix A2 as it is rarer, and its proof, while instructive for its perturbation argument, is technically too involved.
Case 1: J (cid:54) = ∅ .
Pick any j 0 ∈ J .
We can observe that (cid:80) i ≤ j 0 (
¯ y
i − y i )
= − (cid:80) i>j 0 (
¯ y
i − y i ) , because of (2).
Deﬁne β
= ¯ y
j 0 +¯ y j
+1, so
that ¯ y i − β < 0 for all i ≤ j 0 and ¯ y
i − β > 0 for all i > j 0 .
Then, let γ be a constant satisfying 0 < | γ | ≤ ¯ y j
+1 − ¯ y j 0, whose value will be speciﬁed later.
Since | γ | is small enough, sign(¯ y
i − β ) =
sign(¯ y
i − β + γ ) =
sign(¯ y
i − β − γ ) .
Now select parameters
[Equation]
Recall again that
[ ¯ W ] [ d x ] x i + [ ¯ W ] d x +
=
¯ y i .
For i ≤ j 0 , ¯ y i − β + γ < 0 and − ¯
y i + β + γ > 0 , so
[Equation]
Similarly, for i > j 0 , ¯ y i − β + γ > 0 and − ¯ y i + β + γ < 0 results in ˆ y i = ¯ y i + s + − s − s
+ + s − γ .
Here, we push the outputs ˆ y i of the network by s + − s − s + + s − γ from ¯ y i , and the direction of the “push” varies depending on whether i ≤ j
or i > j 0 .
The empirical risk for this choice of parameters is
[Equation]
Since (cid:80) i ≤ j 0 (¯ y i − y i ) (cid:54) =
and s + (cid:54) =
s − , we can choose sign( γ ) =
sign([ (cid:80) i ≤ j 0 (¯ y
i − y i )]
( s + − s − )) , and choose small | γ | so that (cid:96) (( ˜ W j , ˜ b j ) 2 j =1 ) <
(cid:96)
( ¯ W ) =
(cid:96) (( ˆ W j , ˆ b j ) 2 j =1 ) , proving that ( ˆ W j , ˆ b j ) 2 j = is a spurious local minimum.
C OUNTEREXAMPLE : BAD LOCAL MINIMA FOR MANY ACTIVATIONS
The proof of Theorem 1 crucially exploits the piecewise linearity of the activation functions.
Thus, one may wonder whether the spurious local minima seen there are an artifact of the speciﬁc nonlinearity.
We show below that this is not the case.
We provide a counterexample nonlinear network and a dataset for which a wide range of nonlinear activations result in a local minimum that is strictly inferior to the global minimum with exactly zero empirical risk.
Examples of such activation functions include popular activation functions such as sigmoid, tanh, arctan, ELU, SELU, and ReLU.
We consider again the squared error empirical risk of a one-hidden-layer nonlinear neural network:
[Equation]
where we ﬁx d x = d 1 = 2 and d y = 1 .
Also, let h ( k ) ( x ) be the k -th derivative of h : R (cid:55)→ R , whenever it exists at x .
For short, let h (cid:48) and h (cid:48)(cid:48) denote the ﬁrst and second derivatives.
3. M AIN RESULTS AND DISCUSSION
Theorem 2.
Let the loss (cid:96) (( W j , b j ) 2 j =1 ) and network be as deﬁned above.
Consider the dataset
[Equation]
For this network and dataset the following results hold:
If there exist real numbers v 1 , v 2 , v 3 , v 4 ∈ R such that (C2.1)
h ( v 1 ) h ( v 4 )
=
h ( v 2 ) h ( v 3 ) , and (C2.2) h ( v 1 ) h (cid:0) v 3 + v 4(cid:1)
(cid:54) =
h ( v 3 ) h (cid:0) v 1 + v 2(cid:1) , then there is a tuple
˜ W j , ˜ b j ) 2 j =1 at which (cid:96) equals 0 . 2.
If there
exist
real
numbers
v 1 , v 2 ,
u u
∈ R such that the following conditions
hold: (C2.3) u 1 h ( v 1 ) + u 2 h ( v 2 ) = 13 , (C2.4) h is inﬁnitely differentiable at v 1 and v 2 , (C2.5) there exists a constant c > 0 such that | h ( n ) ( v 1 ) | ≤ c n n ! and | h ( n ) ( v 2 ) | ≤ c n n ! .
(C2.6) ( u 1 h (cid:48) ( v 1 ))
2 + u 1 h (cid:48)(cid:48) ( v 1 )> 0 , (C2.7) ( u 1 h (cid:48) ( v 1 ) u
2 h (cid:48) ( v 2 ))
2 < (( u 1 h (cid:48) ( v 1 ))
2 + u 1 h (cid:48)(cid:48) ( v 1 ))
(( u 2 h (cid:48) ( v 2 ))
2 + u 2 h
(cid:48)(cid:48) ( v 2 )) ,
then there exists a tuple ( ˆ W j , ˆ b j ) 2 j =1 such that the output of the network is the same as the linear least squares model, the risk (cid:96) (( ˆ W j , ˆ b j )
j =1 ) = 1 and ( ˆ W j , ˆ b j ) 2 j
=
is a local minimum of (cid:96) .
Theorem 2 shows that for this architecture and dataset, activations that satisfy (C2.1)–(C2.7) introduce at least one spurious local minimum.
Notice that the empirical risk is zero at the global minimum.
This means that the data X and Y can actually be “generated” by the network, which satisﬁes the realizability assumption that others use (Safran & Shamir, 2018; Du et al., 2018b; Wu et al., 2018).
Notice that our counterexample is “easy to ﬁt,” and yet, there exists a local minimum that is not global.
This leads us to conjecture that with harder datasets, the problems with spurious local minima could be worse.
The proof of Theorem 2 can be found in Appendix A3.
Discussion.
Note that the conditions (C2.1)–(C2.7) only require existence of certain real numbers rather than some global properties of activation h , hence are not as restrictive as they look.
Conditions (C2.1)–(C2.2) come from a choice of tuple ( ˜ W j , ˜ b j ) 2 j =1 that perfectly ﬁts the data.
Condition (C2.3) is necessary for constructing ( ˆ W j , ˆ b j ) 2 j =1 with the same output as the linear least squares model, and Conditions (C2.4)–(C2.7) are needed for showing local minimality of ( ˆ W j , ˆ b j ) 2 j =1 via
Taylor expansions.
The class of functions that satisfy conditions (C2.1)–(C2.7) is quite large, and includes the nonlinear activation functions used in practice.
The next corollary highlights this observation (for a proof with explicit choices of the involved real numbers, please see Appendix A5).
Corollary 3.
For the counterexample in Theorem 2, the set of activation functions satisfying conditions (C2.1)–(C2.7) include sigmoid, tanh, arctan, quadratic, ELU, and SELU.
Admittedly, Theorem 2 and Corollary 3 give one counterexample instead of stating a claim about generic datasets.
Nevertheless, this example shows that for many practical nonlinear activations, the desirable “local minimum is global” property cannot hold even for realizable datasets, suggesting that the situation could be worse for non-realizable ones.
Remark: “ReLU-like” activation functions.
Recall the piecewise linear nonnegative homogeneous activation function ¯ h s + ,s − .
They do not satisfy condition (C2.7), so Theorem 2 cannot be directly applied.
Also, if s − =
(i.e., ReLU), conditions (C2.1)–(C2.2) are also violated.
However, the statements of Theorem 2 hold even for ¯ h s + ,s − , which is shown in Appendix A6.
Recalling again s + = 1 + (cid:15) and s − = 1 , this means that even with the “slightest” nonlinearity in activation function, the network has a global minimum with risk zero while there exists a bad local minimum that performs just as linear least squares models.
In other words, “local minima are global” property is rather brittle and can only hold for linear neural networks.
Another thing to note is that in
Appendix A6, the bias parameters are all zero, for both ( ˜ W j , ˜ b j ) 2 j =1 and ( ˆ W j , ˆ b j ) 2 j =1 .
For models without bias parameters, ( ˆ W j ) 2 j =1 is still a spurious local minimum, thus showing that Wu et al.
(2018) fails to extend to empirical risks and non-unit weight vectors.
G LOBAL OPTIMALITY IN LINEAR NETWORKS
In this section we present our results on deep linear neural networks.
Assuming that the hidden layers are at least as wide as either the input or output, we show that critical points of the loss with a multilinear parameterization inherit the type of critical points of the loss with a linear parameterization.
As a corollary, we show that for differentiable losses whose critical points are globally optimal, deep linear networks have only global minima or saddle points .
Furthermore, we provide an efﬁciently checkable condition for global minimality.
Suppose the network has H hidden layers having widths d 1 , . . .
, d H .
To ease notation, we set d 0 = d x and d H +1 =
d y .
The weights between adjacent layers are kept in matrices W j ∈ R d j × d j −( j ∈ [ H + 1] ), and the output ˆ Y of the network is given by the product of weight matrices with the data matrix: ˆ Y = W H +1 W H · · · W 1 X .
Let ( W j ) H +1 j =1 be the tuple of all weight matrices, and W i : j denote the product W i W i − 1 · · · W j +1 W j for i ≥ j , and the identity for i =
j − 1 .
We consider the empirical risk (cid:96) (( W j ) H +1 j =1 ) , which, for linear networks assumes the form
[Equation]
where (cid:96) 0 is a suitable differentiable loss.
For example, when (cid:96)
( R ) = 12 (cid:107) RX − Y (cid:107) 2F , (cid:96) (( W j ) H +1 j =1 ) =
12 (cid:107) W H +1:1 X − Y
(cid:107) 2F = (cid:96)
( W H +1:1 ) .
Lastly, we write ∇ (cid:96)
( M ) ≡ ∇ R (cid:96)
( R ) | R = M .
Remark: bias terms.
We omit the bias terms b 1 , . . .
, b H +1 here.
This choice is for simplicity; models with bias can be handled by the usual trick of augmenting data and weight matrices.
4. M AIN RESULTS AND DISCUSSION
We are now ready to state our ﬁrst main theorem, whose proof is deferred to Appendix A7.
Theorem 4.
Suppose that for all j , d j ≥ min { d x , d y } , and that the loss (cid:96) is given by (4) , where (cid:96)is differentiable on R d y × d x .
For any critical point ( ˆ W j ) H +1 j =1 of the loss (cid:96) , the following claims hold:
If ∇ (cid:96)
( ˆ W H +1:1 ) (cid:54)
= 0 ,
then ( ˆ W j ) H +1 j =1 is a saddle of (cid:96) .
If ∇ (cid:96)
( ˆ W H +1:1 )
= 0 , then (a) ( ˆ W j ) H +1 j =1 is a local min (max) of (cid:96) if ˆ W H +1:1 is a local min (max) of (cid:96) 0 ; moreover, (b) ( ˆ W j ) H +1 j =1 is a global min (max) of (cid:96) if and only if ˆ W H +1:1 is a global min (max) of (cid:96) 0 .
If there exists j ∗ ∈
[ H + 1] such that ˆ W H
+1:
j ∗ +1 has full row rank and ˆ W j ∗ − 1:1 has full column rank, then ∇ (cid:96)
( ˆ W H +1:1 )
= 0 , so 2(a) and 2(b) hold.
Also, (a) ˆ W H +1:1 is a local min (max) of (cid:96)
if ( ˆ W j ) H +1 j =1 is a local min (max) of (cid:96) .
Let us paraphrase Theorem 4 in words.
In particular, it states that if the hidden layers are “wide enough” so that the product W H +1:1 can attain full rank and if the loss (cid:96) assumes the form (4) for a differentiable loss (cid:96) 0 , then the type (optimal or saddle point) of a critical point ( ˆ W j ) H +1 j =1 of (cid:96) is governed by the behavior of (cid:96) 0 at the product
ˆ W H +1:1 .
Note that for any critical point ( ˆ W j ) H +1 j =1 of the loss (cid:96) , either ∇ (cid:96)
( ˆ W H +1:1 ) (cid:54)
= 0 or ∇ (cid:96)
( ˆ W H +1:1 )
= 0 .
Parts 1 and 2 handle these two cases.
Also observe that the condition in Part 3 implies ∇ (cid:96)
= 0 , so Part 3 is a reﬁnement of Part 2.
A notable fact is that a sufﬁcient condition for Part 3 is ˆ W H
+1: having full rank.
For example, if d x ≥ d y , full-rank ˆ W H +1:1 implies rank( ˆ W H +1:2 ) =
d y , whereby the condition in Part 3 holds with j ∗ = 1 .
If ˆ W H +1:1 is not critical for (cid:96) 0 , then ( ˆ W j ) H +1 j =1 must be a saddle point of (cid:96) .
If ˆ W H +1:1 is a local min/max of (cid:96) 0 , ( ˆ W j ) H +1 j =1 is also a local min/max of (cid:96) .
Notice, however, that Part 2(a) does not address the case of saddle points; when ˆ W H +1:1 is a saddle point of (cid:96) 0 , the tuple ( ˆ W j ) H +1 j =1 can behave arbitrarily.
However, with the condition in Part 3, statements 2(a) and 3(a) hold at the same time, so that ˆ W H +1:1 is a local min/max of (cid:96)
if and only if ( ˆ W j ) H +1 j =1 is a local min/max of (cid:96) .
Observe that the same “if and only if” statement holds for saddle points due to their deﬁnition; in summary, the types (min/max/saddle) of the critical points ( ˆ W j ) H +1 j =
and ˆ W H +1:1 match exactly.
Although Theorem 4 itself is of interest, the following corollary highlights its key implication for deep linear networks.
Corollary 5.
In addition to the assumptions in Theorem 4, assume that any critical point of (cid:96) 0 is a global min (max).
For any critical point ( ˆ W j ) H +1 j =1 of (cid:96) , if ∇ (cid:96)
( ˆ W H +1:1 ) (cid:54)
= 0 ,
then ( ˆ W j ) H +1 j =1 is a saddle of (cid:96) , while if ∇ (cid:96)
( ˆ W H +1:1 )
= 0 , then ( ˆ W j ) H +1 j =1 is a global min (max) of (cid:96) .
Proof
If ∇ (cid:96)
( ˆ W H +1:1 ) (cid:54)
= 0 , then ˆ W H +1:1 is a saddle point by Theorem 4.1.
If ∇ (cid:96)
( ˆ W H +1:1 )
= 0 , then ˆ W H +1:1 is a global min (max) of (cid:96) 0 by assumption.
By Theorem 4.2(b), ( ˆ W j ) H +1 j =1 must be a global min (max) of (cid:96) .
Corollary 5 shows that for any differentiable loss function (cid:96) 0 whose critical points are global minima, the loss (cid:96) has only global minima and saddle points, therefore satisfying the “local minima are global” property.
In other words, for such an (cid:96) 0 , the multilinear re-parametrization introduced by deep linear networks does not introduce any spurious local minima/maxima ; it only introduces saddle points.
Importantly, Corollary 5 also provides a checkable condition that distinguishes global minima from saddle points.
Since (cid:96) is nonconvex, it is remarkable that such a simple necessary and sufﬁcient condition for global optimality is available.
Our result generalizes previous works on linear networks such as Kawaguchi (2016);
Yun et al.
; Zhou & Liang (2018), because it provides conditions for global optimality for a broader range of loss functions without assumptions on datasets.
Laurent & Brecht (2018b) proved that if
( ˆ W j ) H +1 j =1 is a local min of (cid:96) , then ˆ W H
+1:1 is a critical point of (cid:96) 0 .
First, observe that this result is implied by Theorem 4.1.
So our result, which was proved in parallel and independently, is strictly more general.
With additional assumption that critical points of (cid:96) 0 are global minima, Laurent & Brecht (2018b) showed that “local min is global” property holds for linear neural networks;
our
Corollay 5 gives a simple and efﬁcient test condition as well as proving there are only global minima and saddles, which is clearly stronger.
D ISCUSSION AND FUTURE WORK
We investigated the loss surface of deep linear and nonlinear neural networks.
We proved two theorems showing existence of spurious local minima on nonlinear networks, which apply to almost all datasets (Theorem 1) and a wide class of activations (Theorem 2).
We concluded by Theorem 4, showing a general result studying the behavior of critical points in multilinearly parametrized functions, which uniﬁes other existing results on linear neural networks.
Given that spurious local minima are common in neural networks, a valuable future research direction will be investigating how far local minima are from global minima in general, and how the size of the network affects this gap.
Another thing to note is that even though we showed the existence of spurious local minima in the whole parameter space, things can be different in restricted sets of parameter space (e.g., by adding regularizers).
Understanding the loss surface in such sets would be valuable.
Additionally, one can try to show algorithmic/trajectory results of (stochastic) gradient descent.
We hope that our paper will be a stepping stone to such future research.
A CKNOWLEDGMENTS
This work was supported by the DARPA Lagrange Program.
Suvrit Sra also acknowledges support from an Amazon Research Award.
A N OTATION
We ﬁrst list notation used throughout the appendix.
For integers a ≤ b , [ a, b ] denotes the set of integers between them.
We write [ b ] , if a = 1 .
For a vector v , we use [ v ] i to denote its i -th component, while [ v ] [ i ] denotes a vector comprised of the ﬁrst i components of v .
Let 1 d (or 0 d ) be the all ones (zeros) column vector in R d .
For a subspace V ⊆ R d , we denote by V ⊥ its orthogonal complement.
For a matrix A , [ A ] i,j is the ( i, j ) -th entry and [ A ] · ,j its j -th column.
Let σ max ( A ) and σ min ( A ) denote the largest and smallest singular values of A , respectively; row( A ) , col( A ) , rank( A ) , and (cid:107) A (cid:107) F denote respectively the row space, column space, rank, and Frobenius norm of matrix A .
Let null( A ) :
= { v | Av = 0 } and leftnull( A ) :
= { v | v T A = 0 } be the null space and the left-null space of A , respectively.
When A is a square matrix, let tr( A ) be the trace of A .
For matrices
A and B of the same size, (cid:104) A, B (cid:105) =
tr( A T B ) denotes the usual trace inner product of A and B .
Equivalently, (cid:104) A, B (cid:105) =
tr( A T B ) = tr( AB T ) .
Let 0 d × m be the all zeros matrix in R d × m .
A P ROOF OF T HEOREM 1, S TEP 2, C ASE
Case 2.
J = ∅ .
We start with a lemma discussing what J = ∅ implies.
Lemma A.1.
If J = ∅ , the following statements hold:
There are some ¯ y j ’s that are duplicate; i.e. for some i (cid:54) =
j , ¯ y i = ¯
y j . 2.
If ¯ y j is non-duplicate, meaning that ¯ y j − 1 < ¯ y j < ¯ y j +1 , ¯ y j = y j holds.
If ¯ y j is duplicate, (cid:80) i :¯ y i =
¯ y j (¯ y i − y i )
= 0 holds.
There exists at least one duplicate ¯ y j such that, for that ¯ y j , there exist at least two different i ’s that satisfy
¯ y i
=
¯ y j and ¯
y i (cid:54) =
y i .
Proof
We prove this by showing if any of these statements are not true, then we have J (cid:54) = ∅ or a contradiction.
If all the ¯ y j ’s are distinct and J = ∅ , by deﬁnition of J , ¯ y j = y j for all j .
This violates our assumption that linear models cannot perfectly ﬁt Y .
If we have ¯ y j (cid:54) =
y j for a non-duplicate ¯ y j , at least one of the following statements must hold: (cid:80) i ≤ j − 1 (¯ y i − y i ) (cid:54) = 0 or (cid:80) i ≤ j (¯ y i − y i )
(cid:54) = 0 , meaning that j − 1 ∈ J or j ∈ J . 3.
Suppose ¯ y j is duplicate and
(cid:80) i :¯ y i =
¯ y j (¯ y i − y i )
(cid:54) = 0 .
Let k = min
{ i | ¯ y i =
¯ y j } and l = max { i | ¯ y i =
¯ y j } .
Then at least one of the following statements must hold: (cid:80) i ≤ k − 1 (¯ y i − y i ) (cid:54) = 0 or (cid:80) i ≤ l (¯ y
i − y i ) (cid:54) = 0 .
If (cid:80) i ≤ k − 1 (¯ y i − y i ) (cid:54) = 0 , we can also see that ¯ y k − 1 < ¯ y k , so k − 1 ∈ J .
Similarly, if (cid:80) i ≤ l (¯ y i − y i )
(cid:54) = 0 , then l ∈ J .
Since (cid:80) i :¯ y i =¯ y j (¯ y
i − y i )
= 0 holds for any duplicate ¯ y j , if ¯ y i (cid:54) =
y i holds for one i then there must be at least two of them that satisﬁes
¯
y i (cid:54) =
y i .
If this doesn’t hold for all duplicate ¯ y i , with Part 2 this means that ¯ y j = y j holds for all j .
This violates our assumption that linear models cannot perfectly ﬁt Y .
From Lemma A.1.4, we saw that there is a duplicate value of ¯ y j such that some of the data points i satisfy ¯
y i
=
¯ y j and ¯
y i (cid:54) =
y i .
The proof strategy in this case is essentially the same, but the difference is that we choose one of such duplicate ¯ y j , and then choose a vector v ∈ R d x to “perturb” the linear least squares solution [ ¯ W ] [ d x ] in order to break the tie between i ’s that satisﬁes
¯ y i =
¯ y j and ¯
y i (cid:54) =
y i .
We start by deﬁning the minimum among such duplicate values ¯ y ∗ of ¯ y j ’s, and a set of indices j that satisﬁes ¯ y j = ¯ y ∗ .
¯ y ∗ =
min { ¯ y j
| ∃ i (cid:54) =
j such that ¯ y i =
¯ y j and ¯
y i (cid:54) =
y i } ,
[Equation]
Then, we deﬁne a subset of J ∗ :
[Equation]
By Lemma A.1.4, cardinality of J ∗ (cid:54) = is at least two.
Then, we deﬁne a special index in J ∗ (cid:54) = :
[Equation]
= argmax
[Equation]
Index j 1 is the index of the “longest” x j among elements in J ∗ (cid:54) = .
Using the deﬁnition of j 1 , we can partition J ∗ into two sets:
[Equation]
For the indices in J ∗ , we can always switch the indices without loss of generality.
So we can assume that j ≤ j 1 =
max J ∗≥ for all j ∈ J ∗≥ and j > j 1 for all j ∈ J ∗ < .
We now deﬁne a vector that will be used as the “perturbation” to [ ¯ W ] [ d x ] .
Deﬁne a vector v ∈ R d x , which is a scaled version of x j 1 :
[Equation]
where the constants g and M are deﬁned to be
[Equation]
The constant M is the largest (cid:107) x
i (cid:107) 2 among all the indices, and g is one fourth times the minimum gap between all distinct values of ¯ y i .
Now, consider perturbing [ ¯ W ] [ d x ] by a vector − αv T .
where α ∈ (0 , 1] will be speciﬁed later.
Observe that
[Equation]
Recall that j ≤ j 1 = max J ∗≥ for all j ∈ J ∗≥ and j > j 1 for all j ∈ J ∗ < .
We are now ready to present the following lemma:
Lemma A.2.
Deﬁne
[Equation]
Then, ¯ y i − αv T x
i − β < 0 for all i ≤ j 1 ,
¯ y
i − αv T x
i − β > 0 for all
i > j 1 .
Also, (cid:80) i>j 1 (¯ y i − y i ) − (cid:80) i ≤ j 1 (¯ y i − y i )
= − 2(¯ y j 1 − y j 1 ) (cid:54) = 0 .
Proof
First observe that, for any x i , | αv T x i | ≤ α (cid:107) v (cid:107) 2 (cid:107) x
i (cid:107) 2 ≤ gM (cid:107) x i (cid:107) 2 ≤ g .
By deﬁnition of g , we have 2 g < ¯ y j − ¯ y i for any ¯
y i < ¯ y j .
Using this, we can see that
[Equation]
In words, if ¯ y i and ¯ y j are distinct and there is an order
¯ y
i < ¯ y j , perturbation of [ ¯ W ] [ d x ] by − αv T does not change the order.
Also, since v is only a scaled version of x j 1 , from the deﬁnitions of J ∗≥ and J ∗ < , v T ( x j − x j 1 ) ≥ 0 for j ∈ J ∗≥ and v T ( x j − x j 1 ) < 0 for j ∈ J ∗ < .
(A.2)
By deﬁnition of j 2 , v
[Equation]
and
[Equation]
all j ∈ J ∗ < .
(A.3)
It is left to prove the statement of the lemma using case analysis, using the inequalities (A.1), (A.2), and (A.3).
For all i ’s such that ¯
y i < ¯ y ∗ =
¯ y j 1 ,
[Equation]
Similarly, for all i such that ¯ y i > ¯
[Equation]
For j ∈ J ∗≥ ( j ≤ j 1 ), we know ¯ y j
= ¯ y ∗ , so
[Equation]
Also, for j ∈ J ∗ < ( j > j 1 ),
[Equation]
This ﬁnishes the case analysis and proves the ﬁrst statements of the lemma.
One last thing to prove is that (cid:80) i>j 1 (¯ y
i − y i ) − (cid:80) i ≤ j 1 (¯ y i − y i )
= − 2(¯ y j 1 − y j 1 ) (cid:54) = 0 .
Recall from
Lemma A.1.2 that for non-duplicate ¯ y j , we have ¯ y j = y j .
Also by Lemma A.1.3 if ¯ y j is duplicate, (cid:80) i :¯ y i =
¯ y j (¯ y i − y i )
= 0 .
So, (cid:88) i>
j(¯ y i − y i ) − (cid:88)
i ≤ j(¯ y
i − y i )
= (cid:88)
i ∈J ∗ <
(¯ y i − y i ) − (cid:88)
i ∈J ∗≥ (¯ y
i − y i ) .
Recall the deﬁnition of J ∗ (cid:54)
=
=
{ j ∈ J ∗ | ¯ y j (cid:54) =
y j } .
For j ∈ J ∗ \J ∗ (cid:54)
= ,
¯ y j = y j .
So, (cid:88) i ∈J ∗ <
(¯ y i − y i ) − (cid:88)
i ∈J ∗≥ (¯ y
i − y i )
= (cid:88)
i ∈J ∗ < ∩J ∗(cid:54) =
(¯ y i − y i ) − (cid:88) i ∈J ∗≥ ∩J ∗(cid:54)
=
(¯ y i − y i ) .
Recall the deﬁnition of j 1 = argmax j ∈J ∗(cid:54)
= (cid:107) x j (cid:107) 2 .
For any other j ∈ J ∗ (cid:54)
=
\{ j 1 } , (cid:107) x
j 1 (cid:107) 22 ≥ (cid:107) x j (cid:107) 2 (cid:107) x
j 1 (cid:107) 2 ≥ (cid:104) x
j , x j 1 (cid:105) , where the ﬁrst ≥ sign is due to deﬁnition of j 1 , and the second is from Cauchy-Schwarz inequality.
Since x j 1 and x j are distinct by assumption, they must differ in either length or direction, or both.
So, we can check that at least one of “ ≥ ” must be strict inequality, so (cid:107) x j 1 (cid:107) 22 > (cid:104) x
j , x j 1 (cid:105) for all j ∈ J ∗ (cid:54) =
\{ j 1 } .
Thus,
[Equation]
and {
[Equation]
proving that
[Equation]
Also, by Lemma A.1.3,
[Equation]
Wrapping up all the equalities, we can conclude that
[Equation]
ﬁnishing the proof of the last statement.
It is time to present the parameters ( ˜ W j , ˜ b j ) 2 j =1 , whose empirical risk is strictly smaller than the local minimum ( ˆ W j , ˆ b j ) 2 j =1 with a sufﬁciently small choice of α ∈ (0 , 1] .
Now, let γ be a constant such that
[Equation]
Its absolute value is proportional to α ∈ (0 , 1] , which is a undetermined number that will be speciﬁed at the end of the proof.
Since | γ | is small enough, we can check that
[Equation]
Then, assign parameter values
[Equation]
With these parameter values,
[Equation]
As we saw in Lemma A.2, for i ≤ j 1 , ¯ y
i − αv T x
i − β + γ < 0 and − ¯
y i +
αv T x
i + β + γ > 0 .
So ˆ y i =
˜
W 2 ¯ h s + ,s − ( ˜ W
[Equation]
Similarly, for i > j
[Equation]
¯ y
i − αv T x
i − β + γ > 0 and − ¯
y i +
αv T x
i + β + γ < 0 ,
so
[Equation]
Now
[Equation]
the squared error loss of this point is
[Equation]
Recall that (cid:80) mi =1 (¯ y i − y i ) x
Ti = 0 for least squares estimates ¯ y i .
From Lemma A.2, we saw that (cid:80) i>j 1 (¯ y i − y i ) − (cid:80) i ≤ j 1 (¯ y i − y i )
=
− 2(¯ y j 1 − y j 1 ) .
As seen in the deﬁnition of γ (A.4), the magnitude of γ is proportional to α .
Substituting (A.4), we can express the loss as
[Equation]
Recall that v T ( x j 1 − x j 2 ) > 0 from (A.3).
Then, for sufﬁciently small α ∈ (0 , 1] , (cid:96) (( ˜ W j , ˜ b j ) 2 j =1 ) <
(cid:96)
( ¯ W ) =
(cid:96) (( ˆ W j , ˆ b j ) 2 j =1 ) , therefore proving that ( ˆ W j , ˆ b j ) 2 j =1 is a spurious local minimum.
A P ROOF OF T HEOREMA3. P ROOF OF P ART
Given v 1 , v 2 , v 3 , v 4 ∈ R satisfying conditions (C2.1) and (C2.2), we can pick parameter values ( ˜ W j , ˜ b j ) 2 j =1 to perfectly ﬁt the given dataset:
[Equation]
With these values, we can check that ˆ Y =
[0 0 1] , hence perfectly ﬁtting Y , thus the loss (cid:96) (( ˜ W j , ˜ b j )
j =1 ) = 0 .
A3. P ROOF OF P ART
Given conditions (C2.3)–(C2.7) on v 1 , v 2 , u 1 , u 2 ∈ R , we prove below that there exists a local minimum ( ˆ W j , ˆ b j ) 2 j =1 for which the output of the network is the same as linear least squares model, and its empirical risk is (cid:96) (( ˆ W j , ˆ b j ) 2 j =1 ) = 1 3 .
If the conditions of Part 1 also hold, this local minimum is strictly inferior to the global one.
First, compute the output ¯ Y of linear least squares model to obtain ¯ Y = (cid:2) 13 (cid:3) .
Now assign parameter values
[Equation]
With these values we can check that ˆ Y = (cid:2) 13 (cid:3) , under condition (C2.3): u 1 h ( v 1 )+ u 2 h ( v 2 ) = 13 .
The empirical risk is (cid:96) (( ˆ W j , ˆ b j ) 2 j =1 ) =
12 ( 19 + 19 + 49 ) = 13 .
It remains to show that this is indeed a local minimum of (cid:96) .
To show this, we apply perturbations to the parameters to see if the risk after perturbation is greater than or equal to (cid:96) (( ˆ W j , ˆ b j ) 2 j =1 ) .
Let the perturbed parameters be
[Equation]
where δ 11 , δ 12 , δ 21 , δ 22 , β 1 , β 2 , (cid:15) 1 , (cid:15) 2 , and γ are small real numbers.
The next lemma rearranges the terms in (cid:96) (( ˇ W j , ˇ b j ) 2 j =1 ) into a form that helps us prove local minimality of ( ˆ W j , ˆ b j ) 2 j =1 .
Appendix A4 gives the proof of Lemma A.3, which includes as a byproduct some equalities on polynomials that may be of wider interest.
Lemma A.3.
Assume there exist real numbers v 1 , v 2 , u 1 , u 2 such that conditions (C2.3)–(C2.5) hold.
Then, for perturbed parameters ( ˇ W j , ˇ b j ) 2 j =1 deﬁned in (A.5) , (cid:96) (( ˇ W j , ˇ b j )
[Equation]
j =1 ) ≥
[Equation]
+
[Equation] [Equation]
[Equation] [Equation]
[Equation]
+ α 3 ( δ 11 − δ 12 )( δ 21 − δ 22 ) , (A.6) where α i =
u
i h (cid:48)(cid:48) ( v i )+ u 2 i ( h (cid:48) ( v i ))
+ o (1) , for i = 1 , 2 , and α 3 = u 1 u 2 h (cid:48) ( v 1 ) h (cid:48)
( v 2 )
+ o (1) , and
o (1) contains terms that diminish to zero as perturbations vanish.
To make the the sum of the last three terms of (A.6) nonnegative, we need to satisfy α 1 ≥and α 23 −
α 1 α 2 ≤ 0 ; these inequalities are satisﬁed for small enough perturbations because of conditions (C2.6)–(C2.7).
Thus, we conclude that (cid:96) (( ˇ W j , ˇ b j ) 2 j =1 ) ≥
=
(cid:96) (( ˆ W j , ˆ b j ) 2 j =1 ) for small enough perturbations, proving that ( ˆ W j , ˆ b j ) 2 j =1 is a local minimum.
A P ROOF OF L EMMA A.
The goal of this lemma is to prove that
[Equation]
[Equation] [Equation]
where o (1) contains terms that diminish to zero as perturbations decrease.
Using the perturbed parameters,
[Equation]
so the empirical risk can be expressed as
[Equation]
So, the empirical risk (A.8) consists of three terms, one for each training example.
By expanding the activation function h using Taylor series expansion and doing algebraic manipulations, we will derive the equation (A.7) from (A.8).
Using the Taylor series expansion, we can express h ( v 1 + δ 11 + β 1 ) as
[Equation]
Using a similar expansion for h ( v 2 + δ 21 + β 2 ) , the ﬁrst term of (A.8) can be written as
[Equation]
2 ) ∞ (cid:88) n = h ( n ) ( v 2 ) n !
( δ 21 + β 2 ) n + γ (cid:35), where we used u 1 h ( v 1 )+ u 2 h ( v 2 ) = 13 .
To simplify notation, let us introduce the following function: t ( δ 1 , δ 2 )
= (cid:15) 1 h ( v 1 ) + (cid:15) 2 h ( v 2 ) + γ + ( u 1 + (cid:15) 1 ) ∞ (cid:88) n = h ( n ) ( v 1 ) n !
( δ 1 + β 1 ) n + ( u 2 + (cid:15) 2 ) ∞ (cid:88) n = h ( n ) ( v 2 ) n !
( δ 2 + β 2 ) n .
With this new notation t ( δ 1 , δ 2 ) , after doing similar expansions to the other terms of (A.8), we get
[Equation]
=  (cid:20)3 + t ( δ 11 , δ 21 ) (cid:21)+ (cid:20)3 + t ( δ 12 , δ 22 )
(cid:21)+ (cid:20) −
+ t (cid:18)
δ 11 + δ , δ 21 + δ (cid:19)(cid:21)=3 +  (cid:20) t ( δ 11 , δ 21 ) + t ( δ 12 , δ 22 ) − 2 t (cid:18)
δ 11 + δ , δ 21 + δ (cid:19)(cid:21) +
[ t ( δ 11 , δ 21 )] 2 + 1
[ t ( δ 12 , δ 22 )] 2 + 1(cid:20) t (cid:18)
δ 11 + δ , δ 21 + δ (cid:19)(cid:21)
[Equation]
Before we show the lower bounds, we ﬁrst present the following lemmas that will prove useful shortly.
These are simple yet interesting lemmas that might be of independent interest.
Lemma A.4.
For n ≥ 2 ,
[Equation]
where p n is a polynomial in a and b .
All terms in p n have degree exactly n − 2 .
When n = 2 , p 2 ( a, b ) = 12 .
Proof
The exact formula for p n ( a, b ) is as the following:
[Equation]
Using this, we can check the lemma is correct just by expanding both sides of the equation.
The rest of the proof is straightforward but involves some complicated algebra.
So, we omit the details for simplicity.
Lemma A.5.
For n 1 , n 2 ≥ 1 ,
[Equation]
where q n 1 ,n 2 and r n 1 ,n 2 are polynomials in a , b , c and d .
All terms in q n 1 ,n 2 and r n 1 ,n 2 have degree exactly n 1 + n 2 − 2 .
When n 1 = n 2 = 1 , q 1 , 1 ( a, b, d ) = 0 and r 1 , 1 ( a, b, c, d ) = 12 .
Proof
The exact formulas for q n 1 ,n 2 ( a, b, d ) , q n 2 ,n 1 ( c, d, b ) , and r n 1 ,n 2 ( a, b, c, d ) are as the following:
[Equation]
Similarly, we can check the lemma is correct just by expanding both sides of the equation.
The remaining part of the proof is straightforward, so we will omit the details.
Using Lemmas A.4 and A.5, we will expand and simplify the “cross terms” part and “squared terms” part of (A.9).
For the “cross terms” in (A.9), let us split t ( δ 1 , δ 2 ) into two functions t 1 and t 2 : t 1 ( δ 1 , δ 2 ) =
(cid:15) 1 h ( v 1 ) + (cid:15)
[Equation]
( v
[Equation]
+
[Equation]
( u
[Equation] [Equation]
[Equation]
1 ) + ( u 2 + (cid:15) 2 )
[Equation]
(cid:48) (
[Equation] [Equation]
[Equation]
2 ) t 2 ( δ 1 , δ 2 )
[Equation]
( u 1 + (cid:15) 1 )
[Equation] [Equation]
so that t ( δ 1 , δ 2 ) =
t 1 ( δ 1 , δ 2 ) + t 2 ( δ 1 , δ 2 ) .
It is easy to check that
[Equation]
Also, using Lemma A.4, we can see that
[Equation] [Equation]
Consider the summation
[Equation]
We assumed that there exists a constant c > 0 such that | h ( n ) ( v 1 ) | ≤ c n n ! .
From this, for small enough perturbations δ 11 , δ 12 , and β 1 , we can see that the summation converges, and the summands converge to zero as n increases.
Because all the terms in p n ( n ≥ 3 ) are of degree at least one, we can thus write
[Equation]
So, for small enough δ 11 , δ 12 , and β 1 , the term h
[Equation]
dominates the summation.
Similarly, as long as δ 21 , δ 22 , and β 2 are small enough, the summation (cid:80) ∞ n =2 h ( n ) ( v 2 ) n !
p n ( δ 21 + β 2 , δ 22 + β 2 ) is dominated by h (cid:48)(cid:48) ( v 2 ).
In conclusion, for small enough perturbations, t ( δ 11 , δ 21 ) + t ( δ 12 , δ 22 ) − 2 t (cid:18)
δ 11 + δ , δ 21 + δ (cid:19) =
t 2 ( δ 11 , δ 21 ) + t 2 ( δ 12 , δ 22 ) − 2 t(cid:18)
δ 11 + δ , δ 21 + δ (cid:19) =(
u 1 + o (1))
(cid:18) h (cid:48)(cid:48) ( v 1 ) + o (1) (cid:19) ( δ 11 − δ 12 )
+ ( u 2 + o (1))
(cid:18) h (cid:48)(cid:48) ( v 2 ) + o (1) (cid:19) ( δ 21 − δ 22 )=
(cid:18) u 1 h (cid:48)(cid:48) ( v 1 ) + o (1) (cid:19) ( δ 11 − δ 12 ) 2 + (cid:18) u 2 h (cid:48)(cid:48) ( v 2 )
+ o (1) (cid:19) ( δ 21 − δ 22 ) 2 .
(A.10)
Now, it is time to take care of the “squared terms.”
We will express the terms as
[Equation] [Equation]
=
[Equation]
t
[Equation]
and expand and simplify the terms in
[Equation]
This time
[Equation]
we split t ( δ 1 , δ 2 ) in another way
[Equation]
this time into three parts:
[Equation] [Equation]
so that t ( δ 1 , δ 2 ) =
t 3 + t 4 ( δ 1 ) + t 5 ( δ 2 ) .
With this,
[Equation]
We now have to simplify the equation term by term.
We ﬁrst note that t 4 ( δ 11 ) + t 4 ( δ 12 ) − 2 t(cid:18)
δ 11 + δ (cid:19) + t 5 ( δ 21 ) + t 5 ( δ 22 ) − 2 t(cid:18)
δ 21 + δ (cid:19) =
t 2 ( δ 11 , δ 21 ) + t 2 ( δ 12 , δ 22 ) − 2 t(cid:18)
δ 11 + δ , δ 21 + δ (cid:19) , so t(cid:20) t 4 ( δ 11 ) + t 4 ( δ 12 ) − 2 t(cid:18)
δ 11 + δ (cid:19) + t 5 ( δ 21 ) + t 5 ( δ 22 ) − 2 t(cid:18)
δ 21 + δ (cid:19)(cid:21) =
t(cid:20) t 2 ( δ 11 , δ 21 ) + t 2 ( δ 12 , δ 22 ) − 2 t(cid:18)
δ 11 + δ , δ 21 + δ (cid:19)(cid:21) =
o (1) (cid:20)(cid:18) u 1 h (cid:48)(cid:48) ( v 1 ) + o (1) (cid:19) ( δ 11 − δ 12 ) 2 + (cid:18) u
2 h (cid:48)(cid:48) ( v 2 )
+ o (1) (cid:19) ( δ 21 − δ 22 )(cid:21) , (A.13) as seen in (A.10).
Next, we have ( t 4 ( δ 11 ))
+ ( t 4 ( δ 12 )) 2 −(cid:18)
t(cid:18)
δ
+ δ (cid:19)(cid:19)=( u 1 + (cid:15) 1 )∞ (cid:88) n 1 ,n 2 = h ( n 1 ) ( v 1 ) h ( n 2 ) ( v 1 ) n 1 !
n 2 ! (cid:34) ( δ
[Equation]
) n
[Equation]
( δ
[Equation] [Equation]
(cid:18)
[Equation]
+ β
[Equation]
(cid:19) n 1 + n 2 (cid:35) , =( u 1 + (cid:15) 1 ) 2 ( δ 11 − δ 12 )∞ (cid:88) n 1 ,n 2 = h ( n 1 ) ( v 1 ) h ( n 2 ) ( v 1 ) n
[Equation]
n
[Equation]
p n
[Equation]
( δ
[Equation]
β
[Equation]
, δ
[Equation]
β 1 )
= (cid:18)
u 21 ( h (cid:48) ( v 1 ))
+ o (1) (cid:19) ( δ 11 − δ 12 ) 2 , (A.14) when perturbations are small enough.
We again used Lemma A.4 in the second equality sign, and the facts that p n 1 + n 2 ( · ) =
o (1) whenever n 1 + n 2 > 2 and that p 2 ( · ) = 12 .
In a similar way, ( t 5 ( δ 21 )) 2 + ( t 5 ( δ 22 )) 2 −(cid:18)
t(cid:18)
δ
+ δ (cid:19)(cid:19)= (cid:18) u 22 ( h (cid:48) ( v 2 ))
+ o (1) (cid:19) ( δ 21 − δ 22 ) 2 .
(A.15)
Lastly, t 4 ( δ 11 ) t 5 ( δ 21 ) + t 4 ( δ 12 ) t 5 ( δ 22 ) − 2 t(cid:18)
δ 11 + δ (cid:19) t(cid:18)
δ 21 + δ (cid:19) =(
u 1 + (cid:15) 1 )( u 2 + (cid:15) 2 ) ∞ (cid:88) n 1 ,n 2 =
[Equation]
[Equation]
where the second equality sign used Lemma A.5 and the third equality sign used the facts that q n 1 ,n 2 ( · ) =
o (1) and r n 1 ,n 2 ( · ) =
o (1) whenever n 1 + n 2 > 2 , and that q 1 , 1 ( · ) = 0 and r 1 , 1 ( · ) = 1 2 .
If we substitute (A.13), (A.14), (A.15), and (A.16) into (A.12),
[Equation]
[Equation] [Equation]
− δ 12 ) 2 + (cid:18) u 22 ( h (cid:48) ( v 2 ))
+ o (1) (cid:19) ( δ 21 − δ 22 )
+ (cid:18) u 1 u 2 h (cid:48) ( v 1 ) h (cid:48)
( v 2 )
+ o (1) (cid:19) ( δ 11 − δ 12 )( δ 21 − δ 22 ) .
(A.17)
We are almost done.
If we substitute (A.10), (A.11), and (A.17) into (A.9), we can get (cid:96) (( ˇ W j , ˇ b j ) 2 j =1 ) =
3 +  (cid:20) t (cid:18)
δ 11 + δ , δ 21 + δ (cid:19)(cid:21)+ (cid:18) u 1 h (cid:48)(cid:48) ( v 1 ) + o (1) (cid:19) ( δ 11 − δ 12 )
+ (cid:18) u
2 h (cid:48)(cid:48) ( v 2 )
+ o (1) (cid:19) ( δ 21 − δ 22 )
+ (cid:18) u 21 ( h (cid:48) ( v 1 ))
+ o (1) (cid:19) ( δ 11 − δ 12 )
+ (cid:18) u 22 ( h (cid:48) ( v 2 ))
+ o (1) (cid:19) ( δ 21 − δ 22 )
+ (cid:18) u 1 u 2 h (cid:48) ( v 1 ) h (cid:48)
( v 2 )
+ o (1) (cid:19) ( δ 11 − δ 12 )( δ 21 − δ 22 ) =3 +  (cid:20) t (cid:18)
δ 11 + δ , δ 21 + δ
[Equation]
+
[Equation]
which is the equation (A.7) that we were originally aiming to show.
A P ROOF OF C OROLLARY
For the proof of this corollary, we present the values of real numbers that satisfy assumptions (C2.1)– (C2.7), for each activation function listed in the corollary: sigmoid, tanh, arctan, exponential linear units (ELU, Clevert et al.
(2015)), scaled exponential linear units (SELU, Klambauer et al.
To remind the readers what the assumptions were, we list the assumptions again.
For (C2.1)–(C2.2), there exist real numbers v 1 , v 2 , v 3 , v 4 ∈ R such that
[Equation]
For (C2.3)–(C2.7), there exist real numbers v 1 , v 2 , u 1 , u 2 ∈ R such that the following assumptions hold:
[Equation]
(C2.4) h is inﬁnitely differentiable at v 1 and v 2 , (C2.5)
There exists a constant c > 0 such that | h ( n ) ( v 1 ) | ≤ c n n ! and | h ( n ) ( v 2 ) | ≤ c n n ! .
[Equation]
C2.6) ( u 1 h
[Equation]
( v 1 ))
[Equation]
u 1 h
[Equation]
( v
[Equation] [Equation]
[Equation]
For each function, we now present the appropriate real numbers that satisfy the assumptions.
A5. S IGMOID
When h is sigmoid,
[Equation]
Assumptions (C2.1)–(C2.2) are satisﬁed by
[Equation]
and assumptions (C2.3)–(C2.7) are satisﬁed by
[Equation]
Among them, (C2.4)–(C2.5) follow because sigmoid function is an real analytic function Krantz &
Parks (2002).
A5. TANH
When h is hyperbolic tangent, assumptions (C2.1)–(C2.2) are satisﬁed by ( v 1 , v
[Equation]
v
[Equation]
v 4 ) =
[Equation] [Equation]
and assumptions (C2.3)–(C2.7) are satisﬁed by
[Equation]
Assumptions (C2.4)–(C2.5) hold because hyperbolic tangent function is real analytic.
A5. ARCTAN
When h is inverse tangent, assumptions (C2.1)–(C2.2) are satisﬁed by ( v 1 , v 2 , v 3 , v 4 )
= (cid:18) tan (cid:18) (cid:19) , tan (cid:18) (cid:19) , tan (cid:18) (cid:19) , tan (cid:18) (cid:19)(cid:19) , and assumptions (C2.3)–(C2.7) are satisﬁed by ( v 1 , v 2 , u 1 , u 1 )
= (cid:18) tan (cid:18) (cid:19) , tan (cid:18) (cid:19) , 1 , − 1(cid:19) ,
Assumptions (C2.4)–(C2.5) hold because inverse tangent function is real analytic.
A5. Q UADRATIC
When h is quadratic, assumptions (C2.1)–(C2.2) are satisﬁed by ( v 1 , v 2 , v 3 , v 4 )
= (cid:18) 1 , 1 2 , 1 2 , − 1(cid:19) , and assumptions (C2.3)–(C2.7) are satisﬁed by ( v 1 , v 2 , u 1 , u 1 )
= (cid:18) 1 , 1 , 1 6 , 1(cid:19) ,
Assumptions (C2.4)–(C2.5) hold because quadratic function is real analytic.
A5. ELU AND SELU
When h is ELU or SELU,
[Equation]
where α > 0 , and λ = 1 (ELU) or λ > 1 (SELU).
In this case, assumptions (C2.1)–(C2.2) are satisﬁed by
[Equation]
Assumptions (C2.3)–(C2.7) are satisﬁed by
[Equation]
λ
[Equation]
λα
[Equation]
where (C2.4)–(C2.5) are satisﬁed because h ( x ) is real analytic at v 1 and v 2 .
A P ROOF OF T HEOREM 2 FOR “R E LU- LIKE ” ACTIVATION FUNCTIONS . Recall
the piecewise linear nonnegative homogeneous activation function
[Equation]
where s + > 0 , s − ≥
and s + (cid:54) =
s − , we will prove that the statements of Theorem 2 hold for ¯ h s + ,s − .
A6. P ROOF OF P ART
In the case of s − > 0 , assumptions (C2.1)–(C2.2) are satisﬁed by
[Equation]
The rest of the proof can be done in exactly the same way as the proof of Theorem 2.1, provided in
Appendix A3.
For s − = 0 , which corresponds to the case of ReLU, deﬁne parameters
[Equation]
We can check that
[Equation]
so
[Equation]
A6. P ROOF OF P ART
Assumptions (C2.3)–(C2.6) are satisﬁed by
[Equation]
Assign parameter values
[Equation]
It is easy to compute that the output of the neural network is ˆ Y = (cid:2)
[Equation]
ˆ
[Equation]
=
[Equation]
= 13 .
Now, it remains to show that this is indeed a local minimum of (cid:96) .
To show this, we apply perturbations to the parameters to see if the risk after perturbation is greater than or equal to (cid:96) (( ˆ W j , ˆ b j ) 2 j =1 ) .
Let the perturbed parameters be
[Equation]
where δ 11 , δ 12 , δ 21 , δ 22 , β 1 , β 2 , (cid:15) 1 , (cid:15) 2 , and γ are small enough real numbers.
Using the perturbed parameters,
[Equation]
so the empirical risk can be expressed as
[Equation]
+ δ 22 + β 2 ) + γ ]+ (cid:20) ( u 1 + (cid:15) 1 ) s + (cid:18) v 1 + δ 11 + δ + β(cid:19)
+ ( u 2 + (cid:15) 2 ) s
+ (cid:18) v 2 + δ 21 + δ + β(cid:19) + γ −(cid:21).
To simplify notation, let us introduce the following function:
[Equation]
It is easy to check that
[Equation]
With this new notation t ( δ
[Equation]
δ 2 )
[Equation]
we get
[Equation]
A P ROOF OF T HEOREM
Before we start, note the following partial derivatives, which can be computed using straightforward matrix calculus
[Equation]
for all j ∈ [ H + 1] .
A7. P ROOF OF P ART 1, IF d y ≥ d x
For Part 1, we must show that if ∇ (cid:96)
( ˆ W H +1:1 ) (cid:54)
=
then ( ˆ W j ) H +1 j =1 is a saddle point of (cid:96) .
Thus, we show that ( ˆ W j ) H +1 j =
is neither a local minimum nor a local maximum.
More precisely, for each j , let
B (cid:15) ( W j ) be an (cid:15) -Frobenius-norm-ball centered at W j , and (cid:81) H +1 j =1 B (cid:15) ( W j ) their Cartesian product.
We wish to show that for every (cid:15) > 0 , there exist tuples ( P j ) H +1 j =1 , ( Q j ) H +1 j =1 ∈ (cid:81)
H +1 j =1 B (cid:15) ( ˆ W j ) such that
[Equation]
To prove (A.18), we exploit (cid:96) (( ˆ W j ) H +1 j =1 ) =
(cid:96)
( ˆ W H +1:1 ) , and the assumption ∇ (cid:96)
( ˆ W H +1:1 ) (cid:54)
= 0 .
The key idea is to perturb the tuple ( ˆ W j ) H +1 j =1 so that the directional derivative of (cid:96) 0 along P H +1:1 − ˆ W H +1:1 is positive.
Since (cid:96) 0 is differentiable, if P H +1:1 − ˆ W H +1:1 is small, then (cid:96) (
( P j ) H +1 j =1 )=
(cid:96)
( P H +1:1 ) >(cid:96)
( ˆ W H +1:1 )=
(cid:96) (( ˆ W j )
H +1 j =1 ) .
Similarly, we can show (cid:96) (( Q j ) H +1 j =1 ) < (cid:96) (( ˆ W j ) H +1 j =1 ) .
The key challenge lies in constructing these perturbations; we outline our approach below; this construction may be of independent interest too.
For this section, we assume that d x ≥ d y for simplicity; the case d y ≥ d x is treated in
Appendix A7.2.
Since ∇ (cid:96)
( ˆ W H +1:1 ) (cid:54)
= 0 , col( ∇ (cid:96)
( ˆ W H +1:1 ))
⊥ must be a strict subspace of R d y .
Consider ∂(cid:96)/∂W 1 at a critical point to see that ( ˆ W H +1:2 ) T ∇ (cid:96)
( ˆ W H +1:1 )
= 0 ,
so col( ˆ W H +1:2 ) ⊆
col( ∇ (cid:96)
( ˆ W H +1:1 )) ⊥ (cid:40) R
d y .
This strict inclusion implies rank( ˆ W H +1:2 ) <
d y ≤ d 1 , so that null( ˆ W H +1:2 ) is not a trivial subspace.
Moreover, null( ˆ W H +1:2 ) ⊇ null( ˆ W H :2 ) ⊇ · · · ⊇ null( ˆ W 2 ) .
We can split the proof into two cases: null( ˆ W H +1:2 ) (cid:54) =
null( ˆ W H :2 ) and null( ˆ W H +1:2 ) =
null( ˆ W H :2 ) .
Let the SVD of ∇ (cid:96)
( ˆ W H +1:1 )
=
U l Σ U
Tr .
Recall [ U l ] · , 1 and [ U r ] · , 1 denote ﬁrst columns of U l and
U r , respectively.
Case 1: null( ˆ W H +1:2 ) (cid:54) =
null( ˆ W H :2 ) .
In this case, null( ˆ W H +1:2 ) (cid:41) null( ˆ W H :2 ) .
We will perturb ˆ W 1 and ˆ W H +1 to obtain the tuples ( P j ) H +1 j =1 and ( Q j ) H +1 j =1 .
To create our perturbation, we choose two unit vectors as follows:
[Equation]
Then, deﬁne ∆ 1 :=
(cid:15)v 1 v T 0 ∈ R d 1 × d x , and V 1 :
= ˆ W 1 + ∆ 1 ∈ B (cid:15) ( ˆ W 1 ) .
Since v 1 lies in null( ˆ W H +1:2 ) , observe that
[Equation]
With this deﬁnition of V 1 , we can also see that
[Equation]
Note that ∇ (cid:96)
( ˆ W H +1:1 )
( ˆ W H :1 ) T is equal to ∂(cid:96)/∂W H +1 at a critical point, hence is zero.
Since v 0 =
[ U r ] · , 1 , we have ∇ (cid:96)
( ˆ W H +1:1 ) v 0 = σ max ( ∇ (cid:96)
( ˆ W H +1:1 ))
[ U l ] · , 1 , which is a nonzero column vector, and since v 1 ∈ null( ˆ W H :2 ) ⊥ = row( ˆ W H :2 ) , v T 1 ( ˆ W H :2 ) T is a nonzero row vector.
From this observation, ∇ (cid:96)
( ˆ W H +1:1 ) v 0 v T 1 ( ˆ W H :2 ) T is nonzero, and so is ∇ (cid:96)
( ˆ W H +1:1 ) V T 1 ( ˆ W H :2 ) T .
We are now ready to deﬁne the perturbation on ˆ W H +1 :
[Equation]
so that ˆ W H +1 + ∆ H +1 ∈ B (cid:15) ( ˆ W H +1 ) .
Then, observe that
[Equation]
by deﬁnition of ∆ H +1 .
In other words, ∆ H +1 ˆ W H :2 V 1 is an ascent direction of (cid:96) 0 at ˆ W H +1:1 .
Now choose the tuples
[Equation]
where η ∈ (0 , 1] is chosen suitably.
It is easy to verify that ( P j ) H +1 j =1 , ( Q j ) H +1 j =1 ∈
[Equation]
H +1 j =1 B (cid:15) ( ˆ W j ) , and that the products
[Equation]
Since (cid:96) 0 is differentiable, for small enough η ∈ (0 , 1] , (cid:96)
( P H +1:1 ) > (cid:96)
( ˆ W H +1:1 ) > (cid:96)
( Q H +1:1 ) , proving (A.18).
This construction is valid for any (cid:15) > 0 , so we are done.
Case 2: null( ˆ W H +1:2 ) =
null( ˆ W H :2 ) .
By and large, the proof of this case goes the same, except that we need a little more care on what perturbations to make.
Deﬁne
[Equation]
When you start from j = H down to j = 2 and compare null( ˆ W j :2 ) and null( ˆ W j − 1:2 ) , the ﬁrst iterate j at which you have null( ˆ W j :2 ) (cid:54) =
null( ˆ W j − 1:2 ) is j ∗ .
If all null spaces of matrices from ˆ W H :2 to ˆ W 2 are equal, j ∗ = 2 which follows from the notational convention that null( ˆ W 1:2 ) =
null( I d 1 ) = { 0 } .
According to j ∗ , in Case 2 we perturb ˆ W 1 , ˆ W H +1 , ˆ W H , . . .
, ˆ W j ∗ to get (
P j ) H +1 j
= and ( Q j ) H +1 j =1 .
Recall the deﬁnition of left-null space of matrix A : leftnull( A )
= { v | v T A = 0 } .
By deﬁnition of j ∗ , note that null( ˆ W H +1:2 ) =
null( ˆ W
[Equation]
) =
[Equation]
· · = null(
[Equation]
W
[Equation]
2 ) ⇔ row( ˆ W H +1:2 )
[Equation]
row( ˆ W H :2 ) = · · ·
[Equation]
row( ˆ W j ∗ :2 )
[Equation]
which means the products are all rank-deﬁcient (recall rank( ˆ W H +1:2 )
<
d y and all d j ≥ d y ), and hence they all have nontrivial left-null spaces leftnull( ˆ W H :2 ) , . . .
, leftnull( ˆ W j ∗ :2 ) as well.
We choose some unit vectors as the following:
[Equation]
Then, for a γ ∈ (0 , (cid:15) ] whose value will be speciﬁed later, deﬁne
[Equation]
and V j :=
ˆ W j + ∆ j accordingly for j = 1 , j ∗ + 1 , . .
, H + 1 .
By deﬁnition of ∆ j ’s, note that
[Equation]
where in (A.19) we used the deﬁnition that v j ∗ ∈ leftnull( ˆ W j ∗ :2 ) , in (A.20) that v j ∗ +1 ∈ leftnull( ˆ W j ∗ +1:2 ) , in (A.21) that v H ∈ leftnull( ˆ W H :2 ) , and in (A.22) that v 1 ∈ null( ˆ W j ∗ :2 ) .
Now consider the following matrix product:
[Equation]
We are going to show that for small enough γ ∈ (0 , (cid:15) ]
[Equation]
this product is nonzero.
If we expand (A.23), there are many terms in the summation.
However, note that the expansion can be arranged in the following form:
[Equation]
where C j ∈ R d j ∗ ×
d j ∗− 1 for all j and C j doesn’t depend on γ , and speciﬁcally
[Equation]
Because the C 0 is exactly equal to ∂(cid:96)
∂W j ∗ evaluated at a critical point (( ˆ W j ) H +1 j =1 ) , C 0 = 0 .
Also, due to deﬁnitions of ∆ j ’s,
[Equation] [Equation]
First, v j ∗ is a nonzero column vector.
Since v H + =
[ U l ] · , 1 and v= [ U r ] · , 1 ,
[Equation]
Also, since v 1 ∈ row( ˆ W j ∗ − 1:2 ) , v T 1 ( ˆ W j ∗ − 1:2 ) T will be a nonzero row vector.
Thus, the product C H − j ∗ +2 will be nonzero.
Since C H − j ∗ +2 (cid:54) = 0 , we can pick any index ( α, β ) such that the ( α, β ) -th entry of C H − j ∗ +2 , denoted as [ C H − j ∗ +2 ] α,β , is nonzero.
Then, the ( α, β ) -th entry of (A.24) can be written as
[Equation]
where c j = [ C j ] α,β .
To show that the matrix product (A.23) is nonzero, it sufﬁces to show that its ( α, β ) -th entry (A.25) is nonzero.
If c 1 = · · · = c H − j ∗ +1 = 0 , then with the choice of γ = (cid:15) , (A.25) is trivially nonzero.
If some of c 1 , . . . ,
c H − j ∗ +1 are nonzero, we can scale γ ∈ (0 , (cid:15) ] arbitrarily small, so that
[Equation]
∗ +2 | , and thus (A.25) can never be zero.
From this, with sufﬁciently small γ , the matrix product (A.23) is nonzero.
Now deﬁne the perturbation on ˆ W j ∗ :
[Equation]
so that ˆ W j ∗ + ∆ j ∗ ∈ B (cid:15) ( ˆ W j ∗ ) .
Then, observe that
[Equation]
This means that V H
+1: j ∗ +1 ∆ j ∗ ˆ W j ∗ − 1:2 V 1 and − V H
+1: j ∗ +1 ∆ j ∗ ˆ W j ∗ − 1:2 V 1 are ascent and descent directions, respectively, of (cid:96)
( R ) at ˆ W H +1:1 .
After that, the proof is very similar to the previous case.
We can deﬁne
[Equation]
where 0 < η ≤ 1 is small enough, to show that by differentiability of (cid:96)
( R ) , we get (cid:96) (
( P j ) H +1 j =1 ) > (cid:96) (( ˆ W j ) H +1 j =1 ) > (cid:96) (( Q j ) H +1 j =1 ) .
A7. P ROOF OF P ART 1, IF d y ≥ d x
First, note that ∇ (cid:96)
( ˆ W H +1:1 )
( ˆ W H :1 ) T = 0 , because it is ∂(cid:96)
∂W H +1 evaluated at a critical point ( ˆ W j ) H +1 j =1 .
This equation implies row( ∇ (cid:96)
( ˆ W H +1:1 ))
⊥ ⊇ row( ˆ W H :1 ) .
Since ∇ (cid:96)
( ˆ W H +1:1 ) (cid:54)
= 0 , row( ∇ (cid:96)
( ˆ W H +1:1 ))
⊥ cannot be the whole R d x , and it is a strict subspace of R d x .
Observe that ˆ W H :1 ∈ R d H × d x and d x ≤ d H .
Since row( ˆ W H :1 ) ⊆ row( ∇ (cid:96)
( ˆ W H +1:1 )) ⊥ (cid:40)
R d x , this means rank( ˆ W H :1 ) <
d x , hence leftnull( ˆ W H :1 ) is not a trivial subspace.
Now observe that
[Equation]
where some of left-null spaces in the right could be zero-dimensional.
The procedure of choosing the perturbation depends on these left-null spaces.
We can split the proof into two cases: leftnull( ˆ W H :1 )
(cid:54) =
leftnull( ˆ W H :2 ) and leftnull( ˆ W H :1 ) =
leftnull( ˆ W H :2 ) .
Because the former case is simpler, we prove the former case ﬁrst.
Before we dive in, again take SVD of ∇ (cid:96)
( ˆ W H +1:1 )
=
U l Σ U
Tr .
Since ∇ (cid:96)
( ˆ W H +1:1 ) (cid:54)
=
0 , there is at least one positive singular value, so σ max ( ∇ (cid:96)
( ˆ W H +1:1 )) > 0 .
Recall the notation that
[ U l ] · ,and [ U r ] · , 1 are ﬁrst column vectors of U l and U r , respectively.
Case 1: leftnull( ˆ W H :1 ) (cid:54) =
leftnull( ˆ W H :2 ) .
In this case, leftnull( ˆ W H :1 )
(cid:41) leftnull( ˆ W H :2 ) .
We will perturb ˆ W 1 and ˆ W H +1 to obtain the desired tuples ( P j ) H +1 j =1 and ( Q j ) H +1 j =1 .
Now choose two unit vectors v H and v H +1 , as the following:
[Equation]
, and then deﬁne ∆ H
+1 :
= (cid:15)v H +1 v TH ∈ R d y × d H , and V H
+1 :
=
ˆ W H +1 + ∆ H
+1 .
We can check
V H +1 ∈ B (cid:15) ( ˆ W H +1 )
from the fact that v H and v H +1 are unit vectors.
Since v H ∈ leftnull( ˆ W H :1 ) , observe that
[Equation]
With this deﬁnition of V H +1 , we can also see that
[Equation]
Note that ( ˆ W H +1:2 ) T ∇ (cid:96)
( ˆ W H +1:1 ) is exactly equal to ∂(cid:96)∂W 1 evaluated at ( ˆ W j ) H +1 j =1 , hence is zero by assumption that ( ˆ W j ) H +1 j =1 is a critical point.
Since v H ∈ leftnull( ˆ W H :2 ) ⊥ =
col( ˆ W H :2 ) , ( ˆ W H :2 ) T v H is a nonzero column vector, and since v H +
=
[ U l ] · , 1 , v T H +1 ∇ (cid:96)
( ˆ W H +1:1 )
=
σ max ( ∇ (cid:96)
( ˆ W H +1:1 ))
([ U r ] · , 1 ) T , which is a nonzero row vector.
From this observation, we can see that ( ˆ W H :2 ) T v H v TH +1 ∇ (cid:96)
( ˆ W H +1:1 ) is nonzero, and so is ( ˆ W H :2 ) T V TH +1 ∇ (cid:96)
( ˆ W H +1:1 ) .
Now deﬁne the perturbation on ˆ W 1 :
[Equation]
so that ˆ W 1 + ∆ 1 ∈ B (cid:15) ( ˆ W 1 ) .
Then, observe that
[Equation]
by deﬁnition of ∆ 1 .
This means that V H
+1 ˆ W H :2 ∆ 1 and − V H
+1 ˆ W H :2 ∆ 1 are ascent and descent directions, respectively, of (cid:96)
( R ) at ˆ W H +1:1 .
Since (cid:96) 0 is a differentiable function, there exists small enough 0 < η ≤ 1 that satisﬁes
[Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
(cid:96)
[Equation] [Equation]
+1:1 ) .
Now deﬁne
[Equation]
We can check ( P j ) H +1 j =1 , ( Q j ) H +1 j =1 ∈ (cid:81) H +1 j =1 B (cid:15) ( ˆ W j ) , and
P H +1:
=
ˆ W H +1:1 + ηV H
+1 ˆ W H :2 ∆ 1 .
[Equation]
H +
[Equation] [Equation]
H
[Equation] [Equation]
[Equation]
H
[Equation]
By deﬁnition of (cid:96) (( W j ) H +1 j =1 ) , this shows that (cid:96) (( P j ) H +1 j =1 ) > (cid:96) (( ˆ W j )
H +1 j =1 ) > (cid:96) (( Q j ) H +1 j =1 ) .
This construction holds for any (cid:15) > 0 , proving that ( ˆ W j ) H +1 j =1 can be neither a local maximum nor a local minimum.
Case 2: leftnull( ˆ W H :1 )
=
leftnull( ˆ W H :2 ) .
By and large, the proof of this case goes the same, except that we need a little more care on what perturbations to make.
Deﬁne j ∗ =
min { j ∈ [2 , H ] | leftnull( ˆ W H : j ) (cid:41) leftnull( ˆ W H : j +1 ) } .
When you start from j = 2 up to j = H and compare leftnull( ˆ W H : j ) and leftnull( ˆ W H : j +1 ) , the ﬁrst iterate j at which you have leftnull( ˆ W H : j ) (cid:54) =
leftnull( ˆ W H : j +1 ) is j ∗ .
If all left-null spaces of matrices from ˆ W H :2 to ˆ W H are equal, j ∗ =
H which follows from the notational convention that leftnull( ˆ W H : H +1 )
= leftnull( I d H ) = { 0 } .
According to j ∗ , in Case 2 we perturb ˆ W H
+1 , ˆ W 1 , ˆ W 2 , . . .
, ˆ W j ∗ to get ( P j ) H +1 j =
and
( Q j ) H +1 j =1 .
By deﬁnition of j ∗ , note that
[Equation]
which means the products are all rank-deﬁcient (recall rank( ˆ W H :1 )
<
d x and all d j ≥ d x ), and hence they all have nontrivial null spaces null( ˆ W H :2 ) , . .
, null( ˆ W H : j ∗ ) as well.
We choose some unit vectors as the following:
[Equation]
Then, for a γ ∈ (0 , (cid:15) ] whose value will be speciﬁed later, deﬁne
[Equation]
TH
[Equation]
and V j :=
ˆ W j + ∆ j accordingly for j = 1 , . .
, j ∗ − 1 , H + 1 .
By deﬁnition of ∆ j ’s, note that
[Equation]
where in (A.26) we used the deﬁnition that v j ∗ − 1 ∈ null( ˆ W H : j ∗ ) , in (A.27) that v j ∗ − 2 ∈ null( ˆ W H : j ∗ − 1 ) , in (A.28) that v 1 ∈ null( ˆ W H :2 ) , and in (A.29) that v H ∈ leftnull( ˆ W H : j ∗ ) .
Now consider the following matrix product:
[Equation]
We are going to show that for small enough γ ∈ (0 , (cid:15) ]
[Equation]
this product is nonzero.
If we expand (A.30), there are many terms in the summation.
However, note that the expansion can be arranged in the following form:
[Equation] [Equation]
where C j ∈ R d j ∗ ×
d j ∗− 1 for all j and C j doesn’t depend on γ , and speciﬁcally
[Equation]
Because the C 0 is exactly equal to ∂(cid:96)
∂W j ∗ evaluated at a critical point (( ˆ W j ) H +1 j =1 ) , C 0 = 0 .
Also, due to deﬁnitions of ∆ j ’s,
[Equation]
First, since v H ∈ col( ˆ W H : j ∗ +1 ) , ( ˆ W H : j ∗ +1 ) T v H is a nonzero column vector.
Also, since v H +
=
[ U l ] · , 1 and v 0 =
[ U r ] · , 1 , the product v TH +1 ∇ (cid:96)
( ˆ W H +
[Equation]
) v
[Equation]
= σ max ( ∇ (cid:96)
( ˆ W H +1:1 )) > 0 .
Finally, v T j ∗ − 1 is a nonzero row vector.
Thus, the product C j ∗ will be nonzero.
Since C j ∗ (cid:54) = 0 , we can pick any index ( α, β ) such that the ( α, β ) -th entry of C j ∗ , denoted as [ C j ∗ ] α,β , is nonzero.
Then, the ( α, β ) -th entry of (A.31) can be written as
[Equation]
where c j = [ C j ] α,β .
To show that the matrix product (A.30) is nonzero, it sufﬁces to show that its ( α, β ) -th entry (A.32) is nonzero.
If c 1 = · · · = c j ∗ − 1 = 0 , then with the choice of γ = (cid:15) , (A.32) is trivially nonzero.
If some of c 1 , . . . ,
c j ∗ − 1 are nonzero, we can scale γ ∈ (0 , (cid:15) ] arbitrarily small, so that
[Equation]
and thus (A.32) can never be zero.
From this, with sufﬁciently small γ , the matrix product (A.30) is nonzero.
Now deﬁne the perturbation on ˆ W j ∗ :
[Equation]
so that ˆ W
[Equation]
∆
[Equation]
∈ B
[Equation]
( ˆ W j ∗ ) .
Then, observe that
[Equation]
This means that V H +1 ˆ W H : j ∗ +1 ∆ j ∗ V j ∗ − 1:1 and − V H
+
ˆ W H : j ∗ +1 ∆ j ∗ V j ∗ − 1:1 are ascent and descent directions, respectively, of (cid:96)
( R ) at ˆ W H +1:1 .
After that, the proof is very similar to the previous case.
We can deﬁne
[Equation]
where 0 < η ≤ 1 is small enough, to show that by differentiability of (cid:96)
( R ) , we get (cid:96) (
( P j ) H +1 j =1 ) > (cid:96) (( ˆ W j ) H +1 j =1 ) > (cid:96) (( Q j ) H +1 j =1 ) .
A7. P ROOF OF P ART 2( A )
In this part, we show that if ∇ (cid:96)
( ˆ W H +1:1 )
=
and ˆ W H +1:1 is a local min of (cid:96) 0 , then ( ˆ W j ) H +1 j =1 is a local min of (cid:96) .
The proof for local max case can be done in a very similar way.
Since ˆ W H +1:1 is a local minimum of (cid:96) 0 , there exists (cid:15) > 0 such that, for any R satisfying (cid:107) R −
ˆ W H
+1:1 (cid:107) F ≤ (cid:15) , we have (cid:96) 0 ( R ) ≥ (cid:96)
( ˆ W H +1:1 ) .
We prove that ( ˆ W j ) H +1 j =1 is a local minimum of (cid:96) by showing that there exists a neighborhood of ( ˆ W j ) H +1 j =1 in which any point ( V j ) H +1 j = satisﬁes (cid:96) (( V j ) H +1 j =1 ) ≥ (cid:96) (( ˆ W j )
H +1 j =1 ) .
Now deﬁne
[Equation]
Observe that a max { a, 1 } ≤ 1 for a ≥ 0 .
Then, for all j ∈ [ H +1] , pick any V j such that (cid:107) V j − ˆ W j (cid:107) F ≤ (cid:15) j .
Denote ∆ j = V j −
ˆ
W
j for all j .
Now, by triangle inequality and submultiplicativity of Frobenius norm,
[Equation]
H +1: j +1 (cid:107) F (cid:107) ∆ j (cid:107) F (cid:107) ˆ W j − 1:1 (cid:107) F + O (max j (cid:15) 2 j ) ≤ (cid:15) 2 + O (max j (cid:15) 2 j ) ≤ (cid:15), for small enough (cid:15) j ’s.
Given this, for any ( V j ) H +1 j = in the neighborhood of ( ˆ W j ) H +1 j = deﬁned by (cid:15) j ’s, (cid:107) V H
+1:
− ˆ W H
+1:1 (cid:107) F ≤ (cid:15) , so (cid:96)
( V H +1:1 ) ≥ (cid:96)
( ˆ W H +1:1 ) , meaning (cid:96) (( V j ) H +1 j =1 ) ≥ (cid:96) (( ˆ W j ) H +1 j =1 ) .
Thus, ( ˆ W j ) H +1 j =1 is a local minimum of (cid:96) .
A7. P ROOF OF P ART 2( B )
For this part, we want to show that if ∇ (cid:96)
( ˆ W H +1:1 )
= 0 , then ( ˆ W j ) H +1 j =1 is a global min (or max) of (cid:96) if and only if ˆ W H +1:1 is a global min (or max) of (cid:96) 0 .
We prove this by showing the following: if d j ≥ min { d x , d y } for all j ∈ [ H ] , for any R ∈ R d y ×
d x there exists a decomposition ( W j ) H +1 j = such that R = W H +1:1 .
We divide the proof into two cases: d x ≥ d y and d y ≥ d x .
Case 1: d x ≥ d y .
If d x ≥ d y , by assumption d j ≥ d y for all j ∈
[ H ] .
Recall that W 1 ∈ R
d
1 × d x .
Given R ∈ R d y × d x , we can ﬁll the ﬁrst d y rows of W 1 with R and let any other entries be zero.
For all the other matrices W 2 , . . .
, W H +1 , we put ones to the diagonal entries while putting zeros to all the other entries.
We can check that, by this construction, R = W H +1:1 for this given R .
Case 2: d y ≥ d x .
If d y ≥ d x , we have d j ≥ d x for all j ∈
[ H ] .
Recall W H +1 ∈ R d y × d H .
Given
R ∈ R d y × d x , we can ﬁll the ﬁrst d x columns of W H +1 with R and let any other entries be zero.
For all the other matrices W 1 , . . .
, W H , we put ones to the diagonal entries while putting zeros to all the other entries.
By this construction, R = W H +1:1 for given R .
Once this fact is given, by (cid:96) (( W j ) H +1 j =1 ) =
(cid:96)
( W H +1:1 ) ,
[Equation]
Thus, any ( ˆ W j ) H +1 j =1 attaining a global min of (cid:96) must have inf R (cid:96) 0 ( R ) =
(cid:96)
( ˆ W H +1:1 ) , so ˆ W H
+1: is also a global min of (cid:96)
( R ) .
Conversely, if (cid:96)
( ˆ W H +1:1 )
= inf (cid:96)
( R ) , then (cid:96) (( ˆ W j )
H +1 j =1 ) = inf (cid:96) (( W j ) H +1 j =1 ) , so ( ˆ W j ) H +1 j =1 is a global min of (cid:96) .
We can prove the global max case similarly.
A7. P ROOF OF P ART 3 AND 3( A )
Suppose there exists j ∗ ∈
[ H + 1] such that ˆ W H
+1:
j ∗ +1 has full row rank and ˆ W j ∗ − 1:1 has full column rank.
For simplicity, deﬁne A :
=
ˆ W H
+1: j ∗ +1 and B :=
ˆ W j ∗ − 1:1 .
Since A T has linearly independent columns, B T has linearly independent rows, and ∂(cid:96)/∂W j ∗ = 0 at ( ˆ W j ) H +1 j =1 ,
A T ∇ (cid:96)
( ˆ W H +1:1 ) B T =
= ⇒ ∇ (cid:96)
( ˆ W H +1:1 )
= 0 , hence Parts 2(a) and 2(b) are implied.
For Part 3(a), we want to prove that if ( ˆ W j ) H +1 j =1 is a local min of (cid:96) , then ˆ W H +1:1 is a local min of (cid:96) 0 .
By deﬁnition of local min, ∃ (cid:15) > 0 such that, for any ( V j ) H +1 j =1 for which (cid:107) V j − ˆ W j (cid:107) F ≤ (cid:15) (for j ∈
[ H + 1] ), we have (cid:96) (( V j ) H +1 j =1 ) ≥ (cid:96) (( ˆ W j ) H +1 j =1 ) .
To show that ˆ W H +1:1 is a local min of (cid:96) 0 , we have to show there exists a neighborhood of ˆ W H +1:1 such that, any point R in that neighborhood satisﬁes (cid:96)
( R ) ≥ (cid:96)
( ˆ W H +1:1 ) .
To prove this, we state the following lemma:
Lemma A.6.
Suppose A :=
ˆ W H
+1:
j ∗ +1 has full row rank and B :
= ˆ W j ∗ − 1:1 has full column rank.
Then, any R satisfying (cid:107) R −
ˆ W H
+1:1 (cid:107) F ≤ σ min ( A ) σ min ( B ) (cid:15) can be decomposed into R =
V H +1:1 , where
V
[Equation]
T , and V j = ˆ W j for j (cid:54) =
j ∗ .
Also, (cid:107) V j − ˆ W j (cid:107) F ≤ (cid:15) for all j .
Proof
Since A :
=
ˆ W H
+1:
j ∗ +1 has full row rank and B :
= ˆ W j ∗ − 1:1 has full column rank, σ min ( A ) > 0 , σ min ( B ) > 0 , and AA T and B T B are invertible.
Consider any R satisfying (cid:107) R − ˆ W
[Equation]
+
[Equation] [Equation]
F ≤ σ min ( A ) σ min ( B ) (cid:15) .
Given the deﬁnitions of V j ’s in the statement of the lemma, we can check the identity that R = V H +1:1 by
[Equation]
Now It is left to show that (cid:107) V j ∗ −
ˆ W j ∗
[Equation]
F ≤ (cid:15) , so that ( V j ) H +1 j =1 indeed satisﬁes (cid:107) V j − ˆ W j (cid:107) F ≤ (cid:15) for all j .
We can show that
[Equation]
Therefore,
[Equation]
B T B ) − 1 B
[Table]
F ≤ σ max ( A T ( AA T ) − 1 ) σ max (( B T B
[Table]
− 1 B T ) (cid:107) R − ˆ W H
+1:1 (cid:107) F ≤ σ min
[Table]
σ min
[Table]
B
[Table]
· σ min
[Table]
σ min
[Table]
B
[Table]
(cid:15) =
(cid:15).
The lemma shows that for any R
=
V H +1:1 satisfying (cid:107) R − ˆ W H
+1:1 (cid:107) F ≤ σ min ( A ) σ min ( B ) (cid:15) , we have (cid:96)
( R ) =
(cid:96)
( V H +1:1 ) =
(cid:96) (( V j ) H +1 j =1 ) ≥ (cid:96) (( ˆ W j )
H +1 j =1 )
= (cid:96)
( ˆ W H +1:1 ) .
We can prove the local maximum part by a similar argument.