P IECEWISE LINEAR ACTIVATIONS SUBSTANTIALLY SHAPE THE LOSS SURFACES OF NEURAL NETWORKS
Understanding the loss surface of a neural network is fundamentally important to the understanding of deep learning.
This paper presents how piecewise linear activation functions substantially shape the loss surfaces of neural networks.
We ﬁrst prove that the loss surfaces of many neural networks have inﬁnite spurious local minima , which are deﬁned as the local minima with higher empirical risks than the global minima.
Our result holds for any neural network with arbitrary depth and arbitrary piecewise linear activation functions (excluding linear functions) under most loss functions in practice with some mild assumptions.
This result demonstrates that the networks with piecewise linear activations possess substantial differences to the well-studied linear neural networks.
Essentially, the underlying assumptions for the above result are consistent with most practical circumstances where the output layer is narrower than any hidden layer.
In addition, the loss surface of a neural network with piecewise linear activations is partitioned into multiple smooth and multilinear cells by nondifferentiable boundaries.
The constructed spurious local minima are concentrated in one cell as a valley: they are connected with each other by a continuous path, on which empirical risk is invariant.
Further for one-hidden-layer networks, we prove that all local minima in a cell constitute an equivalence class; they are concentrated in a valley; and they are all global minima in the cell.
I NTRODUCTION
Neural networks have been successfully deployed in many real-world applications (LeCun et al., 2015;
Witten et al., 2016; Silver et al., 2016;
He et al., 2016; Litjens et al., 2017).
In spite of this, the theoretical foundations of neural networks are somewhat premature.
To the many deﬁciencies in our knowledge of deep learning theory, the investigation into the loss surfaces of neural networks is of fundamental importance.
Understanding the loss surface would be helpful in several relevant research areas, such as the ability to estimate data distributions, the optimization of neural networks, and the generalization to unseen data.
This paper studies the role of the nonlinearities in activation functions in shaping the loss surfaces of neural networks.
Our results demonstrate that the impact of nonlinearities is profound.
First, we prove that the loss surfaces of nonlinear neural networks are substantially different to those of linear neural networks, in which local minima are created equal, and also, they are all global minima (Kawaguchi, 2016; Baldi & Hornik, 1989; Lu & Kawaguchi, 2017; Freeman & Bruna, 2017; Zhou & Liang, 2018; Laurent & von Brecht, 2018; Yun et al., 2018).
By contrast,
Neural networks with arbitrary depth and arbitrary piecewise linear activations (excluding linear functions) have inﬁnitely many spurious local minima under arbitrary continuously differentiable loss functions.
This result only relies on three mild assumptions that cover most practical circumstances: (1) the training sample set is linearly inseparable; (2) all training sample points are distinct; (3) the output layer is narrower than the other hidden layers; and (4) there exists some turning point in the piecewise linear activations that the sum of the slops on the two sides does not equal to 0 .
Our result signiﬁcantly extends the existing study on the existence of spurious local minimum.
For example, Zhou & Liang (2018) prove that one-hidden-layer neural networks with two nodes in the hidden layer and two-piece linear (ReLU-like) activations have spurious local minima; Swirszcz et al.
(2016) prove that ReLU networks have spurious local minima under the squared loss when most of the neurons are not activated; Safran & Shamir (2018) present a computer-assisted proof that two-layer ReLU networks have spurious local minima; a recent work (Yun et al., 2019b) have proven that neural networks with two-piece linear activations have inﬁnite spurious local minima, but the results only apply to the networks with one hidden layer and one-dimensional outputs; and a concurrent work (Goldblum et al., 2020) proves that for multi-layer perceptrons of any depth, the performance of every local minimum on the training data equals to a linear model, which is also veriﬁed by experiments.
The proposed theorem is proved in three stages: (1) we prove that neural networks with one hidden layer and two-piece linear activations have spurious local minima; (2) we extend the conditions to neural networks with arbitrary hidden layers and two-piece linear activations; and (3) we further extend the conditions to neural networks with arbitrary depth and arbitrary piecewise linear activations.
Since some parameters of the constructed spurious local minima are from continuous intervals, we have obtained inﬁnitely many spurious local minima.
At each stage, the proof follows a two-step strategy that: (a) constructs an inﬁnite series of local minima; and (b) constructs a point in the parameter space whose empirical risk is lower than the constructed local minimum in Step (a).
This strategy is inspired by Yun et al.
(2019b)
but we have made signiﬁcant and non-trivial development.
Second, we draw a “big picture” for the loss surfaces of nonlinear neural networks.
Soudry & Hoffer (2018) highlight a smooth and multilinear partition of the loss surfaces of neural networks.
The nonlinearities in the piecewise linear activations partition the loss surface of any nonlinear neural network into multiple smooth and multilinear open cells.
Speciﬁcally, every nonlinear point in the activation functions creates a group of the non-differentiable boundaries between the cells, while the linear parts of activations correspond to the smooth and multilinear interiors.
Based on the partition, we discover a degenerate nature of the large amounts of local minima from the following aspects:
• Every local minimum is globally minimal within a cell.
This property demonstrates that the local geometry within every cell is similar to the global geometry of linear networks, although technically, they are substantially different.
It applies to any one-hidden-layer neural network with two-piece linear activations for regression under convex loss.
We rigorously prove this property in two stages: (1) we prove that within every cell, the empirical risk
ˆ R is convex with respect to a variable ˆ W mapped from the weights W by a mapping Q .
Therefore, the local minima with respect to the variable ˆ W are also the global minima in the cell; and then (2) we prove that the local optimality is maintained under the constructed mapping.
Speciﬁcally, the local minima of the empirical risk ˆ R with respect to the parameter W are also the local minima with respect to the variable ˆ W .
We thereby prove this property by combining the convexity and the correspondence of the minima.
This proof is technically novel and non-trivial, though the intuitions are natural.
• Equivalence classes and quotient space of local minimum valleys.
All local minima in a cell are concentrated as a local minimum valley : on a local minimum valley, all local minima are connected with each other by a continuous path, on which the empirical risk is invariant. Further, all these local minima constitute an equivalence class.
This local minima valley may have several parallel valleys that are in the same equivalence class but do not appear because of the restraints of cell boundaries. If ignoring such constraints, all the equivalence classes constitute a quotient space. The constructed mapping Q is exactly the quotient map.
This result coincides with the property of mode connectivity that the minima found by gradient-based methods are connected by a path in the parameter space with almost invariant empirical risk (Garipov et al., 2018; Draxler et al., 2018; Kuditipudi et al., 2019).
Additionally, this property suggests that we would need to study every local minimum valley as a whole.
• Linear collapse.
Linear neural networks are covered by our theories as a simpliﬁed case.
When all activations are linear, the partitioned loss surface collapses to one single cell, in which all local minima are globally optimal, as suggested by the existing works on linear networks (Kawaguchi, 2016; Baldi & Hornik, 1989; Lu & Kawaguchi, 2017; Freeman & Bruna, 2017; Zhou & Liang, 2018; Laurent & von Brecht, 2018; Yun et al., 2018).
Notations.
If M is a matrix, M i,j denotes the ( i, j ) -th component of M .
If M is a vector, M i denotes the i -th component of M .
Deﬁne E ij as a matrix in which the ( i, j ) -th
component iswhile all other components are 0 .
Also, denote e i as a vector such that the i -th component is 1 while all others are 0 .
Additionally, we deﬁne 1 k ∈ R k × 1 is a vector whose components are all 1 , while those of 0 n × m ∈ R n × m (or brieﬂy, 0 ) are all 0 .
For the brevity, [ i : j ] denotes { i, · · · , j } .
R ELATED WORK
Some works suggest that linear neural networks have no spurious local minima.
Kawaguchi (2016) proves that linear neural networks with squared loss do not have any spurious local minimum under three assumptions about the data matrix X and the label matrix Y : (1) both matrices XX T and
XY T have full ranks; and (2) the input layer is wider than the output layer; and (3) the eigenvalues of matrix Y X (cid:62) (cid:0) XX T (cid:1) − 1 XY T are distinct with each other.
Zhou & Liang (2018) give an analytic formulation of the critical points for the loss function of deep linear networks, and thereby obtain a group of equivalence conditions for that critical point is a global minimum.
Lu & Kawaguchi (2017) prove the argument under one assumption that both matrices X and Y have full ranks, which is even more restrictive.
However, in practice, the activations of most neural networks are not linear.
The nonlinearities would make the loss surface extremely non-convex and even non-smooth and therefore far different from the linear case.
The loss surfaces of over-parameterized neural networks have some special properties.
Choromanska et al.
(2015) empirically suggest that: (1) most local minima of over-parameterized networks are equivalent; and (2) small-size networks have spurious local minima but the probability of ﬁnding one decreases rapidly with the network size.
Li et al.
(2018) prove that over-parameterized fullyconnected deep neural networks with continuous activation functions and convex, differentiable loss functions, have no bad strict local minimum.
Nguyen et al.
(2019) suggest that “sufﬁciently overparameterized” neural networks have no bad local valley under the cross-entropy loss.
Nguyen (2019) further suggests that the global minima of sufﬁciently over-parameterized neural networks are connected within a unique valley.
Many other works study the convergence, generalization, and other properties of stochastic gradient descent on the loss surfaces of over-parameterized networks (Chizat & Bach; Arora et al., 2018; Brutzkus et al., 2018; Du et al., 2019; Soltanolkotabi et al., 2018;
Allen-Zhu et al., 2019a;b; Oymak & Soltanolkotabi, 2019).
Many advances on the loss surfaces of neural networks are focused on other problems.
Zhou &
Feng (2017) and Mei et al.
(2018) prove that the empirical risk surface and expected risk surface are linked.
This correspondence highlights the value of investigating loss surfaces (empirical risk surfaces) to the study of generalization (the gap between empirical risks to expected risks).
Hanin &
Rolnick (2019) demonstrate that the input space of neural networks with piecewise linear activations are partitioned by multiple regions, while our work focuses on the partition of the loss surface.
Xie et al.
(2017) proves that the training error and test error are upper bounded by the magnitude of the gradient, under the assumption that the geometry discrepancy of the parameter W is bounded.
Sagun et al.
(2016; 2018) present empirical results that the eigenvalues of the Hessian of the loss surface are two-fold: (1) a bulk centered closed to zero; and (2) outliers away from the bulk.
Kawaguchi & Kaelbling (2019) prove that we can eliminate the spurious local minima by adding one unit per output unit for almost any neural network in practice.
Tian (2017); Andrychowicz et al.
Soltanolkotabi (2017); Zhong et al. (2017); Brutzkus & Globerson (2017); Tian (2017); Li & Yuan (2017); Zou et al.
(2018); Li & Liang (2018);
Du et al.
(2018a; 2019); Zhang et al.
(2019b)
; Zhou et al.
; Wang et al.
(2019) study the optimization methods for neural networks.
Other relevant works include Sagun et al.
; Nguyen & Hein (2018); Du et al. (2018b); Haeffele & Vidal (2017); Liang et al.
; Wu et al.
; Yun et al.
(2019a)
; Zhang et al. (2019a); Kuditipudi et al.
; Garipov et al.
; Draxler et al.
; He et al.
; Kawaguchi & Kaelbling (2019).
N EURAL NETWORK HAS INFINITE SPURIOUS LOCAL MINIMA
This section investigates the existence of spurious local minima on the loss surfaces of neural networks.
We ﬁnd that almost all practical neural networks have inﬁnitely many spurious local minima.
This result stands for any neural network with arbitrary depth and arbitrary piecewise linear activations excluding linear functions under arbitrary continuously differentiable loss.
3. P RELIMINARIES
Consider a training sample set { ( X 1 , Y 1 ) , ( X 2 , Y 2 ) , . .
, ( X n , Y n ) } of size n .
Suppose the dimensions of feature X i and label Y i are d X and d Y , respectively.
By aggregating the training sample set, we obtain the feature matrix
X ∈ R
d X × n and label matrix Y ∈ R d Y × n .
Suppose a neural network has L layers.
Denote the weight matrix, bias, and activation in the jth layer respectively by W j ∈ R d j × d j − 1 , b j ∈ R d j , and h : R d j × n → R d j × n , where d j is the dimension of the output of the j -th layer.
Also, for the input matrix X , the output of the j -th layer is denoted as the Y ( j ) and the output of the j -th layer before the activation is denoted as the ˜ Y ( j ) ,
[Equation]
The output of the network is deﬁned as follows,
[Equation]
Also, we deﬁne Y [Equation]
X , Y ( L ) =
ˆ Y , d 0 = d X , and d L = d Y .
In some situations, we use ˆ Y (cid:16) [ W i ] Li =1 , [ b i ] Li =1 (cid:17) to clarify the parameters, as well as ˜ Y ( j ) , Y ( j ) , etc.
This section discusses neural networks with piecewise linear activations.
A part of the proof uses two-piece linear activations h s − ,s + which are deﬁned as follows,
[Equation]
where | s + | (cid:54)
= | s − | and I {·} is the indicator function.
Remark.
Piecewise linear functions are dense in the space of continuous functions.
In other words, for any continuous function, we can always ﬁnd a piecewise linear function to estimate it with arbitrary small distance.
This section uses continuously differentiable loss to evaluate the performance of neural networks.
Continuous differentiability is deﬁned as follows.
Deﬁnition 1 (Continuously differentiable) .
We call a function f : R n → R continuously differentiable with respect to the variable x if: (1) the function f is differentiable with respect to x ; and (2) the derivatives f (cid:48)
( x ) of the function f with respect to the variable x are continuous with respect to the variable x .
3. M AIN RESULT
The theorem in this section relies on the following assumptions.
Assumption 1.
The training data cannot be ﬁt by a linear model.
Assumption 2.
All data points are distinct.
Assumption 3.
All hidden layers are wider than the output layer.
Assumption 4.
For the piece-wise linear activations, there exists some turning point that the sum of the slops on the two sides does not equal to 0 .
To our best knowledge, our assumptions are the least restrictive compared with the relevant works in the literature.
These assumptions are respectively justiﬁed as follows: (1) most real-world datasets are extremely complex and cannot be simply ﬁt using linear models; (2) it is easy to guarantee that the data points are distinct by employing data cleansing methods; (3) for regression and many classiﬁcation tasks, the width of output layer is limited and narrower than the hidden layers; and (4) this assumption is invalid only for activations like f ( x ) = a | x | .
Based on these four assumptions, we can prove the following theorem.
Theorem 1.
Neural networks with arbitrary depth and arbitrary piecewise linear activations (excluding linear functions) have inﬁnitely many spurious local minima under arbitrary continuously differentiable loss whose derivative can equal 0 only when the prediction and label are the same.
In practice, most loss functions are continuously differentiable and the derivative can equal 0 only when the prediction and label are the same, such as squared loss and cross-entropy loss (see Appendix A.1, Lemmas 2 and 3).
Squared loss is a standard loss for regression and is deﬁned as the Lnorm of the difference between the ground-truth label and the prediction as follows.
[Equation]
Meanwhile, cross-entropy loss is used as a standard loss in multiclass classiﬁcation, which is deﬁned as follows.
Here, we treat the softmax function as a part of the loss function.
[Equation]
One can also remove Assumption 4, if Assumption 3 is replaced by the following assumption, which is mildly more restrictive (see a detailed proof in pp. 34–37).
Assumption 5.
The dimensions of the layers satisfy that:
[Equation]
Our result demonstrates that introducing nonlinearities into activations substantially reshapes the loss surface: they bring inﬁnitely many spurious local minima into the loss surface.
This result highlights the substantial difference from linear neural networks that all local minima of linear neural networks are equally good, and therefore, they are all global minima (Kawaguchi, 2016; Baldi &
Hornik, 1989; Lu & Kawaguchi, 2017; Freeman & Bruna, 2017; Zhou & Liang, 2018; Laurent & von Brecht, 2018; Yun et al., 2018).
Some works have noticed the existence of spurious local minima on the loss surfaces of nonlinear neural networks, which however has a limited applicable domain (Choromanska et al., 2015;
Swirszcz et al., 2016; Safran & Shamir, 2018; Yun et al., 2019b).
A notable work by Yun et al.
(2019b) proves that one-hidden-layer neural networks with two-piece linear (ReLU-like) activations for one-dimensional regression have inﬁnitely many spurious local minima under squared loss.
This work ﬁrst constructs a series of local minima and then prove they are spurious.
This idea inspires some of this work.
However, our work makes signiﬁcant and non-trivial development that extends the conditions to arbitrary depth, piecewise linear activations excluding linear functions, and continuously differentiable loss.
3. P ROOF SKELETON
This section presents the skeleton of the proof.
Theorem 1 is proved in three stages.
We ﬁrst prove a simpliﬁed version of Theorem 1 and then extend the conditions in the last two stages.
The proof is partially inspired by Yun et al.
(2019b)
but the proof in this paper has made nontrivial development and the results are signiﬁcantly extended.
Yun et al.
(2019b)
and our paper both employ the following strategy: (a) construct a series of local minima based on a linear classiﬁer; and (b) construct a new point with smaller empirical risk and thereby we prove that the constructed local minima are spurious.
However, due to the differences in the loss function and the output dimensions, the exact constructions of local minima are substantially different.
Our extensions from Yun et al.
(2019b) are three-fold: (1) From one hidden layer to arbitrary depth:
To prove that networks with an arbitrary depth have inﬁnite spurious local minima, we develop a novel strategy that employs transformation operations to force data ﬂow through the same linear parts of the activations, in order to construct the spurious local minima; (2) From squared loss to arbitrary differentiable loss: Yun et al.
(2019b) calculate the analytic formations of derivatives of the loss to construct the local minima and then prove they are spurious.
This technique cannot be transplanted to the case of arbitrary differentiable loss functions, because we cannot assume the analytic formation.
To prove that the loss surface under an arbitrary differentiable loss has an inﬁnite number of spurious local minima, we employ a new proof technique based on Taylor series and a new separation lemma; and (3) From one-dimensional output to arbitrary-dimensional output:
To prove the loss surface of a neural network with an arbitrary-dimensional output has an inﬁnite number of spurious local minima, we need to deal with the calculus of functions whose domain and codomain are a matrix space and a vector space, respectively.
By contrast, when the output dimension is one, the codomain is only the space of real numbers.
Therefore, the extension of the output dimension signiﬁcantly mounts the difﬁculty of the whole proof.
Stage (1): Neural networks with one hidden layer and two-piece linear activations.
We ﬁrst prove that nonlinear neural networks with one hidden layer and two-piece linear activation functions (ReLU-like activations) have spurious local minima.
The proof in this stage further follows a two-step strategy:
(a) We ﬁrst construct local minima of the empirical risk ˆ R (see Appendix A.2, Lemma 4).
These local minimizers are constructed based on a linear neural network which has the same network size (dimension of weight matrices) and evaluated under the same loss.
The design of the hidden layer guarantees that the components of the output ˜ Y (1) in the hidden layer before the activation are all positive.
The activation is thus effectively reduced to a linear function.
Therefore, the local geometry around the local minima with respect to the weights W is similar to those of linear neural networks.
Further, the design of the output layer guarantees that its output
ˆ Y is the same as the linear neural network.
This construction helps to utilize the results of linear neural networks to solve the problems in nonlinear neural networks.
(b) We then prove that all the constructed local minima in Step (a) are spurious (see Appendix A.2, Theorem 4).
Speciﬁcally, we assumed by Assumption 1 that the dataset cannot be ﬁt by a linear model.
Therefore, the gradient ∇ ˆ Y ˆ R of the empirical risk ˆ R with respect to the prediction ˆ Y is not zero.
Suppose the i -th row of the gradient ∇ ˆ Y ˆ R is not zero.
Then, we use Taylor series and a preparation lemma (see Appendix A.5, Lemma 7) to construct another point in the parameter space that has smaller empirical risk.
Therefore, we prove that the constructed local minima are spurious.
Furthermore, the constructions involve some parameters that are randomly picked from a continuous interval.
Thus, we constructed inﬁnitely many spurious local minima.
Stage (2) - Neural networks with arbitrary hidden layers and two-piece linear activations.
We extend the condition in Stage (1) to any neural network with arbitrary depth and two-piece linear activations.
The proof in this stage follows the same two-step strategy but has different implementations:
(a) We ﬁrst construct a series of local minima of the empirical risk ˆ R (see Appendix A.3, Lemma 5).
The construction guarantees that every component of the output ˜ Y ( i ) in each layer before the activations is positive, which secure all the input examples ﬂow through the same part of the activations.
Thereby, the nonlinear activations are reduced to linear functions.
Also, our construction guarantees that the output ˆ Y of the network is the same as a linear network with the same weight matrix dimensions.
(b) We then prove that the constructed local minima are spurious (see Appendix A.3, Theorem 5).
The idea is to ﬁnd a point in the parameter space that has the same empirical risk ˆ R with the constructed point in Stage (1), Step (b).
Stage (3) - Neural networks with arbitrary hidden layer and piecewise linear activations.
We further extend the conditions in Stage (2) to any neural network with arbitrary depth and arbitrary piecewise linear activations.
We continue to adapt the two-step strategy in this stage:
(a) We ﬁrst construct a local minimizer of the empirical risk ˆ R based on the results in Stages (1) and (2) (see Appendix A.4, Lemma 6).
This construction is based on Stage (2), Step (a).
The difference of the construction in this stage is that every linear part in activations can be a ﬁnite interval.
The constructed weight matrices use several uniform scaling and translation operations to the outputs of hidden layers in order to guarantee that all the input training sample points ﬂow through the same linear parts of the activations.
We thereby reduce the nonlinear activations to linear functions, effectively.
Also, our construction guarantees that the output ˆ Y of the neural network equals to that of the corresponding linear neural network.
(b) We then prove that the constructed local minima are spurious (see Appendix A.4).
We use the same strategy in Stage (2), Step (b).
Some adaptations are implemented for the new conditions.
A BIG PICTURE OF THE LOSS SURFACE
This section draws a big picture for the loss surfaces of neural networks.
We present a theorem that demonstrates four profound properties of the loss surface that collectively characterize how the nonlinearities in activations shape the loss surface.
4. P RELIMINARIES
The discussions in this section use the following concepts.
Deﬁnition 2 (Open ball and open set) .
The open ball in H centered at x ∈ H and of radius r >is deﬁned by B ( h, r ) =
{ x : (cid:107) x − h (cid:107) < r } .
A subset A ⊂ H of a space H is called a open set, if for every point h ∈ A , there exists a positive real r > 0 , such that the open ball B ( h, r ) with center h and radius r is in the subset A : B ( h, r ) ⊂
A .
Deﬁnition 3 (Interior point and interior) .
For a subset A ⊂ H of a space H , a point h ∈ A is called an interior point of A , if there exists a positive real r > 0 , such that the open ball B ( h, r ) with center h and radius r is in the subset A : B ( h, r ) ⊂
A .
The set of all the interior points of the set
A is called the interior of the set A .
Deﬁnition 4 (Limit point, closure, and boundary) .
For a subset A ⊂ H of a space H , a point h ∈ A is called a limit point, if for every r > 0 , the open ball B ( h, r ) with center h and radius r contains some point of A : B ( h, r )
∩ A (cid:54) = ∅ .
The closure ¯ A of the set A consists of the union of the set A and all its limit points.
The boundary ∂A is deﬁned as the set of points which are in the closure of set A but not in the interior of set A .
Deﬁnition 5 (Multilinear) .
A function f : X 1 ×X 2 → Y is called multilinear if for arbitrary x 11 , x 21 ∈
X , x 12 , x 22 ∈ X 2 , and constants λ 1 , λ 2 , µ 1 , and µ 2 , we have
[Equation]
Remark.
The deﬁnition of “multilinear” implies that the domain of any multilinear function f is a connective and convex set, such as the smooth and multilinear cells below.
Deﬁnition 6 (Equivalence class, and quotient space) .
Suppose X is a linear space.
[ x ] = { v ∈ X : v ∼ x } is an equivalence class, if there is an equivalent relation ∼ on [ x ] , such that for any a, b, c ∈ [ x ] , we have: (1) reﬂexivity: a ∼ a ; (2) symmetry: if a ∼ b , b ∼ a ; and (3) transitivity: if a ∼ b and b ∼ c , a ∼ c .
The quotient space and quotient map are deﬁned to be
X/ ∼ = {{ v ∈ X : v ∼ x } : x ∈ X } and x → [ x ] , respectively.
4. M AIN RESULTS
In this section, the loss surface is deﬁned under convex loss with respect to the prediction ˆ Y of the neural network.
Convex loss covers many popular loss functions in practice, such as the squared loss for the regression tasks and many others based on norms.
The triangle inequality of the norms secures the convexity of the corresponding loss functions.
The convexity of the squared loss is checked in the appendix (see Appendix B, Lemma 8).
We now present four propositions to express the loss surfaces of nonlinear neural networks.
These propositions give four major properties of the loss surface that collectively draw a big picture for the loss surface.
We ﬁrst recall a lemma by Soudry & Hoffer (2018). It proves that the loss surfaces of neural networks have smooth and multilinear partitions. Lemma 1 (Smooth and multilinear partition; cf.
Soudry & Hoffer (2018)) .
The loss surfaces of neural networks of arbitrary depth with piecewise linear functions excluding linear functions are partitioned into multiple smooth and multilinear open cells, while the boundaries are nondifferentiable. Based on the smooth and multilinear partition, we prove three propositions as follows.
Theorem 2 (Analogous convexity) . For one-hidden-layer neural networks with two-piece linear activation for regression under convex loss, within every cell, all local minima are equally good, and also, they are all global minima in the cell.
Theorem 3 (Equivalence classes of local minimum valleys) . Suppose all conditions of Theoremhold.
Assume the loss function is strictly convex. Then, all local minima in a cell are concentrated as a local minimum valley: they are connected with each other by a continuous path and have the same empirical risk. Additionally, all local minima in a cell constitute an equivalence class. Corollary 1 (Quotient space of local minimum valleys) .
Suppose all conditions of Theorem 3 hold.
There might exist some “parallel” local minimum valleys in the equivalence class of a local minimum valley. They do not appear because of the constraints from the cell boundaries. If we ignore such constraints, all equivalence classes of local minima valleys constitute a quotient space. Corollary 2 (Linear collapse) .
The partitioned loss surface collapses to one single smooth and multilinear cell, when all activations are linear.
4. D ISCUSSIONS AND PROOF TECHNIQUES
This theorem characterizes how the nonlinearities in activations shape the loss surfaces of neural networks.
This section discusses the results and the structure of the proofs.
A detailed proof is omitted here and given in Appendix B.
Smooth and multilinear partition.
Intuitively, the nonlinearities in the piecewise linear activation functions partition the surface into multiple smooth and multilinear cells.
Zhou & Liang (2018);
Soudry & Hoffer (2018) highlight the partition of the loss surface.
We restate it here to make the picture self-contained.
A similar but also markedly different notions recently proposed by Hanin & Rolnick (2019) demonstrate that the input data space is partitioned into multiple linear regions, while our work focuses on the partition in the parameter space.
Every local minimum is globally minimal within a cell.
In convex optimization, convexity guarantees that all the local minima are global minima.
This theorem proves that the local minima within a cell are equally good, and also, they are all global minima in the cell.
This result is not surprising provided the excellent training performance of deep learning algorithms.
However, the proof is technically non-trivial.
Soudry & Hoffer (2018) proved that the local minima in a cell are the same.
However, there would be some point near the boundary has a smaller empirical risk and is not locally minimal.
Unfortunately, the proof by Soudry & Hoffer (2018) cannot exclude this possibility.
By contrast, our proof completely solves this problem.
Furthermore, our proof holds for any convex loss, including squared loss and cross-entropy loss, but Soudry & Hoffer (2018) only stands for squared loss.
It is challenging to prove, because the proof techniques for the case of linear networks cannot be transplanted here.
Technically, linear networks can be expressed by the product of a sequence of weight matrices, which guarantees good geometrical properties.
Speciﬁcally, the effect of every linear activation function is just equivalently multiplying a real constant to the output.
However, the loss surface within a cell of a nonlinear neural network does not have this property.
Below is the skeleton of our proof.
We ﬁrst prove that the empirical risk
ˆ
R is a convex function within every cell with respect to a variable ˆ W which is calculated from the weights W .
Therefore, all local minima of the empirical risk ˆ R with respect to ˆ W are also globally optimal in the cell.
Every cell corresponds to a speciﬁc series of linear parts of the activations.
Therefore, in any ﬁxed cell, the activation h s − ,s + can be expressed by the slopes of the corresponding linear parts as the following equations,
[Equation]
where A · ,i is the i -th column of matrix
[Equation]
Matrix A is constituted by collecting the slopes of the activation h at every point ( W 1 ) i, · x j .
Different elements of the matrix A can be multiplied either one of { s − , s + } .
Therefore, we cannot use a single constant to express the effect of this activation, and thus, even within the cell, a nonlinear network cannot be expressed as the product of a sequence of weight matrices.
This difference ensures that the proofs of deep linear neural networks cannot be transplanted here.
Then, we prove that (see p. 40)
[Equation]
Applying eq.
(8) to eq.
(7), the empirical risk ˆ R equals to a formulation similar to the linear neural networks,
[Equation]
Afterwards, deﬁne ˆ W 1 = diag ( W 2 ) W 1 and then straighten the matrix ˆ W 1 to a vector ˆ W ,
[Equation]
Deﬁne Q : ( W 1 , W 2 ) (cid:55)→ ˆ W , and also deﬁne,
[Equation]
We can prove the following equations (see p. 41),
[Equation]
Applying eq.
(9), the empirical risk is transferred to a convex function as follows,
[Equation]
We then prove that the local optimality of the empirical risk ˆ R is maintained when the weights W are mapped to the variable ˆ W .
Speciﬁcally, the local minima of the empirical risk ˆ R with respect to the weight W are also the local minima with respect to the variable ˆ W .
The maintenance of optimality is not surprising but the proof is technically non-trivial (see a detailed proof in pp.
Equivalence classes and quotient space of local minimum valleys.
The constructed mapping Q is a quotient map. Under the setting in the previous property, all local minima in a cell is an equivalence class; they are concentrated as a local minimum valley. However, there might exist some “parallel” local minimum valley in the equivalence class, which do not appear because of the constraints from the cell boundaries.
Further for neural networks of arbitrary depth, we also constructed a local minimum valley (the spurious local minima constructed in Section 3).
This result explains the property of mode connectivity that the minima found by gradient-based methods are connected by a path in the parameter space with almost constant empirical risk, which is proposed in two empirical works (Garipov et al., 2018; Draxler et al., 2018).
A recent theoretical work (Kuditipudi et al., 2019) proves that dropout stability and noise stability guarantee the mode connectivity.
Linear collapse.
Our theories also cover the case of linear neural networks.
Linear neural networks do not have any nonlinearity in their activations.
Correspondingly, the loss surface does not have any non-differentiable boundaries.
In our theories, when there is no nonlinearity in the activations, the partitioned loss surface collapses to a single smooth, multilinear cell.
All local minima wherein are equally good, and also, they are all global minima as follows.
This result unites the existing results on linear neural networks (Kawaguchi, 2016; Baldi & Hornik, 1989; Lu & Kawaguchi, 2017;
Freeman & Bruna, 2017; Zhou & Liang, 2018; Laurent & von Brecht, 2018;
Yun et al., 2018).
C ONCLUSION AND FUTURE DIRECTIONS
This paper reports that the nonlinearities in activations substantially shape the loss surfaces of neural networks.
First, we prove that neural networks have inﬁnitely many spurious local minima which are in contrast to the circumstance of linear neural networks.
This result stands for any neural network with arbitrary hidden layers and arbitrary piecewise linear activations (excluding linear functions) under many popular loss functions in practice (e.g., squared loss and cross-entropy loss).
This result signiﬁcantly extends the conditions of the relevant results and has the least restrictive assumptions that cover most practical circumstances: (1) the training data is not linearly separable; (2) the training sample points are distinct; (3) all hidden layers are wider than the output layer; and (4) there exists some turning point in the piece-wise linear activation that the sum of the slops on the two sides does not equal to 0 .
Second, based on a recent result that the loss surface has a smooth and multilinear partition, we draw a big picture of the loss surface from the following aspects: (1) local minima in any cell are equally good, and also, they are all global minima in the cell; (2) all local minima in one cell constitute an equivalence class and are concentrated as a local minimum valley; and (3) the loss surface collapses to one single cell when all activations are linear functions, which explains the results of linear neural networks.
The ﬁrst and second properties are rigorously proved for any onehidden-layer nonlinear neural networks with two-piece linear (ReLU-like) activations for regression tasks under convex/strictly convex loss without any assumption.
Theoretically understanding deep learning is of vital importance to both academia and industry.
A major barrier recognized by the whole community is that deep neural networks’ loss surfaces are extremely non-convex and even non-smooth.
Such non-smoothness and non-convexity make the analysis of the optimization and generalization properties prohibitively difﬁcult.
A natural idea is to bypass the geometrical properties and then approach a theoretical explanation.
We argue that such “intimidating” geometrical properties are exactly the major factors that shape the properties of deep neural networks, and also the key to explaining deep learning.
We propose to explore the magic of deep learning from the geometrical structures of its loss surface.
Future directions towards fully understanding deep learning are summarized as follows,
• Investigate the (potential) equivalence classes and quotient space of local minimum valleys for deep neural networks.
This paper suggests a degenerate nature of the large amounts of local minima: all the local minima within one cell constitute an equivalence class. We construct a quotient map for one-hidden-layer neural networks with two-piece activations for regression.
Whether deep neural networks have similar properties remains an open problem.
Understanding the quotient space would be a major step of understanding the approximation, optimization, and generalization of deep learning.
• Explore the sophisticated geometry of local minimum valleys.
The quotient space of local minima suggests a strategy that treats every local minimum valley as a whole.
However, the sophisticated local geometrical properties around the local minimum valleys are still premature, such as the sharpness/ﬂatness of the local minima, the potential categorization of the local minimum valley according to their performance, and the volumes of the local minima valleys from different categories.
• Tackle the optimization and generalization problems of deep learning.
Empirical results overwhelmingly suggest that deep learning has excellent optimization and generalization capabilities, which is, however, beyond the current theoretical understanding: (1) one can employ stochastic convex optimization methods (such as SGD) to minimize the extremely non-convex and non-smooth loss function in deep learning, which is expected to be NP-hard but practically solved by computationally cheap optimization methods; and (2) heavily-parametrized neural networks can generalize well in many tasks, which is beyond the expectation of most current theoretical frameworks based on hypothesis complexity and the variants.
The sophisticated geometrical expression, if fortunately we possess in the future, would be a compelling push to tackle the generalization and optimization muses of deep learning.
A CKNOWLEDGMENTS
This work was supported by Australian Research Council Project FL-170100117.
The authors sincerely appreciate Micah Goldblum and the anonymous reviewers for their constructive comments.
A P ROOF OF T HEOREM
This appendix gives a detailed proof of Theorem 1 omitted from the main text.
It follows the skeleton presented in Section 3.3.
A. S QUARED LOSS AND CROSS - ENTROPY LOSS
We ﬁrst check whether squared loss and cross-entropy loss are covered by the requirements of
Theorem 1.
Lemma 2.
The squared loss (deﬁned by eq. 5) is continuously differentiable with respect to the prediction of the model, whose gradient of loss equal to zero when the prediction and the label are different.
Proof.
Apparently, the squared loss is differentiable with respect to ˆ Y .
Speciﬁcally, the gradient with respect to ˆ Y is as follows,
[Equation]
which is continuous with respect to ˆ Y .
Also, when the prediction ˆ Y does not equals to the label Y , we have
[Equation]
The proof is completed.
Lemma 3.
The cross-entropy loss eq.
(6) is continuously differentiable with respect to the prediction of the model, whose gradient of loss equal to zero when the prediction and the label are different.
Also, we assume that the ground-truth label is a one-hot vector.
Proof.
For any i ∈ [1 : n ] , the cross-entropy loss is differentiable with respect to ˆ Y i .
The j -th component of the gradient with respect to the prediction ˆ
Y i is as follows,
[Equation]
which is continuous with respect to ˆ Y i .
So, the cross-entropy loss is continuously differentiable with respect to ˆ Y i .
Additionally, if the gradient (eq. (10)) is zero, we have the following equations,
[Equation]
Rewrite it into the matrix form, we have
[Equation]
Since d Y (cid:80) k =1 Y k,i = 1 , we can easily check the rank of the left matrix is d Y − 1 .
So the dimension of the solution space is one.
Meanwhile, we have  
[Equation]
−
[Equation]
,i · · · − Y 1 ,
i
[Equation]
d
[Equation]
k =
[Equation]
−
[Equation]
,i · · · − Y 2 ,i ...
− Y d
[Equation]
,i · · · − Y d Y ,i d Y (cid:80) i =1 Y k,i − Y d Y ,i 
  
Y 1 ,i Y 2 ,i ...
Y d Y ,i   = 0 .
Therefore, 0 (cid:54) =
e ˆ Y k,i = λY
k,i , for some λ ∈ R , which contradicts to the assumption that some of the components of Y is 0 ( Y i, · is a one-hot vector).
The proof is completed.
A. S TAGE (1)
In Stage (1), we prove that deep neural networks with one hidden layer, two-piece linear activation h s − ,s + , and multi-dimensional outputs have inﬁnite spurious local minima.
This stage is organized as follows: (a) we construct a local minimizer by Lemma 4; and (b) we prove that the local minimizer is spurious in Theorem 4 by constructing a set of parameters with smaller empirical risk.
Without loss of generality, we assume that s + (cid:54) = 0 .
Otherwise, suppose that s + = 0 .
From the deﬁnition of ReLU-like activation (eq. (4)), we have s − (cid:54) = 0 .
Since
[Equation]
, the output of the neural network with parameters (cid:110)
[ W
[Equation]
] Li =1 , [ b i ] Li =1 (cid:111) and activation h s − ,s + equals to that of the neural network with parameters (cid:110)
[ W (cid:48)
i ]
Li =1 , [ b (cid:48)
i ]
Li =1 (cid:111) and activation h − s + , − s − where
W (cid:48)
i = − W i , b (cid:48)
i = − b i , i = 1 , 2 , · · · , L − 1 and W (cid:48)
L = W L , b (cid:48)
L = b L .
Since (cid:110)
[ W i ]
Li
=1 , [ b i ] Li =1 (cid:111)
→ (cid:110)
[ W (cid:48)
i ]
Li =1 , [ b (cid:48)
i ]
Li =1 (cid:111) is an one-to-one map, it is equivalent to consider either the two networks, with h − s + , − s − ( x ) has non-zero slope when x > 0 .
Step (a).
Construct local minima of the loss surface.
Lemma 4.
Suppose that ˜ W is a local minimizer of
[Equation]
Under Assumption 3, any one-hidden-layer neural network has a local minimum at
[Equation] [Equation]
and
[Equation]
(13) where ˆ W 1 and ˆ b 1 are respectively the weight matrix and the bias of the ﬁrst layer, ˆ W 2 and ˆ b 2 are respectively the weight matrix and the bias of the second layer, and η is a negative constant with absolute value sufﬁciently large such that
[Equation]
where > is element-wise.
Also, the loss in this lemma is continuously differentiable loss whose gradient does not equals towhen the prediction is not the same as the ground-truth label.
Proof.
We show that the empirical risk is higher in the neiborhood of (cid:26)(cid:104)
[Equation]
, in order to prove that (cid:26)(cid:104)
ˆ W i (cid:105)i =
[Equation]
(cid:104)
ˆ
b i (cid:105)i = (cid:27) is a local minimizer.
The output of the ﬁrst layer before the activation is
[Equation]
Because η is a negative constant with absolute value sufﬁciently large such that eq.
(27)) holds, the output above is positive (element-wise), the output of the neural network with parameters { ˆ W 1 , ˆ W 2 , ˆ b 1 , ˆ b 2 } is
[Equation]
where ˜ X is deﬁned as
[Equation]
Therefore, the empirical risk ˆ R in terms of parameters { ˆ W 1 , ˆ W 2 , ˆ b 1 , ˆ b 2 } is
[Equation]
Then, we introduce a sufﬁciently small disturbance (cid:110)
[ δ Wi ] 2 i =1 , [ δ bi ] 2 i =1 (cid:111) into the parameters (cid:26)(cid:104)
ˆ W i (cid:105)i =1 , (cid:104)
ˆ
b i (cid:105)i = (cid:27) .
When the disturbance is sufﬁciently small, all components of the output of the ﬁrst layer remain positive.
Therefore, the output after the disturbance is ˆ
[Equation]
=
[Equation] [Equation]
=
[Equation]
Tn )
[Equation]
where eq.
( ∗ ) is because all components of (cid:16) ˆ W 1 + δ W 1 (cid:17) X
+ ( b (cid:48)
+ δ b 1 ) 1 Tn are positive
[Equation]
and δ is deﬁned as the following matrix
[Equation]
Therefore
[Equation]
the empirical risk ˆ R with respect to
[Equation]
δ approaches zero when the disturbances { δ W 1 , δ W 2 , δ b 1 , δ b 2 } approach zero (element-wise).
Since ˆ
W is the local minimizer of f ( W )
[Equation]
we have
[Equation]
Because the disturbances { δ W 1 , δ W 2 , δ b 1 , δ b 2 } are arbitrary
[Equation]
eq.
(16) demonstrates that
[Equation]
i = (cid:27) is a local minimizer.
The proof is completed.
Step (b).
Prove the constructed local minima are spurious.
Theorem 4.
Under the same conditions of Lemma 4 and Assumptions 1, 2, and 4, the constructed spurious local minima in Lemma 4 are spurious.
Proof.
The minimizer ˜ W is the solution of the following equation
[Equation]
Speciﬁcally, we have
[Equation]
Applying the deﬁnition of f ( W ) (eq. (11)),
[Equation]
x
[Equation]
where ˆ Y i =
˜ W (cid:20) x i 1 (cid:21) , ∇ ˆ Y i l (cid:18) Y i , ˜ W (cid:20) x i 1 (cid:21)(cid:19) ∈ R 1 × d Y .
Since k , j are arbitrary in { 1 , · · · , d Y } and { 1 , · · · , d X } , respectively, we have
[Equation]
(17) where
V =
(cid:20)(cid:18) ∇ ˆ Y 1 l (cid:18) Y 1 , ˜
[Equation]
We then deﬁne ˜ Y =
˜ W ˜ X .
Applying Assumption 1, we have
[Equation]
Thus, there exists some k -th row of ˜ Y − Y that does not equal to 0 .
We can rearrange the rows of ˜ W and Y simultaneously, while ˜ W is maintained as the local minimizer of f ( W ) and f ( ˜ W ) invariant 1 .
Without loss of generality, we assume k =
( k is the index of the row).
Set u
= V 1 , · and v i =
˜ Y 1 ,i in Lemma 7.
There exists a non-empty separation I =
[1 : l (cid:48) ] and J =
[ l (cid:48) + 1 : n ] of S
= { 1 , 2 , · · · , n } and a vector β ∈ R d X , such that (1.1) for any positive constant α small enough, and i ∈
I , j ∈ J , ˜ Y 1 ,i − αβ T x i <
˜
Y 1 ,j − αβ T x j ; (1.2) (cid:80) i ∈
I V 1 ,i (cid:54)
= 0 .
Deﬁne
[Equation]
Applying (1.1), for any i ∈
I
[Equation]
< 0 , while for any j ∈ J ,
[Equation]
> 0 .
Deﬁne γ ∈ R which satisﬁes
[Equation]
where s t +1 is deﬁned in Lemma
[Equation]
We argue that
[Equation]
When l (cid:48)
=
s t
+1 , eq.
(58) stands.
Also,
[Equation]
Therefore, we get eq.
(18) when α is small enough.
When l (cid:48) < s t
+1 , eq.
(57) stands.
Therefore,
[Equation]
, which apparently leads to eq.
Therefore, for any i ∈ I , we have that
[Equation]
<
[Equation]
while for any j ∈ J
[Equation]
Furthermore, deﬁne η i ( 2 ≤ i ≤ d Y ) as negative reals with absolute value sufﬁciently large, such that for any i ∈ [2 : d Y ] and any j ∈ [1 : n ] ,
[Equation]
Now we construct a point in the parameter space whose empirical risk is smaller than the proposed local minimum in Lemma 4 as follows
[Equation] [Equation]

[Equation]
· · · 0 · · ·   1 s
[Equation]
1 s + · · · ...
[Equation]
· 
[Equation]
(21) and
[Equation]
(22) where ˜ W i and ˜
b i are the weight matrix and the bias of the i -th layer, respectively.
After some calculations, the network output of the ﬁrst layer before the activation in terms of (cid:26)(cid:104)
[Equation]
Therefore, the output of the whole neural network is
[Equation]
Speciﬁcally, if j ≤ l (cid:48) ,
[Equation]
1 ,j − αβ T x j − η 1 + γ < 0 , (cid:18)
˜ Y
(1) (cid:18)(cid:104) ˜ W
i (cid:105)i =1 , (cid:104) ˜
b i (cid:105)i = (cid:19)(cid:19) 2 ,j = − ˜ W 1 , · (cid:20) x j (cid:21) + αβ T x j + η 1 + γ = − ˜ Y 1 ,j + αβ
T x j + η 1 + γ > 0 .
Therefore, (1 , j ) -th component of ˆ Y (cid:18)(cid:104)
˜ W
i (cid:105) 2 i =1 , (cid:104) ˜
b i (cid:105) 2 i = (cid:19) is (cid:18)
ˆ Y (cid:18)(cid:104)
˜ W i (cid:105)i =1 , (cid:104) ˜
b i (cid:105)i = (cid:19)(cid:19) 1 ,j = (cid:16) 1 s + + s − , − 1 s + + s − , 0 , · · · , 0 (cid:17) h s − ,s +     ˜ W 1 , · X − αβ T X − η 1 1 Tn + γ 1 Tn − ˜ W 1 , · X + αβ T X + η 1 1 Tn + γ 1 Tn ˜
W 2 , · X − η 2 1 Tn .
˜ W d Y , · X − η d Y 1 Tn 0 d 1 − d Y − 1 1 Tn     · ,j + η= s
+
+ s − h s − ,s + ( ˜ Y 1 ,j − αβ T x
j − η 1 + γ ) − s
+
+ s − h s − ,s + ( − ˜ Y 1 ,j + αβ T x j + η 1 + γ ) + η= s − s + + s − ( ˜ Y 1 ,
[Equation]
Similarly, when j > l (cid:48) , the ( 1 , j )-th component is
[Equation]
and
[Equation]
Thus
[Equation]
the empirical risk of the neural network with parameters (cid:26)(cid:104)
[Equation]
is
[Equation]
=n
[Equation]
Applying eqs.
[Equation] [Equation]
and (25)
[Equation]
we have n
[Equation]
where eq.
( ∗ ) is because
[Equation]
2 − ˜ W (cid:20)
[Equation]
1 (cid:21)(cid:19)
[Equation]
 [Equation] 0 , j ≥. Furthermore, note that α = O ( γ ) (from the deﬁnition of γ ).
We have n (cid:88) i = o
(cid:18)(cid:13)(cid:13) (cid:13) (cid:13)
˜
W 2 (cid:16) ˜ W
[Equation]
(cid:17) + ˜ b
[Equation]
− ˆ [Equation] (cid:20)
[Equation]
(cid:21)(cid:13)(cid:13) (cid:13) (cid:13) (cid:19) =
n (cid:88)
i =
o   (cid:118)(cid:117)(cid:117)(cid:116)
n (cid:88)
j = (cid:18)
˜ W 2 (cid:16) ˜
W 1 x i + ˜
b 1 (cid:17) + ˜ b 2 − ˜ W (cid:20) x i 1 (cid:21)(cid:19)j 
 =
o ( γ ) .
Let α be sufﬁciently small while sgn ( γ ) =
− sgn
[Equation]
We have
[Equation]
where inequality ( ∗∗ ) comes from (1.2) (see p. 18).
From Lemma 4, there exists a local minimizer (cid:26)(cid:104)
[Equation]
(cid:27) with empirical risk that equals to f ( ˜ W ) .
Meanwhile, we just construct a point in the parameter space with empirical risk smaller than f ( ˜ W ) .
Therefore, (cid:26)(cid:104)
ˆ W i (cid:105)i =1 , (cid:104) ˆ
[Equation]
i (cid:105)i =
[Equation]
(cid:27) is a spurious local minimum.
The proof is completed.
A. S TAGE (2)
Stage (2) proves that neural networks with arbitrary hidden layers and two-piece linear activation h s − ,s + have spurious local minima.
Here, we still assume s + (cid:54) = 0 .
We have justiﬁed this assumption in Stage (1).
This stage is organized similarly with Stage (1): (a) Lemma 5 constructs a local minimum; and (b)
Theorem 5 proves the minimum is spurious.
Step (a).
Construct local minima of the loss surface.
Lemma 5.
Suppose that all the conditions of Lemma 4 hold, while the neural network has L −hidden layers.
Then, this network has a local minimum at
[Equation]
and ˆ
[Equation]
where ˆ W (cid:48)
i and ˆ b (cid:48)
i are the weight matrix and the bias of the i -th layer, respectively, and η is a negative constant with absolute value sufﬁciently large such that
[Equation]
27) where > is element-wise.
Proof.
Recall the discussion in Lemma 4 that all components of ˆ W 1 X + ˆ b 1 1 Tn are positive.
Specifically,
[Equation]
where ˜ Y is deﬁned in Lemma 4.
Similar to the discussions in Lemma 4, when the parameters equal to (cid:26)(cid:104)
[Equation]
, the output of the ﬁrst layer before the activation function is
[Equation]
and
[Equation]
Here > is deﬁned element-wise.
After the activation function
[Equation]
the output of the ﬁrst layer is
[Equation]
We prove by induction that for all i ∈ [1 : L − 1] that ˜ Y ( i ) > 0 , element-wise ,
[Equation] [Equation]
Suppose that for 1 ≤ k ≤ L − 2 , ˜ Y ( k ) is positive (element-wise) and
[Equation]
Then the output of the ( k + 1) -th layer before the activation is
[Equation]
d Y
[Equation]

[Equation]
Applying eqs.
(28) and (29), we have
[Equation]
where > is deﬁned element-wise.
Therefore,
[Equation]
We thereby prove eqs.
(30) and (31).
Therefore, Y ( L ) can be calculated as
[Equation]
[Equation]
Then, we show the empirical risk is higher around (cid:26)(cid:104)
ˆ
[Equation]
in order to prove that
[Equation]
i =
[Equation]
(cid:27) is a local minimizer.
Let (cid:26)(cid:104)
[Equation] [Equation]
(cid:105) L
[Equation]
= (cid:27) be point in the parameter space which is close enough to the point (cid:26)(cid:104)
ˆ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˆ b (cid:48) i (cid:105)
L i = (cid:27) .
Since the disturbances δ (cid:48)
Wi and δ (cid:48) bi are both close to 0 (element-wise), all components of ˜ Y ( i ) (cid:18)(cid:104) ˆ W (cid:48)
i + δ (cid:48)
Wi (cid:105) L i =1 , (cid:104) ˆ b (cid:48)
i + δ (cid:48) bi (cid:105)
L i =1 (cid:19) remains positive.
Therefore, the output of the neural network in terms of parameters (cid:26)(cid:104)
ˆ
[Equation]
+ δ
[Equation]
Wi
[Equation]
L
[Equation]
is ˆ Y (cid:18)(cid:104)
[Equation]
where M 1 and M 2 can be obtained from (cid:26)(cid:104)
ˆ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˆ b (cid:48) i (cid:105)
L i = (cid:27) and (cid:110)
[ δ (cid:48)
Wi ]
Li =1 , [ δ (cid:48) bi ] Li =1 (cid:111) through several multiplication and summation operations 2 .
Rewrite the output as
[Equation]
1 Tn (cid:21) .
Therefore, the empirical risk ˆ R before and after the disturbance can be expressed as f ( ˜ W ) and f ([ M
M 2 ]) , respectively.
When the disturbances (cid:110)
[ δ (cid:48)
Wi ]
Li =1 , [ δ (cid:48) bi ] Li =1 (cid:111) approach
(element-wise), [ M
M 2 ] approaches ˜ W .
Therefore, when (cid:110)
[ δ (cid:48)
Wi ]
Li =1 , [ δ (cid:48) bi ] Li =1 (cid:111) are all small enough, we have
[Equation]
Since (cid:26)(cid:104)
ˆ W (cid:48)
i + δ (cid:48)
Wi (cid:105) L i =1 , (cid:104) ˆ
[Equation]
(cid:48)
i
[Equation]
(cid:48)
[Equation]
(cid:105) L i = (cid:27) are arbitrary within a sufﬁciently small neighbour of (cid:26) (cid:104) ˆ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˆ b (cid:48) i (cid:105)
L i = (cid:27) , eq.
(33) yields that
[Equation]
ˆ b (cid:48)
i (cid:105)i = (cid:27) is a local minimizer.
Step (b).
Prove the constructed local minima are spurious.
Theorem 5.
Under the same conditions of Lemma 5 and Assumptions 1, 2, and 4, the constructed spurious local minima in Lemma 5 are spurious.
Proof.
We ﬁrst construct the weight matrix and bias of the i -th layer as follows,
[Equation]
and
[Equation]
where ˜ W 1 , ˜ W 2 , ˜ b 1 and ˜ b 2 are deﬁned by eqs.
(19), (20), (21), and (22), respectively, and λ is a sufﬁciently large positive real such that
[Equation]
34) where > is deﬁned element-wise.
We argue that (cid:26)(cid:104)
[Equation]
corresponds to a smaller empirical risk than f ( ˜ W ) which is deﬁned in Lemma 4.
First, Theorem 4 has proved that the point (cid:110) ˜ W 1 , ˜ W 2 , ˜ b 1 , ˜ b 2 (cid:111) corresponds to a smaller empirical risk than f ( ˜ W ) .
We prove by induction that for any i ∈ { 3 , 4 , ..., L − 1 } ,
[Equation]
Apparently the output of the ﬁrst layer before the activation is
[Equation]
Therefore, the output of the ﬁrst layer after the activation is
[Equation]
, (cid:104) ˜ b (cid:48)
i (cid:105) L i =
(cid:19)
=
h s − ,s + (cid:18)
˜ Y (1) (cid:18)(cid:104) ˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i = (cid:19)(cid:19) =
h s − ,s + (cid:18)
˜ Y (1) (cid:18)(cid:104) ˜
W
i (cid:105) L i =1 , (cid:104) ˜
b i (cid:105) L i =
(cid:19)(cid:19) =
Y (1) (cid:18)(cid:104) ˜
W
i (cid:105)i =1 , (cid:104) ˜
b i (cid:105)i = (cid:19) .
Thus, the output of the second layer before the activation is
[Equation]
Applying the deﬁnition of λ (eq. (34)),
[Equation]
Therefore, the output of the second layer after the activation is
[Equation]
˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =
(cid:19)
=
h s − ,s + (cid:18) ˜ Y (2) (cid:18)(cid:104) ˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i = (cid:19)(cid:19) =
s +  
ˆ Y (cid:18)(cid:104)
˜ W i (cid:105)i =1 , (cid:104) ˜
b i (cid:105)i = (cid:19) + λ 1 d Y 1 Tn λ 1 d 2 − d Y 1 Tn   .
Meanwhile, the output of the third layer before the activation is ˜ Y (3) (cid:18)(cid:104) ˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =1 (cid:19) can be calculated based on Y (2) (cid:18)(cid:104) ˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =
(cid:19) :
˜ Y (3) (cid:18)(cid:104) ˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =
(cid:19) =
˜ W (cid:48) 3 Y (2) (cid:18)(cid:104) ˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =
(cid:19)
+ ˜ b (cid:48) 3 1 Tn
[Equation]
Applying eq.
[Equation]
Therefore
[Equation]
the output of the third layer after the activation is
[Equation]
Suppose eqs.
(35) and (36) hold for k ( 3 ≤ k ≤ L − 2 ),
when k + 1 ,
[Equation]
Applying eq.
[Equation]
Therefore, the output of the ( k + 1 )-th layer after the activation is
[Equation]
˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =
(cid:19)
=
h s − ,s + (cid:18)
˜ Y ( k +1) (cid:18)(cid:104) ˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i = (cid:19)(cid:19) =
s + ˜ Y
( k +1) (cid:18)(cid:104) ˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =
(cid:19)
=
s +  
ˆ Y (cid:18)(cid:104)
˜ W i (cid:105)i =1 , (cid:104) ˜
b i (cid:105)i = (cid:19) + λ 1 d Y 1 Tn 0 ( d k +1 − d Y ) × n   .
Therefore, eqs.
(35) and (36) hold for any i ∈ { 3 , 4 , ..., L − 1 } .
Finally, the output of the network is
[Equation]
(cid:19)
[Equation]
L i
[Equation] [Equation]
Applying Theorem
[Equation]
we have
[Equation]
The proof is completed.
A. S TAGE (3)
Finally, we prove Theorem 1.
This stage also follows the two-step strategy.
Step (a).
Construct local minima of the loss surface.
Lemma 6.
Suppose t is a non-differentiable point for the piece-wise linear activation function h and σ is a constant such that the activation h is differentiable in the intervals ( t − σ, t ) and ( t, t + σ ) .
Assume that M is a sufﬁciently large positive real such that
[Equation]
Let α i be any positive real such that
[Equation]
Then, under Assumption 3, any neural network with piecewise linear activations and L − 1 hidden layers has local minima at
[Equation]
and ˆ
[Equation]
where (cid:26)(cid:104)
ˆ W (cid:48)
i (cid:105) L i =1 , [Equation]
L i = (cid:27) is the local minimizer constructed in Lemma 5.
Also, the loss is continuously differentiable, whose derivative with respect to the prediction ˆ Y i may equal to 0 only when the prediction ˆ Y i and label Y i are the same.
Proof.
Deﬁne s − =
lim θ → 0 − h (cid:48)
( θ ) and s + =
lim θ → 0 + h (cid:48)
( θ ) .
We then prove by induction that for all i ∈ [1 : L − 1] , all components of the i -th layer output before the activation ˜ Y ( i ) (cid:18)(cid:104)
[Equation]
i =1 , (cid:104)
[Equation]
(cid:105) L i = (cid:19) are in interval ( t, t + σ ), and
[Equation]
The ﬁrst layer output before the activation is,
[Equation] [Equation]
We proved in Lemma 5 that ˆ W (cid:48) 1 X + ˆ b (cid:48)
1 1 Tn is positive (element-wise).
Since the Frobenius norm of a matrix is no smaller than any component’s absolute value
[Equation]
applying eq.
[Equation]
we have that for all i ∈ [1 , d 1 ] and j ∈ [1 : n ] ,
[Equation]
Therefore,
[Equation]
where eq.
( ∗ ) is because for any x ∈ ( t − σ, t + σ ) ,
[Equation]
Suppose the above argument holds for k ( 1 ≤ k ≤ L − 2 ).
Then
[Equation]
Lemma 5 has proved that all components of ˜ Y ( k +1) (cid:18)(cid:104)
ˆ W
[Equation]
i
[Equation]
=
[Equation]
ˆ b
[Equation]
L i =1 (cid:19) are contained in ˜ Y (1) (cid:18)(cid:104) ˆ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˆ b (cid:48) i (cid:105)
L i = (cid:19) .
Combining
[Equation]
we have
[Equation]
Here < are all element-wise, and inequality ( ∗ ) comes from the property of α i (eq.
[Equation]
Furthermore
[Equation]
the ( k + 1) -th layer output after the activation is
[Equation]
where eq.
( ∗ ) is because of eq.
The above argument is proved for any index k ∈ { 1 , . . .
, L − 1 } .
Therefore, the output of the network is
[Equation]
Therefore,
[Equation]
We then introduce some small disturbances (cid:110)
[Equation]
into (cid:26)(cid:104)
[Equation]
in order to check the local optimality.
Since all comonents of Y ( i ) are in interval ( t, t + σ ) , the activations in every hidden layers is realized at linear parts.
Therefore, the output of network is
[Equation]
=
[Equation]
M
[Equation]
] (cid:20)
[Equation]
Tn (cid:21) .
Similar to Lemma 5, [ M
M 2 ] approaches ˜ W as disturbances (cid:110)
[ δ Wi ]
Li =1 , [ δ bi ] Li =1 (cid:111) approach(element-wise).
Combining that ˜ W is a local minimizer of f ( W ) , we have ˆ R (cid:18)(cid:104) ˆ W (cid:48)(cid:48) i + δ (cid:48)(cid:48)
Wi (cid:105) L
i =1 , (cid:104)
ˆ b (cid:48)(cid:48)
i + δ (cid:48)(cid:48)
[Equation]
The proof is completed.
Step (b).
Prove the constructed local minima are spurious.
Proof of Theorem 1.
Without loss of generality, we assume that all activations are the same.
Let t be a non-differentiable point of the piece-wise linear activation function h with
[Equation]
Let σ be a constant such that h is linear in interval ( t − σ, t ) and interval ( t, t + σ ) .
Then construct that
[Equation]
and ˜
[Equation]
where (cid:26)(cid:104) ˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =
(cid:27) are constructed in Theorem
[Equation]
M is a large enough positive real such that
[Equation] [Equation]
and ˜ M a large enough positive real such that
[Equation]
L
[Equation]
L
[Equation]
F < σ.
Then, we prove by induction
that for any i ∈ [ :
L − 1] , all components of ˜ Y ( i ) (cid:18)(cid:104) ˜ W (cid:48)(cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)(cid:48)
i (cid:105) L i =
(cid:19) are in interval ( t, t + δ ) , and
Y ( i ) (cid:18)(cid:104) ˜
W (cid:48)(cid:48) i (cid:105) L
i =1 , (cid:104) ˜
b (cid:48)(cid:48)
i (cid:105) L i =
(cid:19)
=
h ( t ) 1 d i 1 Tn + ˜ MM Y ( i ) (cid:18)(cid:104) ˜
W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =
(cid:19) .
First, ˜ Y (1) (cid:18)(cid:104) ˜ W (cid:48)(cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)(cid:48)
i (cid:105) L i =
(cid:19) =
˜ W (cid:48)(cid:48) 1 X + ˜ b (cid:48)(cid:48) 1 1 Tn = 1 M ( ˜ W (cid:48)
1 X + ˜ b (cid:48) 1 1 Tn ) + t
[Equation]
For any i ∈ [1 : d 1 ] and j ∈ [1 : n ] , eq.
(45) implies
[Equation]
Thus
[Equation]
Therefore
[Equation]
the output of the ﬁrst layer after the activation is
[Equation]
[Table]
Also,
[Equation]
Recall in Theorem 5 we prove all components of ˜ Y (2) (cid:18)(cid:104)
˜ W
[Equation]
(cid:105) L i =
[Equation]
˜ b
[Equation]
i (cid:105) L i =1 (cid:19) are positive.
Combining the deﬁnition of ˜ M (eq. (46))
[Equation]
we have
[Equation]
Therefore,
[Equation]
Suppose the above argument holds for k -th layer
[Equation]
The output of ( k + 1) -th layer before the activation is
[Equation]
Recall proved in Theorem 5 that all components of ˜ Y ( k +1) (cid:18)(cid:104) ˜ W
cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =1 (cid:19) except those that are 0 are contained in ˜ Y ( k ) (cid:18)(cid:104)
˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =1 (cid:19) .
We have
[Equation]
Therefore,
[Equation]
Thus
[Equation]
the argument holds for any k ∈ { 2 , . . .
, L − 1 }
[Equation]
So,
[Equation]
Therefore, ˆ R (cid:18)(cid:104)
[Equation]
From eq.
(49) and Theorem 5, we have
[Equation]
which completes the proof of local minimizer.
Furthermore, the parameter M used in Lemma 6 (not those in this proof) is arbitrary in a continuous interval (cf. eq.
(40)), we have actually constructed inﬁnite spurious local minima.
Theorem 1 relies on Assumption 4.
We can further remove it by replacing Assumption 3 by a mildly more restrictive variant Assumption 5.
Corollary 3.
Suppose that Assumptions 1, 2, and 5 hold.
Neural networks with arbitrary depth and arbitrary piecewise linear activations (excluding linear functions) have inﬁnitely many spurious local minima under arbitrary continuously differentiable loss whose derivative can equal 0 only when the prediction and label are the same.
Proof.
The proof is delivered by modiﬁcations of Theorem 4 in Stage 1 of Theorem 1’s proof.
We only need to prove the corollary under the assumption that s − + s + = 0 .
Let the local minimizer constructed in Lemma 4 be (cid:26)(cid:104)
ˆ W i (cid:105)
i =1 , (cid:104) ˆ b
[Equation]
[Equation]
= (cid:27) .
Then, we construct a point in the parameter space whose empirical risk is smaller as follows:
˜ W 1 =   ˜ W 1 , [1: d X ] − αβ T ˜
W 1 , [1: d X ] − ˜ W 1 , [1: d X ] + αβ T ˜
W 2 , [1: d X ] .
˜ W d Y , [1: d X ]( d 1 − d Y − 2) ×
d X   , ˜ b
1 =      ˜ W 1 , [ d X +1] − η 1 + γ ˜
W 1 , [ d X +1] − η − ˜ W 1 , [ d X +1] + η 1 + γ ˜
W 2 , [ d X +1] − η 2 .
˜ W d Y , [ d X +1] − η d Y( d 1 − d
Y − 2) ×     , ˜ W 2 =   12 s + 1 s + − 12 s +  · · · 0 · · ·    1 s + · · · 0 · · ·     1 s + · · · 0 · · · ...
· · · 1 s + 0 · · ·   , and ˜ b 2 =   ηη... η d Y   , where α , β , and η i are deﬁned the same as those in Theorem 4, and η is deﬁned by eq.
Then, the output of the ﬁrst layer is Y
[Equation]
i =
[Equation]
, (cid:104) ˜ b
[Equation]
i = (cid:19)
[Equation]
Further, the output of the whole network is
[Equation]
Therefore, if j ≤ l (cid:48) , the (1 , j ) -th component of ˆ Y (cid:18)(cid:104) ˜ W
i
[Equation]
2 i =1 , (cid:104) ˜
b i (cid:105) 2 i =1 (cid:19) is (cid:16) ˜
W 2 (cid:17)
˜ Y (1) (cid:18)(cid:104) ˜
W
i (cid:105)i =1 , (cid:104) ˜
b i (cid:105)i = (cid:19)
j
=2 s + (cid:16) s − (cid:16)
˜ Y 1 ,j − αβ T x
j − η 1 + γ (cid:17) + 2 s + (cid:16) ˜
Y 1 ,j − η (cid:17) − s + (cid:16) − ˜ Y
j + αβ T x j + η 1 + γ (cid:17)(cid:17) + η =2 s + (cid:16) − s + (cid:16) ˜
Y 1 ,j − αβ T x
j − η 1 + γ (cid:17) + 2 s + (cid:16) ˜
Y 1 ,j − η (cid:17) − s + (cid:16) − ˜ Y
j + αβ T x j + η 1 + γ (cid:17)(cid:17) + η =2 s
+ (cid:16) 2 s + ˜ Y 1 ,j − 2 s + η − 2 s + γ (cid:17) + η = ˜
Y 1 ,j − γ.
Otherwise ( j > l (cid:48) ), the (1 , j ) -th component of ˆ Y (cid:18)(cid:104)
˜ W
i (cid:105) 2 i =1 , (cid:104) ˜
b i (cid:105) 2 i =1 (cid:19) is (cid:16) ˜
W 2 (cid:17)
˜ Y (1) (cid:18)(cid:104) ˜
W
i (cid:105)i =1 , (cid:104) ˜
b i (cid:105)i = (cid:19)
j
=2 s + (cid:16) s + (cid:16)
˜ Y 1 ,j − αβ T x
j − η 1 + γ (cid:17) + 2 s + (cid:16) ˜
Y 1 ,j − η (cid:17) − s − (cid:16) − ˜ Y
j + αβ T x j + η 1 + γ (cid:17)(cid:17) + η =2 s
+ (cid:16) s + (cid:16)
˜ Y 1 ,j − αβ T x
j − η 1 + γ (cid:17) + 2 s + (cid:16) ˜
Y 1 ,j − η (cid:17)
+ s + (cid:16) − ˜ Y
j + αβ T x j + η 1 + γ (cid:17)(cid:17) + η =2 s
+ (cid:16) 2 s + ˜ Y 1 ,j − 2 s + η
+ 2 s + γ (cid:17) +
η = ˜
Y 1 ,j + γ, and the ( i, j ) -th
( i > 1 ) component of ˆ Y (cid:18)(cid:104)
˜ W
i (cid:105) 2 i =1 , (cid:104) ˜
b i (cid:105) 2 i = (cid:19) is ˜ Y i,j .
Therefore
[Equation]
we have
[Equation]
Then
[Equation]
similar to Theorem
[Equation]
we have
[Equation]
where V and l (cid:48) are also deﬁned the same as those in Theorem 4.
When γ is sufﬁciently small and sgn ( γ ) = sgn (cid:16)(cid:80) l (cid:48)
i =1 V 1 ,i (cid:17) , we have that
[Equation]
This complete the proof of Corollary 3.
A. A PREPARATION LEMMA
We now prove the preparation lemma used above.
Lemma 7.
Suppose u = ( u 1 · · · u n ) ∈ R 1 × n which satisﬁes u (cid:54) = 0 and n (cid:88)
i =
u i = 0 , (50) while { x 1 ,..., x n } is a set of vector ⊂
R m × 1 .
Suppose index set S = { 1 , 2 , · · · , n } .
Then for any series of real number { v 1 , · · · , v n } , there exists a non-empty separation I , J of S , which satisﬁes
I ∪ J = S , I ∩ J = ∅
and both I and J are not empty, a vector β ∈ R m × 1 ,such that, (1.1) for any sufﬁciently small positive real α , i ∈ I , and j ∈ J , we have v i − αβ T x
i < v j − αβ
T x j ; (1.2) (cid:80) i ∈ I
u i (cid:54) = 0 .
Proof.
If there exists a non-empty separation I and J of the index set S , such that when β = 0 , (1.1) and (1.2) hold, the lemma is apparently correct.
Otherwise, suppose that there is no non-empty separation I and J of the index set S such that (1.1) and (1.2) hold simultaneously when β = 0 .
Some number v i in the sequence ( v 1 , v 2 , · · · , v n ) are probably equal to each other.
We rerarrange the sequence by the increasing order as follows,
[Equation]
51) where s k = n .
Then, for any j ∈ { 1 , 2 , · · · , k − 1 } , we argue that
[Equation]
Otherwise, suppose there exists a s j , such that
[Equation]
Let I = { 1 , 2 , ..., s j } and J = { s j + 1 , ..., n } .
Then, when β = 0 , we have
[Equation]
and
[Equation]
which are exactly the arguments (1.1) and (1.2).
Thereby we construct a contrary example.
Therefore, for any j ∈ { 1 , 2 , · · · , k − 1 } , we have
[Equation]
Since we assume that u (cid:54) = 0 , there exists an index t ∈ { 1 , . .
, k − 1 } , such that there exists an index i ∈ { s t + 1 , ..., s t +1 } that u i (cid:54) = 0 .
Let l ∈ { s t + 1 , ..., s t +1 } is the index such that x l has the largest norm while u l (cid:54)
= 0 :
[Equation]
[Equation]
We further rearrange the sequence ( v s t +1 , ..., v s t +1 )
such that there is an index l (cid:48) ∈ { s t + 1 , . . .
, s t +1 } ,
[Equation]
, and
[Equation]
∀
[Equation]
It is worth noting that it is probably l (cid:48)
=
s t +1 , but it is a trivial case that would not inﬂuence the result of this lemma.
Let I = { 1 , ..., l (cid:48) } , J = { l (cid:48) + 1 , ..., n } , and β = x l (cid:48) .
We prove (1.1) and (1.2) as follows.
Proof of argument (1.1) .
We argue that for any i ∈
I , v i − αβ T x i ≤ v l (cid:48) − αβ T x l (cid:48) and for any j ∈ J , v j − αβ T x j > v l (cid:48) − αβ T x l (cid:48) .
There are three situations:
(A) i ∈ { 1 , . . .
, s t } and j ∈ { s t +1 + 1 , · · · , n } .
Applying eq.
(51), for any i ∈ { 1 , . . .
, s t } and j ∈ { s t +1 + 1 , · · · , n } , we have that v i < v l (cid:48) and v j > v l (cid:48) .
Therefore, when α is sufﬁciently small, we have the following inequalities,
[Equation]
(B) i ∈ { s t + 1 , · · · , l (cid:48) } .
Applying eq.
(53) and because of α >
[Equation]
we have
[Equation]
Since v i = v l (cid:48) , it further leads to
[Equation]
C)
[Equation]
∈ { l
[Equation]
+ 1 , · · · , s t +1 } .
Similarly, applying eq.
(54) and because of α > 0 , we have
[Equation]
Since v j = v l (cid:48) , it further leads to
[Equation]
which is exactly the argument (1.1).
Proof of argument (1.2).
We argue that for any i ∈ { s t + 1 , · · · , l (cid:48) − 1 } , u i = 0 .
Otherwise, suppose there exists an i ∈ { s t + 1 , · · · , l (cid:48) − 1 } such that u i (cid:54) = 0 .
From eq.
(52), we have (cid:107) x
i (cid:107) ≤ (cid:107) x l (cid:48) (cid:107) .
Therefore,
[Equation]
where the ﬁrst inequality strictly holds if the vector x l (cid:48) and x i have the same direction, while the second inequlity strictly holds when x i and x l (cid:48) have the same norm.
Because x l (cid:48) (cid:54) =
x i , we have the following inequality,
[Equation]
which contradicts to eq.
i
[Equation]
e.,
[Equation]
Therefore,
[Equation]
which is exactly the argument (1.2)
[Equation]
The proof is completed.
Remark.
For any i ∈ { l (cid:48) + 1 , ..., s t +1 } , we have
[Equation]
55) while for any j ∈ { s t +1 + 1 , ..., n } , we have
[Equation]
Because v j > v l (cid:48) , when the real number α is sufﬁciently small, we have
[Equation]
Applying eqs.
(55) and (56), we have
[Equation]
Therefore, if l (cid:48) < s t +1 , we have min i ∈ l [Equation] [Equation]
[Equation]
while if l (cid:48) = s t +1 , min i
[Equation]
[Equation]
[Equation]
Eqs.
(57) and (58) make senses because l (cid:48) < n .
Otherwise, from Lemma 7 we have (cid:80) ni
=1 u i (cid:54) = 0 , which contradicts to the assumption.
B P ROOFS OF T HEOREM 2, T HEOREM 3, C OROLLARY 1, AND C OROLLARY
This appendix gives the proofs of Theorem 2, Theorem 3, Corollary 1, and Corollary 2 omitted from Section 4.
We ﬁrst check that the squared loss is strictly convex, which is even restrictive than “convex”.
Lemma 8.
The empirical risk ˆ R under squared loss (deﬁned by eq.
(5)) is strictly convex with respect to the prediction ˆ Y .
Proof.
The second derivative of the empirical risk ˆ R under squared loss with respect to the prediction ˆ Y is
[Equation]
Therefore, the empirical risk ˆ R under squared loss is strictly convex with respect to prediction ˆ Y .
B. S MOOTH AND MULTILINEAR PARTITION .
If the activations are all linear functions, the neural networks is reduced to a multilinear model.
The loss surface is apparently smooth and multilinear.
The nonlinearity in the activations largely reshape the landscape of the loss surface.
Speciﬁcally, if the input data ﬂows through the linear parts of every activation functions, the output falls in a smooth and multilinear region in the loss surface.
When some parameter changes by a sufﬁciently small swift, the data ﬂow may not move out of the linear parts of the activations.
This fact guarantees that each smooth and multilinear regions expands to an open cell.
Meanwhile, every nonlinear point in the activations is non-differentiable.
If the input data ﬂows through these nonlinear points, the corresponding empirical risk is not smooth with respect to the parameters.
Therefore, the nonlinear points in activations correspond to the non-differentiable boundaries between cells on the loss surface.
B. E VERY LOCAL MINIMUM IS GLOBALLY MINIMAL WITHIN A CELL .
Proof of Theorem 2.
In every cell, the input sample points ﬂows through the same linear parts of the activations no matter what values the parameters are.
(1) We ﬁrst proves that the empirical risk ˆ R equals to a convex function with respect to a variable ˆ
W that is calculated from the parameters W .
Suppose ( W 1 , W 2 ) is a local minimum within a cell.
We argue that
[Equation]
where A · ,i is the i -th column of the following matrix
[Equation]
The left-hand side (LHS) is as follows,
[Equation]
Meanwhile, the right-hand side (RHS) is as follows,
[Equation]
Apparently, LHS = RHS.
Thereby, we proved eq.
Afterwards, we deﬁne
[Equation]
and then straighten the matrix ˆ W 1 to a vector ˆ W ,
[Equation]
Also deﬁne
[Equation]
Then, we can prove that the following equations,
[Equation]
Applying eq.
(64), the empirical risk is transferred to a convex function as follows,
[Equation]
We can see that the empirical risk is rearranged as a convex function in terms of ˆ W which unite the two weight matrices W 1 and W 2 and the activation h are together as ˆ W .
Applying eqs.
(61) and (62), we have
[Equation]
(2) We then prove that the local minima (including global minima) of the empirical risk ˆ R with respect to the parameter W is also local minima with respect to the corresponding variable ˆ W .
We ﬁrst prove that for any i ∈ [1 : d 1 d 2 ] , we have
[Equation]
where ∇ is deﬁned as follows,
[Equation]
To see this, we divide i into two cases:
( W 2 ) i (cid:54) =
and ( W 2 ) i = 0 .
Case 1: ( W 2 ) i (cid:54) = 0 .
The local minimizer of the empirical risk ˆ R with respect to the parameter W satisﬁes the following equation,
[Equation]
Therefore,
[Equation]
where ( W 2 ) is a vector and ( W 2 )
i is its i -th component.
Then, divid the both hand sides of eq.
(66) with ( W 2 ) i , we can get the following equation,
[Equation]
Case 2: ( W 2 ) i = 0 .
Suppose u 1 ∈ R d 0 is a unitary vector, u 2 ∈ R is a real number [Equation] and ε is a small enough positive constant.
Then, deﬁne a disturbance of W 1 and W 2 as follows,
[Equation]
When ε is sufﬁciently small, ∆ W 1 and ∆ W 2 are also sufﬁciently small.
Since ( W 1 , W 2 ) is a local minimum, we have
[Equation]
where ∆ is deﬁned as follows,
[Equation]
[Equation]
Here, eq.
( ∗ ) comes from ( W 2 ) i = 0 .
Rearrange eq.
(67) and apply the Taylor’s Theorem, we can get that
[Equation]
Applying eq.
(68), we have (cid:2) 0 · · ·
[Table]
u
[Table]
u
[Table] + [Table]
W
[Table]
(cid:124)
[Table]
+ ε 4 O (cid:16) (cid:107) [ 0 · · ·
[Table] u [Table] [Table]
u [Table] + [Table]
W
[Table]
[Table]
[Table]
+ ε
[Table]
[Table]
[Table]
= ε 3 [ 0 · ·
[Table]
0 u 2 u
[Table]
[Table]
[Table]
d
[Table]
[Equation]
Here, eq.
( ∗∗ ) can be obtained from follows.
Because W 2 is a local minimizer, for any component ( W 2 ) i of W 2 ,
[Equation]
which leads to
[Equation]
When ε approaches 0 , eq.
(69) leads to the following inequality,
[Equation]
Since u 1 and u 2 are arbitrarily picked (while the norms equal 1 ), the inequality above further leads to
[Equation]
which ﬁnishes the proof of the argument.
Therefore, for any i and j , we have proven that
[Equation]
which demonstrates that ˆ X ∇ = 0 , which means ˆ W is also a local minimizer of the empirical risk ˆ R ,
[Equation]
(3) Applying the property of convex function, ˆ W is a global minimizer of the empirical risk
ˆ R , which leads to ( W 1 , W 2 ) is a global minimum inside this cell
The proof is completed. B. E QUIVALENCE CLASSES OF LOCAL MINIMUM VALLEYS IN CELLS .
Proof of Theorem 3 and Corollary 1. In the proof of Theorem 2, we constructed a map Q : ( W 1 , W 2 ) → ˆ W . Further, in any ﬁxed cell, the represented hypothesis of a neural network is uniquely determined by ˆ W .
We ﬁrst prove that all local minima in a cell are concentrated as a local minimum valley. Since the loss function l is strictly convex, the empirical risk has one unique local minimum (which is also a global minimum) with respect to ˆ W in every cell, if there exists some local minimum in the cell. Meanwhile, we have proved that all local minima with respect to ( W 1 , W 2 ) are also local minima with respect to the corresponding ˆ W . Therefore, all local minima with respect to ( W 1 , W 2 ) correspond one unique ˆ W .
Within a cell, when W 1 expands by a positive real factor α to W (cid:48) 1 and W 2 shrinks by the same positive real factor α to W (cid:48) 2 , we have Q ( W 1 , W 2 ) = Q ( W (cid:48) 1 , W (cid:48) 2 ) , i.e., the ˆ W remains invariant.
Further, we argue that all local minima in a cell are connected with each other by a continuous path, on which the empirical risk is invariant.
For every local minima pair ( W 1 , W 2 ) and ( W (cid:48) 1 , W (cid:48) 2 ) , we have
[Equation] Since h (cid:48) s − ,s + ( W [Equation] =
h (cid:48) s
[Equation]
,s + ( W (cid:48)
[Equation]
) (element-wise), for every i ∈ [1 , d 1 ] , sgn (
[Equation]
i
[Equation]
=
[Equation]
Therefore, a continuous path from ( W 1 , W 2 ) to ( W (cid:48) 1 , W (cid:48) 2 ) can be constructed by ﬁnite moves, each of which expands a component of W 2 by a real constant α
and then shrinks the corresponding line of W 1 by the same constant α .
We then prove that all local minima in a cell constitute an equivalence class.
Deﬁne an operation ∼ R as follows,
[Equation]
if
Q ( W 11 , W
[Equation]
= Q
[Equation]
W 21 , W 22 ) .
We then argue that ∼ R is an equivalence relation.
The three properties of equivalence relations are checked as follows.
1) Reﬂexivity
For any ( W 1 , W 2 ) , we have
[Equation]
Therefore, (
[Equation]
[Equation]
Symmetry:
For any pair ( W 11 , W 12 ) and ( W 21 , W 22 ) , Suppose that
[Equation]
Thus,
Q ( W 11 , W 12 ) =
Q ( W 21 , W 22 ) .
Apparently,
[Equation] [Equation] [Equation] Therefore,
Q ( W 21 , W
[Equation]
[Equation]
(3) Transitivity
For any ( W 11 , W 12 ) , ( W 21 , W 22 ) , and ( W 31 , W 32 ) , suppose that
[Equation]
Then,
[Equation]
Apparently,
[Equation]
Therefore,
[Equation]
We then prove the mapping Q is the quotient map.
Deﬁne a map as follows,
[Equation]
We then deﬁne an operator ⊕ as,
[Equation]
the inverse of ( W 1 , W 2 ) is deﬁned to be ( − W 1 , W 2 ) and the zero element is {+deﬁned to be ( 0 , 
 1 × d 1 ) . Obviously,+} the following is a linear mapping: [Equation]
For any pair ( W 11 , W 12 ) and ( W 21 , W 22 ) , we have
[Equation]
if and only if [Equation] Therefore, the quotient space ( R d 1 × d X , R 1 × d 1 ) / Ker ( Q ) is a deﬁnition of the equivalence relation ∼ R .
The proof is completed.
B. L INEAR COLLAPSE .
When there is no nonlinearities in the activations, there is apparently no non-differentiable regions on the loss surface.
In other words, the loss surface is a single smooth and multilinear cell.