N ONLINEARITIES IN ACTIVATIONS SUBSTANTIALLY SHAPE THE LOSS SURFACES OF NEURAL NETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Understanding the loss surfaces of neural networks is fundamentally important to understanding deep learning.
This paper presents how the nonlinearities in activations substantially shape the loss surfaces of neural networks.
We ﬁrst prove that the loss surfaces of many neural networks have inﬁnite spurious local minima , which are deﬁned as the local minima with higher empirical risks than the global minima.
Our result holds for any neural network with arbitrary depth and arbitrary piecewise linear activation functions (excluding linear functions) under most loss functions in practice with some mild assumptions.
This result demonstrates that nonlinear networks possess substantial differences to the well-studied linear neural networks.
Essentially, the underlying assumptions for the above result are consistent with most practical circumstances where the output layer is narrower than any hidden layer.
We further prove a theorem that draws a big picture for the loss surfaces of nonlinear neural networks from the following respects.
(1) Smooth and multilinear partition : the loss surface is partitioned into multiple smooth and multilinear open cells by nondifferentiable boundaries.
Local analogous convexity : within every cell, local minima are equally good and also, they are all global minima in the cell.
Local minima valley : some local minima are connected within a cell by a continuous path, on which all points have the same empirical risk.
(4) Linear collapse : when all activations are linear, the partitioned loss surface collapses to one single cell, which includes linear neural networks as a simpliﬁed case.
The second result holds for one-hidden-layer networks for regression under convex loss, while all others apply to networks of arbitrary depth.
I NTRODUCTION
Neural networks have been successfully deployed in many real-world applications (LeCun et al., 2015;
Witten et al., 2016; Silver et al., 2016;
He et al., 2016; Litjens et al., 2017).
In spite of this, the theoretical foundations of neural networks are somewhat premature.
To the many deﬁciencies in our knowledge of deep learning theory, the investigation into the loss surfaces of neural networks is of fundamental importance.
Understanding the loss surface would be helpful in several relevant research areas, such as the ability to estimate data distributions, the optimization of neural networks, and the generalization to unseen data.
This paper studies the role of the nonlinearities in activation functions in shaping the loss surfaces of neural networks.
Our results demonstrate that the impact of nonlinearities is profound.
First, we prove that the loss surfaces of nonlinear neural networks are substantially different to those of linear neural networks, in which local minima are created equal, and also, they are all global minima (Kawaguchi, 2016; Baldi & Hornik, 1989; Lu & Kawaguchi, 2017; Freeman & Bruna, 2017; Zhou & Liang, 2018; Laurent & von Brecht, 2018; Yun et al., 2018).
By contrast,
Neural networks with arbitrary depth and arbitrary piecewise linear activations (excluding linear functions) have inﬁnitely many spurious local minima under arbitrary continuously differentiable loss functions.
This result only relies on three mild assumptions that cover most practical circumstances: (1) the training sample set is linearly inseparable; (2) all training sample points are distinct; and (3) the output layer is narrower than the other hidden layers.
Our result signiﬁcantly extends the existing study on the existence of spurious local minimum.
For example, Zhou & Liang (2018) prove that one-hidden-layer neural networks with two nodes in the hidden layer and two-piece linear (ReLU-like) activations have spurious local minima; Swirszcz et al.
(2016) prove that ReLU networks have spurious local minima under the squared loss when most of the neurons are not activated; Safran & Shamir (2018) present a computer-assisted proof that two-layer ReLU networks have spurious local minima; and a recent work (Yun et al., 2019b) have proven that neural networks with two-piece linear activations have inﬁnite spurious local minima, but the results only apply to the networks with one hidden layer and one-dimensional outputs.
The proposed theorem is proved in three stages: (1) we prove that neural networks with one hidden layer and two-piece linear activations have spurious local minima; (2) we extend the conditions to neural networks with arbitrary hidden layers and two-piece linear activations; and (3) we further extend the conditions to neural networks with arbitrary depth and arbitrary piecewise linear activations.
Since some parameters of the constructed spurious local minima are from continuous intervals, we have obtained inﬁnitely many spurious local minima.
At each stage, the proof follows a two-step strategy that: (a) constructs an inﬁnite series of local minima; and (b) constructs a point in the parameter space whose empirical risk is lower than the constructed local minimum in Step (a).
This strategy is inspired by Yun et al.
(2019b)
but we have made signiﬁcant and non-trivial development.
Second, we prove a “big picture” theorem for the loss surfaces of nonlinear neural networks from the following aspects:
• Smooth and multilinear partition.
The nonlinearities in the piecewise linear activations partition the loss surface of any nonlinear neural network into multiple smooth and multilinear open cells.
Speciﬁcally, every nonlinear point in the activation functions creates a group of the non-differentiable boundaries between the cells, while the linear parts of activations correspond to the smooth and multilinear interiors.
Soudry & Hoffer (2018) also highlight the partition of loss surface. We restate this property to make the “picture” self-contained.
• Local analogous convexity.
The interior of every cell has analogous convexity, which guarantees all local minima to be global optimal in the cell.
This property demonstrates that the local geometry within every cell is similar to the global geometry of linear networks, although technically, they are substantially different.
It applies to any one-hidden-layer neural network with two-piece linear activations for regression under convex loss.
We rigorously prove this property in two stages: (1) we prove that within every cell, the empirical risk
ˆ R is convex with respect to a variable ˆ W mapped from the weights W .
Therefore, the local minima with respect to the variable ˆ W are also the global minima in the cell; and then (2) we prove that the local optimality is maintained under the constructed mapping.
Speciﬁcally, the local minima of the empirical risk ˆ R with respect to the parameter W are also the local minima with respect to the variable ˆ W .
We thereby prove the local analogous convexity of nonlinear neural networks, combining the convexity and the correspondence of the minima.
This proof is technically novel and non-trivial, though the intuitions are natural.
Also, the name analogous convexity is from the proof.
• Local minima valley.
Some local minima (probably including global minima) are connected within a cell by a continuous path, on which all points have the same empirical risk.
Speciﬁcally, the constructed inﬁnitely many spurious local minima are exactly collected in a valley.
This result can help us understand the property of mode connectivity that the minima found by gradient-based methods are connected by a path in the parameter space with almost invariant empirical risk (Garipov et al., 2018;
Draxler et al., 2018; Kuditipudi et al., 2019).
• Linear collapse.
Linear neural networks are covered by our theories as a simpliﬁed case.
When all activations are linear, the partitioned loss surface collapses to one single cell, in which all local minima are globally optimal, as suggested by the existing works on linear networks
(Kawaguchi, 2016; Baldi Hornik, 1989; Lu & Kawaguchi, 2017; Freeman &
Bruna, 2017; Zhou & Liang, 2018; Laurent & von Brecht, 2018; Yun et al., 2018)
Notations.
If M is a matrix, M i,j denotes the ( i, j ) -th component of M .
If M is a vector, M i denotes the i -th component of M .
Deﬁne E ij as a matrix in which the ( i, j ) -th
component iswhile all other components are 0 .
Also, denote e i as a vector such that the i -th component is 1 while all others are 0 .
Additionally, we deﬁne 1 k ∈ R k × 1 is a vector whose components are all 1 , while those of 0 n × m ∈ R n × m (or brieﬂy, 0 ) are all 0 .
For the brevity, [ i : j ] denotes { i, · · · , j } .
R ELATED WORK
Some works suggest that linear neural networks have no spurious local minima.
Kawaguchi (2016) proves that linear neural networks with squared loss do not have any spurious local minimum under three assumptions about the data matrix X and the label matrix Y : (1) both matrices XX T and
XY T have full ranks; and (2) the input layer is wider than the output layer; and (3) the eigenvalues of matrix Y X (cid:62) (cid:0) XX T (cid:1) − 1 XY T are distinct with each other.
Zhou & Liang (2018) give an analytic formulation of the critical points for the loss function of deep linear networks, and thereby obtain a group of equivalence conditions for that critical point is a global minimum.
Lu & Kawaguchi (2017) prove the argument under one assumption that both matrices X and Y have full ranks, which is even more restrictive.
However, in practice, the activations of most neural networks are not linear.
The nonlinearities would make the loss surface extremely non-convex and even non-smooth and therefore far different from the linear case.
The loss surfaces of over-parameterized neural networks have some special properties.
Choromanska et al.
(2015) empirically suggest that: (1) most local minima of over-parameterized networks are equivalent; and (2) small-size networks have spurious local minima but the probability of ﬁnding one decreases rapidly with the network size.
Li et al.
(2018) prove that over-parameterized fullyconnected deep neural networks with continuous activation functions and convex, differentiable loss functions, have no bad strict local minimum.
Nguyen et al.
(2019) suggest that “sufﬁciently overparameterized” neural networks have no bad local valley under the cross-entropy loss.
Nguyen (2019) further suggests that the global minima of sufﬁciently over-parameterized neural networks are connected within a unique valley.
Many other works study the convergence, generalization, and other properties of stochastic gradient descent on the loss surfaces of over-parameterized networks (Chizat & Bach; Arora et al., 2018; Brutzkus et al., 2018; Du et al., 2019; Soltanolkotabi et al., 2018;
Allen-Zhu et al., 2019a;b; Oymak & Soltanolkotabi, 2019).
Many advances on the loss surfaces of neural networks are focused on other problems.
Zhou &
Feng (2017) and Mei et al.
(2018) prove that the empirical risk surface and expected risk surface are linked.
This correspondence highlights the value of investigating loss surfaces (empirical risk surfaces) to the study on generalization (the gap between empirical risks to expected risks).
Hanin & Rolnick (2019) demonstrate that the input space of neural networks with piecewise linear activations are partitioned by multiple regions, while our work focuses on the partition of the loss surface.
Sagun et al.
(2016; 2018) present empirical results that the eigenvalues of the Hessian of the loss surface are two-fold: (1) a bulk centered closed to zero; and (2) outliers away from the bulk.
Kawaguchi & Kaelbling (2019) prove that we can eliminate the spurious local minima by adding one unit per output unit for almost any neural network in practice.
Tian (2017); Andrychowicz et al.
(2016); Soltanolkotabi (2017); Zhong et al. (2017); Brutzkus & Globerson (2017); Tian (2017); Li & Yuan (2017); Zou et al.
(2018); Li & Liang (2018);
Du et al.
(2018a; 2019); Zhang et al.
(2019b);
Zhou et al.
; Wang et al.
(2019) study the optimization methods for neural networks.
Other relevant works include Xie et al.
; Sagun et al. ; Nguyen & Hein (2018); Du et al.
(2018b); Haeffele & Vidal (2017) ; Liang et al.
; Wu et al. ; Yun et al. (2019a) ; Zhang et al. (2019a); Kuditipudi et al.
; Garipov et al. ; Draxler et al. ; Kawaguchi & Kaelbling (2019).
N EURAL NETWORK HAS INFINITE SPURIOUS LOCAL MINIMA
This section investigates the existence of spurious local minima on the loss surfaces of neural networks.
We ﬁnd that almost all practical neural networks have inﬁnitely many spurious local minima.
This result stands for any neural network with arbitrary depth and arbitrary piecewise linear activations excluding linear functions under arbitrary continuously differentiable loss.
3. P RELIMINARIES
Consider a training sample set { ( X 1 , Y 1 ) , ( X 2 , Y 2 ) , . .
, ( X n , Y n ) } of size n .
Suppose the dimensions of feature X i and label Y i are d X and d Y , respectively.
By aggregating the training sample set, we obtain the feature matrix
X ∈ R
d X × n and label matrix Y ∈ R d Y × n .
Suppose a neural network has L layers.
Denote the weight matrix, bias, and activation in the jth layer respectively by W j ∈ R d j × d j − 1 , b j ∈ R d j , and h : R d j × n → R d j × n , where d j is the dimension of the output of the j -th layer.
Also, for the input matrix X , the output of the j -th layer is denoted as the Y ( j ) and the output of the j -th layer before the activation is denoted as the ˜ Y ( j ) ,
[Equation]
The output of the network is deﬁned as follows,
[Equation]
Also, we deﬁne Y (0) =
X , Y ( L ) =
ˆ Y , d 0 = d X , and d L = d Y .
In some situations, we use ˆ Y (cid:16) [ W i ] Li =1 , [ b i ] Li =1 (cid:17) to clarify the parameters, as well as ˜ Y ( j ) , Y ( j ) , etc.
This section discusses neural networks with piecewise linear activations.
A part of the proof uses two-piece linear activations h s − ,s + which are deﬁned as follows,
[Equation]
where | s + | (cid:54)
= | s − | and I {·} is the indicator function.
Remark.
Piecewise linear functions are dense in the space of continuous functions.
In other words, for any continuous function, we can always ﬁnd a piecewise linear function to estimate it with arbitrary small distance.
This section uses continuously differentiable loss to evaluate the performance of neural networks.
Continuous differentiability is deﬁned as follows.
Deﬁnition 1 (Continuously differentiable) .
We call a function f : R n → R continuously differentiable with respect to the variable x if: (1) the function f is differentiable with respect to x ; and (2) the derivatives f (cid:48)
( x ) of the function f with respect to the variable x are continuous with respect to the variable x .
3. M AIN RESULT
The theorem in this section relies on the following assumptions.
Assumption 1.
The training data cannot be ﬁt by a linear model.
Assumption 2.
All data points are distinct.
Assumption 3.
All hidden layers are wider than the output layer.
To our best knowledge, our assumptions are the least restrictive compared with the relevant works in the literature.
These assumptions are respectively justiﬁed as follows: (1) most real-world datasets are extremely complex and cannot be simply ﬁt using linear models; (2) it is easy to guarantee that the data points are distinct by employing data cleansing methods; and (3) for regression and many classiﬁcation tasks, the width of output layer is limited and narrower than the hidden layers.
Based on these three assumptions, we can prove the following theorem.
Theorem 1.
Neural networks with arbitrary depth and arbitrary piecewise linear activations (excluding linear functions) have inﬁnitely many spurious local minima under arbitrary continuously differentiable loss whose derivative can equal 0 only when the prediction and label are the same.
In practice, most loss functions are continuously differentiable and the derivative can equal 0 only when the prediction and label are the same, such as squared loss and cross-entropy loss (see Appendix A.1, Lemmas 1 and 2).
Squared loss is a standard loss for regression and is deﬁned as the Lnorm of the difference between the ground-truth label and the prediction as follows.
[Equation]
Meanwhile, cross-entropy loss is used as a standard loss in multiclass classiﬁcation, which is deﬁned as follows.
Here, we treat the softmax function as a part of the loss function.
[Equation]
Our result demonstrates that introducing nonlinearities into activations substantially reshapes the loss surface: they bring inﬁnitely many spurious local minima into the loss surface.
This result highlights the substantial difference from linear neural networks that all local minima of linear neural networks are equally good, and therefore, they are all global minima (Kawaguchi, 2016; Baldi &
Hornik, 1989; Lu & Kawaguchi, 2017; Freeman & Bruna, 2017; Zhou & Liang, 2018; Laurent & von Brecht, 2018; Yun et al., 2018).
Some works have noticed the existence of spurious local minima on the loss surfaces of nonlinear neural networks, which however has a limited applicable domain (Choromanska et al., 2015;
Swirszcz et al., 2016; Safran & Shamir, 2018; Yun et al., 2019b).
A notable work by Yun et al.
(2019b) proves that one-hidden-layer neural networks with two-piece linear (ReLU-like) activations for one-dimensional regression have inﬁnitely many spurious local minima under squared loss.
This work ﬁrst constructs a series of local minima and then prove they are spurious.
This idea inspires some of this work.
However, our work makes signiﬁcant and non-trivial development that extends the conditions to arbitrary depth, piecewise linear activations excluding linear functions, and continuously differentiable loss.
3. P ROOF SKELETON
This section presents the skeleton of the proof.
Theorem 1 is proved in three stages.
We ﬁrst prove a simpliﬁed version of Theorem 1 and then extend the conditions in the last two stages.
The proof is partially inspired by Yun et al.
(2019b) but the proof in this paper has made nontrivial development and the results are signiﬁcantly extended.
Yun et al.
(2019b)
and our paper both employ the following strategy: (a) construct a series of local minima based on a linear classiﬁer; and (b) construct a new point with smaller empirical risk and thereby we prove that the constructed local minima are spurious. However, due to the differences in the loss function and the output dimensions, the exact constructions of local minima are substantially different. Our extensions from Yun et al. (2019b) are three-fold: (1) From one hidden layer to arbitrary depth:
To prove that networks with an arbitrary depth have inﬁnite spurious local minima, we develop a novel strategy that employs transformation operations to force data ﬂow through the same linear parts of the activations, in order to construct the spurious local minima; (2) From squared loss to arbitrary differentiable loss: Yun et al.
(2019b) calculate the analytic formations of derivatives of the loss to construct the local minima and then prove they are spurious.
This technique cannot be transplanted to the case of arbitrary differentiable loss functions, because we cannot assume the analytic formation.
To prove that the loss surface under an arbitrary differentiable loss has an inﬁnite number of spurious local minima, we employ a new proof technique based on Taylor series and a new separation lemma; and (3) From one-dimensional output to arbitrary-dimensional output:
To prove the loss surface of a neural network with an arbitrary-dimensional output has an inﬁnite number of spurious local minima, we need to deal with the calculus of functions whose domain and codomain are a matrix space and a vector space, respectively.
By contrast, when the output dimension is one, the codomain is only the space of real numbers.
Therefore, the extension of output dimension signiﬁcantly mounts the difﬁculty of the whole proof.
Stage (1): Neural networks with one hidden layer and two-piece linear activations.
We ﬁrst prove that nonlinear neural networks with one hidden layer and two-piece linear activation functions (ReLU-like activations) have spurious local minima.
The proof in this stage further follows a two-step strategy:
(a) We ﬁrst construct local minima of the empirical risk ˆ R (see Appendix A.2, Lemma 3).
These local minimizers are constructed based on a linear neural network which has the same network size (dimension of weight matrices) and evaluated under the same loss.
The design of the hidden layer guarantees that the components of the output ˜ Y (1) in the hidden layer before the activation are all positive.
The activation is thus effectively reduced to a linear function.
Therefore, the local geometry around the local minima with respect to the weights W is similar to those of linear neural networks.
Further, the design of the output layer guarantees that its output
ˆ Y is the same as the linear neural network.
This construction helps to utilize the results of linear neural networks to solve the problems in nonlinear neural networks.
(b) We then prove that all the constructed local minima in Step (a) are spurious (see Appendix A.
Theorem 3)
Speciﬁcally, we assumed by Assumption {+
+} that the dataset cannot be
ﬁt
by a linear model. Therefore, the gradient ∇ ˆ Y ˆ R of the empirical risk ˆ R with respect to the prediction ˆ Y is not zero.
Suppose the i -th row of the gradient ∇ ˆ Y ˆ R is not zero.
Then, we use Taylor series and a preparation lemma (see Appendix A.5, Lemma 6) to construct another point in the parameter space that has smaller empirical risk.
Therefore, we prove that the constructed local minima are spurious.
Furthermore, the constructions involve some parameters that are randomly picked from a continuous interval.
Thus, we constructed inﬁnitely many spurious local minima.
Stage (2) - Neural networks with arbitrary hidden layers and two-piece linear activations.
We extend the condition in Stage (1) to any neural network with arbitrary depth and two-piece linear activations.
The proof in this stage follows the same two-step strategy but has different implementations:
(a) We ﬁrst construct a series of local minima of the empirical risk ˆ R (see Appendix A.3, Lemma 4).
The construction guarantees that every component of the output ˜ Y ( i ) in each layer before the activations is positive, which secure all the input examples ﬂow through the same part of the activations.
Thereby, the nonlinear activations are reduced to linear functions.
Also, our construction guarantees that the output ˆ Y of the network is the same as a linear network with the same weight matrix dimensions.
(b) We then prove that the constructed local minima are spurious (see Appendix A.3, Theorem 4).
The idea is to ﬁnd a point in the parameter space that has the same empirical risk ˆ R with the constructed point in Stage (1), Step (b).
Stage (3) - Neural networks with arbitrary hidden layer and piecewise linear activations.
We further extend the conditions in Stage (2) to any neural network with arbitrary depth and arbitrary piecewise linear activations.
We continue to adapt the two-step strategy in this stage: (a) We ﬁrst construct a local minimizer of the empirical risk ˆ R based on the results in Stages (1) and (2) (see Appendix A.4, Lemma 5).
This construction is based on Stage (2), Step (a).
The difference of the construction in this stage is that every linear part in activations can be a ﬁnite interval.
The constructed weight matrices use several uniform scaling and translation operations to the outputs of hidden layers in order to guarantee that all the input training sample points ﬂow through the same linear parts of the activations.
We thereby reduce the nonlinear activations to linear functions, effectively.
Also, our construction guarantees that the output ˆ Y of the neural network equals to that of the corresponding linear neural network.
b We then prove that the constructed local minima are spurious (see Appendix A.4).
We use the same strategy in Stage (2), Step (b).
Some adaptations are implemented for the new conditions.
A BIG PICTURE OF THE LOSS SURFACE
This section draws a big picture for the loss surfaces of neural networks.
We present a theorem that demonstrates four profound properties of the loss surface that collectively characterize how the nonlinearities in activations shape the loss surface.
4. P RELIMINARIES
The discussions in this section use the following concepts.
Deﬁnition 2 (Interior) .
For a subset A ⊂ H of a space H , a point h ∈ A is called an interior point of A , if there exists a positive real r > 0 , such that the ball B ( h, r ) with center h and radius r is in the subset A : B ( h, r ) ⊂
A .
The set of all the interior points of the set
A is called the interior of the set A .
Deﬁnition 3 (Boundary) .
For a subset A ⊂ H of a space H , a point h ∈ A is called a limit point, if for every r > 0 , the ball B ( h, r ) with center h and radius r contains point of A : B ( h, r )
∩ A (cid:54) = ∅ .
The closure ¯ A of the set A consists of the union of the set A and all its limit points.
The boundary ∂A is deﬁned as the set of points which are in the closure of set A but not in the interior of set A .
Deﬁnition 4 (Open set) .
A subset A ⊂ H of a space H is called open, if it does not contain any point in its boundary: A ∩ ∂A = ∅ .
Deﬁnition 5 (Multilinear) .
A function f : X 1 ×X 2 → Y is called multilinear if for arbitrary x 11 , x 21 ∈
X , x 12 , x 22 ∈ X 2 , and constants λ 1 , λ 2 , µ 1 , and µ 2 , we have f ( λ 1 x 11 + λ 2 x 21 , µ 1 x 12 + µ 2 x 22 ) =
λ
1 µ 1 f ( x 11 , x 12 )+ λ 1 µ 2 f ( x 11 , x 22 )+ λ 2 µ 1 f ( x 21 , x 12 )+ λ 2 µ 2 f ( x 21 , x 22 ) .
Remark.
The deﬁnition of “multilinear” implies that the domain of any multilinear function f is a connective and convex set, such as the smooth and multilinear cells below.
4. M AIN RESULTS
In this section, the loss surface is deﬁned under convex loss with respect to the prediction ˆ Y of the neural network.
Convex loss covers many popular loss functions in practice, such as the squared loss for the regression tasks and many others based on norms.
The triangle inequality of the norms secures the convexity of the corresponding loss functions.
The convexity of the squared loss is checked in the appendix (see Appendix B, Lemma 7).
We now present the theorem to express the loss surfaces of nonlinear neural networks.
This theorem gives four major properties of the loss surface that collectively draw a big picture for the loss surface.
The second property holds for neural networks with one hidden layer, while all the others apply to networks of arbitrary depth.
In addition, the third property needs an assumption (Assumption 3) on the network architecture, while the other three do not rely on any assumption.
Theorem 2.
The loss surfaces of neural networks with two-piece linear functions under any convex loss have the following properties:
• Smooth and multilinear partition.
The loss surface is partitioned into multiple smooth and multilinear open cells, while the boundaries are nondifferentiable.
• Local analogous convexity.
For one-hidden-layer neural networks for regression, within every cell, all local minima are equally good, and also, they are all global minima in the cell.
• Local minima valley.
Under Assumption 3, some local minima (probably including global minima) are connected within a cell by a continuous path, on which all points have the same empirical risk.
• Linear collapse.
The partitioned loss surface collapses to one single smooth and multilinear cell, when all activations are linear.
4. D ISCUSSIONS AND PROOF TECHNIQUES
This theorem characterizes how the nonlinearities in activations shape the loss surfaces of neural networks.
This section discusses the results and the structure of the proofs.
A detailed proof is omitted here and given in Appendix B.
Smooth and multilinear partition.
All the activations in the architecture of linear neural networks (see eqs.
(2) and (3)) are linear functions, which can also be expressed by matrices.
Therefore, the loss surface of every linear neural network is smooth and multilinear with respect to all the parameters.
We thus can expect a very good property that all local minima are global.
The nonlinearities in the activations substantially change the loss surface.
Intuitively, the nonlinearities introduce the non-linear and non-differentiable boundaries to the loss surface, and thereby partition the surface into multiple cells.
If the input data ﬂows through the network and meets all activations at the linear parts, the corresponding loss apparently falls in a multilinear and smooth region on the loss surface.
Meanwhile, each nonlinear point in one activation corresponds to a group of boundaries.
The different combinations of the nonlinear points in the activations through the neural network further determine the boundary between a speciﬁc cell pair.
Soudry & Hoffer (2018) highlight the partition of the loss surface.
We restate it here to make the picture drawn by Theorem 2 self-contained.
Zhou & Liang (2018) highlight the partition of loss surface but the corresponding geometrical properties remain untraced.
A similar but also markedly different notions recently proposed by Hanin & Rolnick (2019) demonstrate that the input space is partitioned by multiple regions, while our work focuses on the partition in the parameter space.
Local analogous convexity.
In convex optimization, convexity guarantees that all the local minima are global minima.
This theorem proves that the interior of every cell has the property of analogous convexity that the local minima wherein are equally good, and also, they are all global minima in the cell.
This result is not surprising provided the excellent training performance of deep learning algorithms.
However, the proof is technically non-trivial.
Soudry & Hoffer (2018) proved that the local minima in a cell are the same. However, there would be some point near the boundary has a smaller empirical risk and is not locally minimal. Unfortunately, the proof by Soudry & Hoffer (2018) cannot exclude this possibility. By contrast, our proof completely solves this problem.
Furthermore, our proof holds for any convex loss, including squared loss and cross-entropy loss, but Soudry & Hoffer (2018) only stands for squared loss.
It is challenging to prove Property 2, because the proof techniques for the case of linear networks cannot be transplanted here.
Technically, linear networks can be expressed by the product of a sequence of weight matrices, which guarantees good geometrical properties.
Speciﬁcally, the effect of every linear activation function is just equivalently multiplying a real constant to the output.
However, the loss surface within a cell of a nonlinear neural network does not have this property.
Below is the skeleton of our proof.
We ﬁrst prove that the empirical risk
ˆ
R is a convex function within every cell with respect to a variable ˆ W which is calculated from the weights W .
Therefore, all local minima of the empirical risk ˆ R with respect to ˆ W are also globally optimal in the cell.
Each cell corresponds to a speciﬁc series of linear parts of the activations.
Therefore, in any ﬁxed cell, the activation h s − ,s + can be expressed by the slopes of the corresponding linear parts as the following equations,
[Equation]
where A · ,i is the i -th column of matrix
[Equation]
Matrix A is constituted by collecting the slopes of the activation h at every point ( W 1 ) i, · x j .
Apparently, different elements of the matrix A can be multiplied either one of { s − , s + } .
Therefore, we cannot use a single constant to express the effect of this activation, and thus, even within the cell, a nonlinear network cannot be expressed as the product of a sequence of weight matrices.
This difference ensures that the proofs of deep linear neural networks cannot be transplanted here.
Then, we prove that (see p. 35)
[Equation]
Applying eq.
(8) to eq.
(7), the empirical risk ˆ R equals to a formulation similar to the linear neural networks,
[Equation]
Afterwards, deﬁne ˆ W 1 = diag ( W 2 ) W 1 and then straighten the matrix ˆ W 1 to a vector ˆ W ,
[Equation]
Also deﬁne
[Equation]
We can prove the following equations (see p. 36),
[Equation]
Applying eq.
(77), the empirical risk is transferred to a convex function as follows,
[Equation]
We then prove that the local optimality of the empirical risk ˆ R is maintained when the weights W are mapped to the variable ˆ W .
Speciﬁcally, the local minima of the empirical risk ˆ R with respect to the weight W are also the local minima with respect to the variable ˆ W .
The maintenance of optimality is not surprising but the proof is technically non-trivial (see a detailed proof in pp.
Local minima.
Some local minima are connected within a cell by a continuous path, on which all points have the same empirical risk.
Speciﬁcally, the constructed inﬁnitely many spurious local minima constructed in Section 3 are exactly collected in a valley.
This result partially explains the property of mode connectivity that the minima found by gradient-based methods are connected by a path in the parameter space with almost constant empirical risk, which is proposed in two empirical works (Garipov et al., 2018; Draxler et al., 2018).
A recent theoretical work (Kuditipudi et al., 2019) proves that for dropout stability and noise stability guarantee the mode connectivity.
Linear collapse.
Our theories also cover the case of linear neural networks.
Linear neural networks do not have any nonlinearity in their activations.
Correspondingly, the loss surface does not have any non-differentiable boundaries.
In our theories, when there is no nonlinearity in the activations, the partitioned loss surface collapses to a single smooth, multilinear, and analogously convex cell.
All local minima wherein are equally good, and also, they are all global minima as follows.
This result unites the existing results on linear neural networks (Kawaguchi, 2016; Baldi & Hornik, 1989; Lu & Kawaguchi, 2017; Freeman & Bruna, 2017; Zhou & Liang, 2018; Laurent & von Brecht, 2018;
Yun et al., 2018).
C ONCLUSION
This paper reports that the nonlinearities in activations substantially shape the loss surfaces of neural networks.
First, we prove that neural networks have inﬁnitely many spurious local minima which are in contrast to the circumstance of linear neural networks.
This result stands for any neural network with arbitrary hidden layers and arbitrary piecewise linear activations (excluding linear functions) under many popular loss functions in practice (e.g., squared loss and cross-entropy loss).
This result signiﬁcantly extends the conditions of the relevant results and has the least restrictive assumptions that cover most practical circumstances: (1) the training data is not linearly separable; (2) the training sample points are distinct; and (3) all hidden layers are wider than the output layer.
Second, we prove a theorem that collects several properties of the loss surfaces of nonlinear neural networks: (1) the loss surface is partitioned by multiple smooth and multilinear open cells; (2) local minima in any cell are equally good, and also, they are all global minima in the cell; (3) some local minima are connected within a cell by a continuous path, on which all points have the same empirical risk; and (4) the loss surface collapses to one single cell when all activations are linear functions, which explains the results of linear neural networks.
The second property is rigorously proved for any one-hidden-layer nonlinear neural networks with two-piece linear (ReLU-like) activations for regression tasks under convex loss without any assumption and the others apply to neural networks of arbitrary depth
A P ROOF OF T HEOREM
This appendix gives a detailed proof of Theorem 1 omitted from the main text.
It follows the skeleton presented in Section 3.3.
A. S QUARED LOSS AND CROSS - ENTROPY LOSS
We ﬁrst check whether squared loss and cross-entropy loss are covered by the requirements of
Theorem 1.
Lemma 1.
The squared loss (deﬁned by eq.
(5)) is continuously differentiable with respect to the prediction of the model, whose gradient of loss equal to zero when the prediction and the label are different.
Proof.
Apparently, the squared loss is differentiable with respect to ˆ Y .
Speciﬁcally, the gradient with respect to ˆ Y is as follows,
[Equation]
which is continuous with respect to ˆ Y .
Also, when the prediction ˆ Y does not equals to the label Y , we have
[Equation]
The proof is completed.
Lemma 2.
The cross-entropy loss eq.
(6) is continuously differentiable with respect to the prediction of the model, whose gradient of loss equal to zero when the prediction and the label are different.
Also, we assume that the ground-truth label is a one-hot vector.
Proof.
For any i ∈ [1 : n ] , the cross-entropy loss is differentiable with respect to ˆ Y i .
The j -th component of the gradient with respect to the prediction ˆ
Y i is as follows,
[Equation]
which is continuous with respect to ˆ Y i .
So, the cross-entropy loss is continuously differentiable with respect to ˆ Y i .
Additionally, if the gradient (eq. (10)) is zero, we have the following equations,
[Equation]
Rewrite it into the matrix form, we have
[Equation]
Since d Y (cid:80) k =1 Y k,i = 1 , we can easily check the rank of the left matrix is d Y − 1 .
So the dimension of the solution space is one.
Meanwhile, we have  
[Equation]
−
[Equation]
−
[Equation] [Equation]
i
[Equation]
− Y d
[Equation]
,i d
[Equation]
(cid:80) i =1 Y k,i − Y d Y ,i 
  
Y 1 ,i Y 2 ,i ...
Y d Y ,i   = 0 .
Therefore, 0 (cid:54) =
e ˆ Y k,i = λY
k,i , for some λ ∈ R , which contradicts to the assumption that some of the components of Y is 0 ( Y i, · is a one-hot vector).
The proof is completed.
A. S TAGE (1)
In Stage (1), we prove that deep neural networks with one hidden layer, two-piece linear activation h s − ,s + , and multi-dimensional outputs have inﬁnite spurious local minima.
This stage is organized as follows: (a) we construct a local minimizer by Lemma 3; and (b) we prove that the local minimizer is spurious in Theorem 3 by constructing a set of parameters with smaller empirical risk.
Without loss of generality, we assume that s + (cid:54) = 0 .
Otherwise, suppose that s + = 0 .
From the deﬁnition of ReLU-like activation (eq. (4)), we have s − (cid:54) = 0 .
Since
[Equation]
, the output of the neural network with parameters (cid:110)
[ W
[Equation]
] Li =1 , [ b i ] Li =1 (cid:111) and activation h s − ,s + equals to that of the neural network with parameters (cid:110)
[ W (cid:48)
i ]
Li =1 , [ b (cid:48)
i ]
Li =1 (cid:111) and activation h − s + , − s − where
W (cid:48)
i = − W i , b (cid:48)
i = − b i , i = 1 , 2 , · · · , L − 1 and W (cid:48)
L = W L , b (cid:48)
L = b L .
Since (cid:110)
[ W i ]
Li
=1 , [ b i ] Li =1 (cid:111)
→ (cid:110)
[ W (cid:48)
i ]
Li =1 , [ b (cid:48)
i ]
Li =1 (cid:111) is an one-to-one map, it is equivalent to consider either the two networks, with h − s + , − s − ( x ) has non-zero slope when x > 0 .
Step (a).
Construct local minima of the loss surface.
Lemma 3.
Suppose that ˜ W is a local minimizer of
[Equation]
Under Assumption 3, any one-hidden-layer neural network has a local minimum at
[Equation] [Equation]
and
[Equation]
(13) where ˆ W 1 and ˆ b 1 are respectively the weight matrix and the bias of the ﬁrst layer, ˆ W 2 and ˆ b 2 are respectively the weight matrix and the bias of the second layer, and η is a negative constant with absolute value sufﬁciently large such that
[Equation]
where > is element-wise.
Also, the loss in this lemma is continuously differentiable loss whose gradient does not equals towhen the prediction is not the same as the ground-truth label.
Proof.
We show that the empirical risk is higher in the neiborhood of (cid:26)(cid:104)
[Equation]
, in order to prove that (cid:26)(cid:104)
ˆ W i (cid:105)i =1 , (cid:104)
ˆ
b i (cid:105)i = (cid:27) is a local minimizer.
The output of the ﬁrst layer before the activation is
[Equation]
Because η is a negative constant with absolute value sufﬁciently large such that eq.
(31)) holds, the output above is positive (element-wise), the output of the neural network with parameters { ˆ W 1 , ˆ W 2 , ˆ b 1 , ˆ b 2 } is
[Equation]
where ˜ X is deﬁned as
[Equation]
Therefore, the empirical risk ˆ R in terms of parameters { ˆ W 1 , ˆ W 2 , ˆ b 1 , ˆ b 2 } is
[Equation]
Then, we introduce a sufﬁciently small disturbance (cid:110)
[ δ Wi ] 2 i =1 , [ δ bi ] 2 i =1 (cid:111) into the parameters (cid:26)(cid:104)
ˆ W
i
[Equation]
i =1 , (cid:104)
ˆ b i
[Equation]
i = (cid:27) .
When the disturbance is sufﬁciently small, all components of the output of the ﬁrst layer remain positive.
Therefore, the output after the disturbance is
[Equation]
=
[Equation] [Equation]
) =
[Equation]
where eq.
( ∗ ) is because all components of (cid:16) ˆ W 1 + δ W 1 (cid:17) X
+ ( b (cid:48)
+ δ b 1 ) 1 Tn are positive
[Equation]
and δ is deﬁned as the following matrix
[Equation]
Therefore
[Equation]
the empirical risk ˆ R with respect to
[Equation]
δ approaches zero when the disturbances { δ W 1 , δ W 2 , δ b 1 , δ b 2 } approach zero (element-wise).
Since ˆ
W is the local minimizer of f ( W )
[Equation]
we have
[Equation]
Because the disturbances { δ W 1 , δ W 2 , δ b 1 , δ b 2 } are arbitrary
[Equation]
eq.
(16) demonstrates that
[Equation]
i = (cid:27) is a local minimizer.
The proof is completed.
Step (b).
Prove the constructed local minima are spurious.
Theorem 3.
Under the same conditions of Lemma 3 and Assumptions 1 and 2, the constructed spurious local minima in Lemma 3 are spurious.
Proof.
The minimizer ˜ W is the solution of the following equation
[Equation]
Speciﬁcally, we have
[Equation]
Applying the deﬁnition of f ( W ) (eq. (11)),
[Equation] [Equation]
where ˆ Y i =
˜ W (cid:20) x i 1 (cid:21) , ∇ ˆ Y i l (cid:18) Y i , ˜ W (cid:20) x i 1 (cid:21)(cid:19) ∈ R 1 × d Y .
Since k , j are arbitrary in { 1 , · · · , d Y } and { 1 , · · · , d X } , respectively, we have
[Equation]
(17) where
V =
(cid:20)(cid:18) ∇ ˆ Y 1 l (cid:18) Y 1 , ˜ W
(cid:20)
[Equation]
[Table]
[Equation]
We then deﬁne ˜ Y =
˜ W ˜ X .
Applying Assumption 1, we have
[Equation]
Thus, there exists some k -th row of ˜ Y − Y that does not equal to 0 .
We can rearrange the rows of ˜ W and Y simultaneously, while ˜ W is maintained as the local minimizer of f ( W ) and f ( ˜ W ) invariant 1 .
Without loss of generality, we assume k =
( k is the index of the row).
Set u
= V 1 , · and v i =
˜ Y 1 ,i in Lemma 6.
There exists a non-empty separation I =
[1 : l (cid:48) ] and J =
[ l (cid:48) + 1 : n ] of S
= { 1 , 2 , · · · , n } and a vector β ∈ R d X , such that (1.1) for any positive constant α small enough, and i ∈
I , j ∈ J , ˜ Y 1 ,i − αβ T x i <
˜
Y 1 ,j − αβ T x j ; (1.2) (cid:80) i ∈
I V 1 ,i (cid:54)
= 0 .
Deﬁne
[Equation]
Applying (1.1), for any i ∈
I
[Equation]
while for any j ∈ J ,
[Equation]
> 0 .
Deﬁne γ ∈ R which satisﬁes
[Equation]
where s t +1 is deﬁned in Lemma
[Equation]
We argue that
[Equation]
When l (cid:48)
=
s t
+1 , eq.
(71) stands.
Also,
[Equation]
Therefore, we get eq.
(19) when α is small enough.
When l (cid:48) < s t
+1 , eq.
(70) stands.
Therefore,
[Equation]
, which apparently leads to eq.
Therefore, for any i ∈ I , we have that
[Equation]
<
[Equation]
while for any j ∈ J ,
[Equation]
Furthermore, deﬁne η i ( 2 ≤ i ≤ d Y ) as negative reals with absolute value sufﬁciently large, such that for any i ∈ [2 : d Y ] and any j ∈ [1 : n ] ,
[Equation]
Now we construct a point in the parameter space whose empirical risk is smaller than the proposed local minimum in Lemma 3 as follows
[Equation] [Equation]

[Equation]
· · · 0 · · ·   1 s
[Equation]
1 s + · · · ...
[Equation]
· 
[Equation]
(22) and
[Equation]
(23) where ˜ W i and ˜
b i are the weight matrix and the bias of the i -th layer, respectively.
After some calculations, the network output of the ﬁrst layer before the activation in terms of (cid:26)(cid:104)
[Equation]
Therefore, the output of the whole neural network is
[Equation]

[Equation]
Speciﬁcally, if j ≤ l (cid:48) ,
[Equation]
Y 1 ,j − αβ T x
j − η 1 + γ < 0 , (cid:18) ˜ Y (1) (cid:18)(cid:104)
˜ W i (cid:105)i =1 , (cid:104) ˜
b i (cid:105)i = (cid:19)(cid:19) 2 ,j = − ˜ W 1 , · (cid:20) x j (cid:21) + αβ T x j + η 1 + γ = − ˜ Y 1 ,j + αβ
T x j + η 1 + γ > 0 .
Therefore, (1 , j ) -th component of ˆ Y (cid:18)(cid:104)
˜ W
i (cid:105) 2 i =1 , (cid:104) ˜
b i (cid:105) 2 i = (cid:19) is (cid:18)
ˆ Y (cid:18)(cid:104)
˜ W i (cid:105)i =1 , (cid:104) ˜
b i (cid:105)i = (cid:19)(cid:19) 1 ,j = (cid:16) 1 s + + s − , − 1 s + + s − , 0 , · · · , 0 (cid:17) h s − ,s +     ˜ W 1 , · X − αβ T X − η 1 1 Tn + γ 1 Tn − ˜ W 1 , · X + αβ T X + η 1 1 Tn + γ 1 Tn ˜
W 2 , · X − η 2 1 Tn .
˜ W d Y , · X − η d Y 1 Tn 0 d 1 − d Y − 1 1 Tn     · ,j + η= s
+
+ s − h s − ,s + ( ˜ Y 1 ,j − αβ T x
j − η 1 + γ ) − s
+
+ s − h s − ,s + ( − ˜ Y 1 ,j + αβ T x j + η 1 + γ ) + η(24)
=
s − s
+ + s −
[Equation]
Similarly, when j > l (cid:48) , the ( 1 , j )-th component is
[Equation]
[Equation]
and
[Equation]
Thus
[Equation]
the empirical risk of the neural network with parameters (cid:26)(cid:104)
[Equation]
=n [Equation] n
[Equation]
Applying eqs.
[Equation] [Equation]
and (29)
[Equation]
we have
[Equation]
where eq.
( ∗ ) is because
[Equation]
+ ˜ b
[Equation]
W (cid:20) x i 1 (cid:21)(cid:19)
[Equation]
=   [Equation] >
[Equation]
(cid:48) 0 , j ≥.
Furthermore, note that α = O ( γ ) (from the deﬁnition of γ ).
We have n (cid:88) i = o
(cid:18)(cid:13)(cid:13) (cid:13) (cid:13)
˜
W 2 (cid:16) ˜
[Equation]
(cid:17)
[Equation]
(cid:20)
[Equation]
1 (cid:21)(cid:13)(cid:13) (cid:13) (cid:13) (cid:19) =
n (cid:88)
i =
o   (cid:118)(cid:117)(cid:117)(cid:116)
n (cid:88)
j = (cid:18)
˜ W 2 (cid:16) ˜
W 1 x i + ˜
b 1 (cid:17) + ˜ b 2 − ˜ W (cid:20) x i 1 (cid:21)(cid:19)j 
 =
o ( γ ) .
Let α be sufﬁciently small while sgn ( γ ) =
− sgn
[Equation]
We have
[Equation]
where inequality ( ∗∗ ) comes from (1.2) (see p. 16).
From Lemma 3, there exists a local minimizer (cid:26)(cid:104)
[Equation]
(cid:27) with empirical risk that equals to f ( ˜ W ) .
Meanwhile, we just construct a point in the parameter space with empirical risk smaller than f ( ˜ W ) .
Therefore, (cid:26)(cid:104)
ˆ W i (cid:105)i =1 , (cid:104)
ˆ
b i (cid:105)i = (cid:27) is a spurious local minimum.
The proof is completed.
A. S TAGE (2)
Stage (2) proves that neural networks with arbitrary hidden layers and two-piece linear activation h s − ,s + have spurious local minima.
Here, we still assume s + (cid:54) = 0 .
We have justiﬁed this assumption in Stage (1).
This stage is organized similarly with Stage (1): (a) Lemma 4 constructs a local minimum; and (b)
Theorem 4 proves the minimum is spurious.
Step (a).
Construct local minima of the loss surface.
Lemma 4.
Suppose that all the conditions of Lemma 3 hold, while the neural network has L −hidden layers.
Then, this network has a local minimum at
[Equation]
and ˆ
[Equation]
where ˆ W (cid:48)
i and ˆ b (cid:48)
i are the weight matrix and the bias of the i -th layer, respectively, and η is a negative constant with absolute value sufﬁciently large such that
[Equation]
where > is element-wise.
Proof.
Recall the discussion in Lemma 3 that all components of ˆ W 1 X + ˆ b 1 1 Tn are positive.
Specifically,
[Equation]
where ˜ Y is deﬁned in Lemma 3.
Similar to the discussions in Lemma 3, when the parameters equal to (cid:26)(cid:104)
[Equation]
, the output of the ﬁrst layer before the activation function is
[Equation]
and
[Equation]
Here > is deﬁned element-wise.
After the activation function
[Equation]
the output of the ﬁrst layer is
[Equation]
We prove by induction that for all i ∈ [1 : L − 1] that ˜ Y ( i ) > 0 , element-wise ,
[Equation]
[Equation] [Equation]
Suppose that for 1 ≤ k ≤ L − 2 , ˜ Y ( k ) is positive (element-wise) and
[Equation]
Then the output of the ( k + 1) -th layer before the activation is
[Equation]
d Y
[Equation]
Applying eqs.
(32) and (33), we have
[Equation]
where > is deﬁned element-wise.
Therefore,
[Equation]
We thereby prove eqs.
(34) and (35).
Therefore, Y ( L ) can be calculated as
[Equation]
Then, we show the empirical risk is higher around (cid:26)(cid:104)
ˆ
[Equation]
in order to prove that
[Equation]
i =
[Equation] [Equation]
i = (cid:27) is a local minimizer.
Let (cid:26)(cid:104)
[Equation]
Wi
[Equation] [Equation]
(cid:105) L i = (cid:27) be point in the parameter space which is close enough to the point (cid:26)(cid:104)
ˆ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˆ b (cid:48) i (cid:105)
L i = (cid:27) .
Since the disturbances δ (cid:48)
Wi and δ (cid:48) bi are both close to 0 (element-wise), all components of ˜ Y ( i ) (cid:18)(cid:104) ˆ W (cid:48)
i + δ (cid:48)
Wi (cid:105) L i =1 , (cid:104) ˆ b (cid:48)
i + δ (cid:48) bi (cid:105)
L i =1 (cid:19) remains positive.
Therefore, the output of the neural network in terms of parameters (cid:26)(cid:104)
ˆ W (cid:48)
i + δ (cid:48)
Wi (cid:105) L i =1 ,
[Equation]
ˆ b (cid:48)
[Equation]
+ δ
[Equation]
bi
[Equation]
L
[Equation]
= (cid:27) is ˆ Y (cid:18)(cid:104)
[Equation]
where M 1 and M 2 can be obtained from (cid:26)(cid:104)
ˆ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˆ b (cid:48) i (cid:105)
L i = (cid:27) and (cid:110)
[ δ (cid:48)
Wi ]
Li =1 , [ δ (cid:48) bi ] Li =1 (cid:111) through several multiplication and summation operations 2 .
Rewrite the output as
[Equation]
1 Tn
[Equation]
Therefore, the empirical risk ˆ R before and after the disturbance can be expressed as f ( ˜ W ) and f ([ M
M 2 ]) , respectively.
When the disturbances (cid:110)
[ δ (cid:48)
Wi ]
Li =1 , [ δ (cid:48) bi ] Li =1 (cid:111) approach
(element-wise), [ M
M 2 ] approaches ˜ W .
Therefore, when (cid:110)
[ δ (cid:48)
Wi ]
Li =1 , [ δ (cid:48) bi ] Li =1 (cid:111) are all small enough, we have
[Equation]
Since (cid:26)(cid:104)
ˆ W (cid:48)
i + δ (cid:48)
[Equation]
(cid:105)
[Equation]
i =
[Equation]
, (cid:104)
ˆ
[Equation]
(cid:48)
[Equation]
= (cid:27) are arbitrary within a sufﬁciently small neighbour of (cid:26) (cid:104) ˆ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˆ b (cid:48) i (cid:105)
L i = (cid:27) , eq.
(37) yields that
[Equation]
ˆ b (cid:48)
i [Equation] i = (cid:27) is a local minimizer.
Step (b).
Prove the constructed local minima are spurious.
Theorem 4.
Under the same conditions of Lemma 4 and Assumptions 1 and 2, the constructed spurious local minima in Lemma 4 are spurious.
Proof.
We ﬁrst construct the weight matrix and bias of the i -th layer as follows,
[Equation]
and
[Equation]
where ˜ W 1 , ˜ W 2 , ˜ b 1 and ˜ b 2 are deﬁned by eqs.
(20), (21), (22), and (23), respectively, and λ is a sufﬁciently large positive real such that
[Equation]
where > is deﬁned element-wise.
We argue that (cid:26)(cid:104) ˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =
(cid:27) corresponds to a smaller empirical risk than f ( ˜ W ) which is deﬁned in Lemma 3.
First, Theorem 3 has proved that the point (cid:110) ˜ W 1 , ˜ W 2 , ˜ b 1 , ˜ b 2 (cid:111) corresponds to a smaller empirical risk than f ( ˜ W ) .
We prove by induction that for any i ∈ { 3 , 4 , ..., L − 1 } ,
[Equation]
Apparently the output of the ﬁrst layer before the activation is
[Equation]
Therefore, the output of the ﬁrst layer after the activation is
[Equation]
(cid:104) ˜ b (cid:48)
i (cid:105) L i =
(cid:19)
=
h s − ,s + (cid:18)
˜ Y (1) (cid:18)(cid:104) ˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i = (cid:19)(cid:19) =
h s − ,s + (cid:18)
˜ Y (1) (cid:18)(cid:104) ˜
W
i (cid:105) L i =1 , (cid:104) ˜
b i (cid:105) L i =
(cid:19)(cid:19) =
Y (1) (cid:18)(cid:104) ˜
W
i (cid:105)i =1 , (cid:104) ˜
b i (cid:105)i = (cid:19) .
Thus, the output of the second layer before the activation is
[Equation]
Applying the deﬁnition of λ (eq. (38)),
[Equation]
Therefore, the output of the second layer after the activation is
[Equation]
−
[Equation]

[Equation]
Meanwhile, the output of the third layer before the activation is ˜ Y (3) (cid:18)(cid:104) ˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =1 (cid:19) can be calculated based on Y (2) (cid:18)(cid:104) ˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =
(cid:19) :
˜ Y (3) (cid:18)(cid:104) ˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =
(cid:19) =
˜ W (cid:48) 3 Y (2) (cid:18)(cid:104) ˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =
(cid:19)
+ ˜ b (cid:48) 3 1 Tn
[Equation]
Applying eq.
[Equation]
Therefore
[Equation]
the output of the third layer after the activation is
[Equation]
Suppose eqs.
(39) and (40) hold for k ( 3 ≤ k ≤ L − 2 ),
when k + 1 ,
[Equation]
Applying eq.
[Equation]
Therefore, the output of the ( k + 1 )-th layer after the activation is
[Equation]
˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =
(cid:19)
=
h s − ,s + (cid:18)
˜ Y ( k +1) (cid:18)(cid:104) ˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i = (cid:19)(cid:19) =
s + ˜ Y
( k +1) (cid:18)(cid:104) ˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =
(cid:19)
=
s +  
ˆ Y (cid:18)(cid:104)
˜ W i (cid:105)i =1 , (cid:104) ˜
b i (cid:105)i = (cid:19) + λ 1 d Y 1 Tn 0 ( d k +1 − d Y ) × n   .
Therefore, eqs.
(39) and (40) hold for any i ∈ { 3 , 4 , ..., L − 1 } .
Finally, the output of the network is
[Equation]
Applying Theorem 3, we have
[Equation]
The proof is completed.
A. S TAGE (3)
Finally, we prove Theorem 1.
This stage also follows the two-step strategy.
Step (a).
Construct local minima of the loss surface.
Lemma 5.
Suppose t is a non-differentiable point for the piece-wise linear activation function h and σ is a constant such that the activation h is differentiable in the intervals ( t − σ, t ) and ( t, t + σ ) .
Assume that M is a sufﬁciently large positive real such that
[Equation]
Let α i be any positive real such that
[Equation]
Then, under Assumption 3, any neural network with piecewise linear activations and L − 1 hidden layers has local minima at
[Equation] [Equation]
and
[Equation]
48) where (cid:26)(cid:104)
ˆ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˆ b (cid:48) i (cid:105)
L i = (cid:27) is the local minimizer constructed in Lemma 4.
Also, the loss is continuously differentiable, whose derivative with respect to the prediction ˆ Y i may equal to 0 only when the prediction ˆ Y i and label Y i are the same.
Proof.
Deﬁne s − =
lim θ → 0 − h (cid:48)
( θ ) and s + =
lim θ → 0 + h (cid:48)
( θ ) .
We then prove by induction that for all i ∈ [1 : L − 1] , all components of the i -th layer output before the activation ˜ Y ( i )
[Equation]
i
[Equation] [Equation]
(cid:48)(cid:48) i (cid:105)
L i =
(cid:19) are in interval ( t, t + σ ), and
Y ( i ) (cid:18)(cid:104)
[Equation]
The ﬁrst layer output before the activation is,
[Equation]
We proved in Lemma 4 that ˆ W (cid:48) 1 X + ˆ b (cid:48)
1 1 Tn is positive (element-wise).
Since the Frobenius norm of a matrix is no smaller than any component’s absolute value
[Equation]
applying eq.
[Equation]
we have that for all i ∈ [1 , d 1 ] and j ∈ [1 : n ] ,
[Equation]
Therefore,
[Equation]
where eq.
( ∗ ) is because for any x ∈ ( t − σ, t + σ ) , [Equation]
[Equation]
Suppose the above argument holds for k ( 1 ≤ k ≤ L − 2 )
[Equation]
Then
[Equation]
Lemma 4 has proved that all components of ˜ Y ( k +1) (cid:18)(cid:104)
ˆ W
[Equation]
i
[Equation]
i =
[Equation]
ˆ b
[Equation]
i
[Equation]
L i =1 (cid:19) are contained in ˜ Y (1) (cid:18)(cid:104) ˆ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˆ b (cid:48) i (cid:105)
L i = (cid:19) .
Combining
[Equation]
we have
[Equation]
Here < are all element-wise
[Equation]
and inequality ( ∗ ) comes from the property of α i (eq.
[Equation]
Furthermore
[Equation]
the ( k + 1) -th layer output after the activation is
[Equation]
where eq.
( ∗ ) is because of eq.
The above argument is proved for any index k ∈ { 1 , . . .
, L − 1 } .
Therefore, the output of the network is
[Equation]
Therefore,
[Equation]
We then introduce some small disturbances (cid:110)
[Equation]
into
[Equation]
in order to check the local optimality.
Since all comonents of Y ( i ) are in interval ( t, t + σ ) , the activations in every hidden layers is realized at linear parts.
Therefore, the output of network is
[Equation]
(cid:20)
[Equation]
(cid:21)
[Equation]
Similar to Lemma 4, [ M
M 2 ] approaches ˜ W as disturbances (cid:110)
[ δ Wi ]
Li =1 , [ δ bi ] Li =1 (cid:111) approach(element-wise).
Combining that ˜ W is a local minimizer of f ( W ) , we have ˆ R (cid:18)(cid:104) ˆ W (cid:48)(cid:48) i + δ (cid:48)(cid:48)
Wi (cid:105) L
i =1 , (cid:104)
ˆ b (cid:48)(cid:48)
i + δ (cid:48)(cid:48)
[Equation]
The proof is completed.
Step (b).
Prove the constructed local minima are spurious.
Proof of Theorem 1.
Without loss of generality, we assume that all activations are the same.
Let t be a non-differentiable point of the piece-wise linear activation function h with
[Equation]
Let σ be a constant such that h is linear in interval ( t − σ, t ) and interval ( t, t + σ ) .
Then construct that
[Equation]
and ˜ W
[Equation]
where (cid:26)(cid:104) ˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =
(cid:27) are constructed in Theorem 4, M is a large enough positive real such that
[Equation]
and ˜ M a large enough positive real such that
[Equation]
F < σ.
Then, we prove by induction
that for any i ∈ [ :
L − 1] , all components of ˜ Y ( i ) (cid:18)(cid:104) ˜ W (cid:48)(cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)(cid:48)
i (cid:105) L i =
(cid:19) are in interval ( t, t + δ ) , and
Y ( i ) (cid:18)(cid:104) ˜
W (cid:48)(cid:48) i (cid:105) L
i =1 , (cid:104) ˜
b (cid:48)(cid:48)
i (cid:105) L i =
(cid:19)
[Equation] [Equation]
[Equation] [Equation]
[Equation]
(cid:48)
[Equation]
(cid:105) L
[Equation]
, (cid:104)
[Equation]
(cid:48)
i (cid:105) L i =
(cid:19) .
First, ˜ Y (1) (cid:18)(cid:104) ˜ W (cid:48)(cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)(cid:48)
i (cid:105) L i =
(cid:19) =
[Equation]
W (cid:48)(cid:48)
[Equation]
(cid:48)(cid:48)
[Equation]
= 1 M (
[Equation]
W (cid:48) 1 X + ˜ b (cid:48) 1 1 Tn )
+
[Equation]
For any i ∈ [1 : d 1 ] and j ∈ [1 : n ] , eq.
(52) implies
[Equation]
Thus,
[Equation]
Therefore
[Equation]
the output of the ﬁrst layer after the activation is
[Equation]
[Table]
Also,
[Equation]
Recall in Theorem 4 we prove all components of ˜ Y (2) (cid:18)(cid:104) ˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =1 (cid:19) are positive.
Combining the deﬁnition of ˜ M (eq.
[Equation]
we have
[Equation]
Therefore,
[Equation]
Suppose the above argument holds for k -th layer
[Equation]
The output of ( k + 1) -th layer before the activation is
[Equation]
Recall proved in Theorem 4 that all components of ˜ Y ( k +1) (cid:18)(cid:104) ˜ W
cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =1 (cid:19) except those that are 0 are contained in ˜ Y ( k ) (cid:18)(cid:104)
˜ W (cid:48)
i (cid:105) L i =1 , (cid:104) ˜
b (cid:48)
i (cid:105) L i =1 (cid:19) .
We have
[Equation]
Therefore,
[Equation]
Thus
[Equation]
the argument holds for any k ∈ { 2 , . . .
, L − 1 }
[Equation]
So,
[Equation]
Therefore, ˆ R (cid:18)(cid:104)
[Equation] [Equation]
From eq.
(56) and Theorem 4, we have
[Equation]
which completes the proof of local minimizer.
Furthermore, the parameter M used in Lemma 5 (not those in this proof) is arbitrary in a continuous interval (cf. eq.
(44)), we have actually constructed inﬁnite spurious local minima.
A. A PREPARATION LEMMA
We now prove the preparation lemma used above.
Lemma 6.
Suppose u = ( u 1 · · · u n ) ∈ R 1 × n which satisﬁes u (cid:54) = 0 and n (cid:88)
i =
u i = 0 , (57) while { x 1 ,..., x n } is a set of vector ⊂
R m × 1 .
Suppose index set S = { 1 , 2 , · · · , n } .
Then for any series of real number { v 1 , · · · , v n } , there exists a non-empty separation I , J of S , which satisﬁes
I ∪ J = S , I ∩ J = ∅
and both I and J are not empty, a vector β ∈ R m × 1 ,such that, (1.1) for any sufﬁciently small positive real α , i ∈ I , and j ∈
J , we have v i − αβ T x
i < v j − αβ
T x j ; (1.2) (cid:80) i ∈ I
u i (cid:54) = 0 .
Proof.
If there exists a non-empty separation I and J of the index set S , such that when β = 0 , (1.1) and (1.2) hold, the lemma is apparently correct.
Otherwise, suppose that there is no non-empty separation I and J of the index set S such that (1.1) and (1.2) hold simultaneously when β = 0 .
Some number v i in the sequence ( v 1 , v 2 , · · · , v n ) are probably equal to each other.
We rerarrange the sequence by the increasing order as follows,
[Equation]
where s k = n .
Then, for any j ∈ { 1 , 2 , · · · , k − 1 } , we argue that
[Equation]
Otherwise, suppose there exists a s j , such that
[Equation]
Let I = { 1 , 2 , ..., s j } and J = { s j + 1 , ..., n } .
Then, when β = 0 , we have
[Equation]
59) and
[Equation]
, (60) which are exactly the arguments (1.1) and (1.2).
Thereby we construct a contrary example.
Therefore, for any j ∈ { 1 , 2 , · · · , k − 1 } , we have
[Equation]
Since we assume that u (cid:54) = 0 , there exists an index t ∈ { 1 , . .
, k − 1 } , such that there exists an index i ∈ { s t + 1 , ..., s t +1 } that u i (cid:54) = 0 .
Let l ∈ { s t + 1 , ..., s t +1 } is the index such that x l has the largest norm while u l (cid:54)
= 0 :
[Equation]
We further rearrange the sequence ( v s t +1 , ..., v s t +1 )
such that there is an index l (cid:48) ∈ { s t + 1 , . . .
, s t +1 } ,
[Equation]
62) and
[Equation] [Equation]
[Equation]
It is worth noting that it is probably l (cid:48)
=
s t +1 , but it is a trivial case that would not inﬂuence the result of this lemma.
Let I = { 1 , ..., l (cid:48) } , J = { l (cid:48) + 1 , ..., n } , and β = x l (cid:48) .
We prove (1.1) and (1.2) as follows.
Proof of argument (1.1) .
We argue that for any i ∈
I , v i − αβ T x i ≤ v l (cid:48)
There are three situations: (A) i ∈ { 1 , . . .
, s t } and j ∈ { s t +1 + 1 , · · · , n } .
Applying eq.
(58), for any i ∈ { 1 , . . .
, s t } and j ∈ { s t +1 + 1 , · · · , n } , we have that v i < v l (cid:48) and v j > v l (cid:48) .
Therefore, when α is sufﬁciently small, we have the following inequalities,
[Equation]
(B) i ∈ { s t + 1 , · · ·
[Equation]
l (cid:48) } .
Applying eq.
(63) and because of α > 0 , we have
[Equation]
Since v i = v l (cid:48) , it further leads to
[Equation]
(C) j ∈ { l (cid:48) + 1 , · · · , s t +1 } .
Similarly, applying eq.
(64) and because of α > 0 , we have
[Equation]
Since v j = v l (cid:48) , it further leads to
[Equation]
which is exactly the argument (1.1).
Proof of argument (1.2).
We argue that for any i ∈ { s t + 1 , · · · , l (cid:48) − 1 } , u i = 0 .
Otherwise, suppose there exists an i ∈ { s t + 1 , · · · , l (cid:48) − 1 } such that u i (cid:54) = 0 .
From eq.
(61), we have (cid:107) x
i (cid:107) ≤ (cid:107) x l (cid:48) (cid:107) .
Therefore,
[Equation]
where the ﬁrst inequality strictly holds if the vector x l (cid:48) and x i have the same direction, while the second inequlity strictly holds when x i and x l (cid:48) have the same norm.
Because x l (cid:48) (cid:54) =
x i , we have the following inequality,
[Equation]
which contradicts to eq.
(63), i.e.,
[Equation]
Therefore, (cid:88)
[Equation]
which is exactly the argument (1.2).
The proof is completed.
Remark.
For any i ∈ { l (cid:48) + 1 , ..., s t +1 } , we have
[Equation] [Equation]
while for any j ∈ { s t +1 + 1 , ..., n } , we have
[Equation]
Because v j > v l (cid:48)
[Equation]
when the real number α is sufﬁciently small
[Equation]
we have
[Equation]
Applying eqs.
(66) and (67)
[Equation]
we have
[Equation] [Equation]
Therefore, if l (cid:48) < s t +
[Equation]
we have
[Equation]
while if l (cid:48) = s t +1 ,
[Equation]
i
[Equation]
Eqs.
(70) and (71) make senses because l (cid:48) < n .
Otherwise, from Lemma 6 we have (cid:80) ni
=1 u i (cid:54) = 0 , which contradicts to the assumption.
B P ROOF OF T HEOREM
This appendix gives the proof of Theorem 2 omitted from Section 4.
We ﬁrst check that the squared loss is convex.
Lemma 7.
The empirical risk ˆ R under squared loss (deﬁned by eq.
(5)) is convex with respect to the prediction
ˆ Y .
Proof.
The second derivative of the empirical risk ˆ R under squared loss with respect to the prediction ˆ Y is
[Equation]
Therefore, the empirical risk ˆ R under squared loss is convex with respect to prediction ˆ Y .
Proof of Theorem 2.
We prove the four arguments separately.
Smooth and multilinear partition.
If the activations are all linear functions, the neural networks is reduced to a multilinear model.
The loss surface is apparently smooth and multilinear.
The nonlinearity in the activations largely reshape the landscape of the loss surface.
Speciﬁcally, if the input data ﬂows through the linear parts of every activation functions, the output falls in a smooth and multilinear region in the loss surface.
When some parameter changes by a sufﬁciently small swift, the data ﬂow may not move out of the linear parts of the activations.
This fact guarantees that each smooth and multilinear regions expands to an open cell.
Meanwhile, every nonlinear point in the activations is non-differentiable.
If the input data ﬂows through these nonlinear points, the corresponding empirical risk is not smooth with respect to the parameters.
Therefore, the nonlinear points in activations correspond to the non-differentiable boundaries between cells on the loss surface.
Local analogous convexity.
In every cell, the input sample points ﬂows through the same linear parts of the activations no matter what values the parameters are.
(1) We ﬁrst proves that the empirical risk ˆ R equals to a convex function with respect to a variable ˆ
W that is calculated from the parameters W .
Suppose ( W 1 , W 2 ) is a local minimum within a cell.
We argue that
[Equation]
( A
[Equation]
where A · ,i is the i -th column of the following matrix
[Equation]
The left-hand side (LHS) is as follows,
[Equation]
Meanwhile, the right-hand side (RHS) is as follows,
[Equation]
Apparently, LHS = RHS.
Thereby, we proved eq.
Afterwards, we deﬁne
[Equation]
and then straighten the matrix ˆ W 1 to a vector ˆ W ,
[Equation]
Also deﬁne
[Equation] [Equation]
Then, we can prove that the following equations,
[Equation]
Applying eq.
[Equation]
the empirical risk is transferred to a convex function as follows,
[Equation]
We can see that the empirical risk is rearranged as a convex function in terms of ˆ W which unite the two weight matrices W 1 and W 2 and the activation h are together as ˆ W .
Applying eqs.
(74) and (75)
[Equation]
we have
[Equation]
(2) We then prove that the local minima (including global minima) of the empirical risk ˆ R with respect to the parameter W is also local minima with respect to the corresponding variable ˆ W .
We ﬁrst prove that for any i ∈ [1 : d 1 d 2 ] , we have
[Equation]
where ∇ is deﬁned as follows,
[Equation]
To see this, we divide i into two cases:
( W 2 ) i (cid:54) =
and ( W 2 ) i = 0 .
Case 1: ( W 2 ) i (cid:54) = 0 .
The local minimizer of the empirical risk ˆ R with respect to the parameter W satisﬁes the following equation,
[Equation]
Therefore,
[Equation]
where ( W 2 ) is a vector and ( W 2 )
i is its i -th component.
Then, divid the both hand sides of eq.
(79) with ( W 2 ) i , we can get the following equation,
[Equation]
Case 2: ( W 2 ) i = 0 .
Suppose u 1 ∈ R d 0 is a unitary vector, u 2 ∈ R is a real number, and ε is a small enough positive constant.
Then, deﬁne a disturbance of W 1 and W 2 as follows,
[Equation]
When ε is sufﬁciently small, ∆ W 1 and ∆ W 2 are also sufﬁciently small.
Since ( W 1 , W 2 ) is a local minimum, we have
[Equation]
where ∆ is deﬁned as follows,
[Equation] [Equation]
Here, eq.
( ∗ ) comes from ( W 2 ) i = 0 .
Rearrange eq.
(80) and apply the Taylor’s Theorem, we can get that
[Equation]
Applying eq
[Equation]
[Equation]
we have
[Equation]
Here, eq.
( ∗∗ ) can be obtained from follows.
Because W 2 is a local minimizer, for any component ( W 2 ) i of W 2 ,
[Equation]
which leads to
[Equation]
When ε approaches 0 , eq.
(82) leads to the following inequality,
[Equation]
Since u 1 and u 2 are arbitrarily picked (while the norms equal 1 ), the inequality above further leads to
[Equation]
which ﬁnishes the proof of the argument.
Therefore, for any i and j , we have proven that
[Equation]
which demonstrates that
[Equation]
which means ˆ W is also a local minimizer of the empirical risk ˆ R ,
[Equation] [Equation]
(3) Applying the property of convex function, ˆ W is a global minimizer of the empirical risk
ˆ R , which leads to ( W 1 , W 2 ) is a global minimum inside this cell.
Local minima valley.
In the proof of Theorem 1, the parameters α i can be arbitrary as long as α i ∈ (0 , 1) and ˜ M can be any positive real sufﬁciently small.
When these parameters change, we can get inﬁnitely many local minima.
Also, when the parameters change, the data still ﬂows through the same parts of the activations.
Therefore, all these local minima are connected within a cell by a continuous path, on which all points have the same empirical risk.
Furthermore, the corresponding empirical rick remains invariant when the parameters change.
Linear collapse.
When there is no nonlinearities in the activations, there is apparently no non-differentiable regions on the loss surface.
In other words, the loss surface is a single smooth and multilinear cell.
The proof is completed.