S PURIOUS L OCAL M INIMA P ROVABLY E XIST FOR D EEP C ONVOLUTIONAL N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper we prove that a general family of inﬁnitely many spurious local minima exist in the loss landscape of deep convolutional neural networks with squared loss or cross-entropy loss.
Our construction of spurious local minima is general and applies to practical dataset and CNNs containing two consecutive convolutional layers.
We develop some new techniques to solve the challenges in construction caused by convolutional layers.
We solve a combinatorial problem to show that a differentiation of data samples is always possible somewhere in feature maps.
Empirical risk is then decreased by perturbation of network parameters that can affect different samples in different ways.
Despite ﬁlters and biases are tied in each feature map, in our construction this perturbation only affects the output of a single ReLU neuron.
We also give an example of nontrivial spurious local minimum in which different activation patterns of samples are explicitly constructed.
Experimental results verify our theoretical ﬁndings
I NTRODUCTION
Convolutional neural networks (CNNs) (e.g. Lecun et al.
; Krizhevsky et al.
; Simonyan & Zisserman (2015); Szegedy et al. (2015); He et al. (2016); Huang et al.
(2017)), one of the most important models in deep learning, have been successfully applied to many domains.
Spurious local minima, whose losses are greater than that of global minimum, play an important role in the training of deep CNNs and understanding of deep learning models.
It is widely believed that spurious local minima exist in the loss landscape of CNNs which is thought to be highly non-convex, as evidenced by some experimental studies (e.g. Dauphin et al.
; Goodfellow et al.
; Liao & Poggio (2017); Freeman & Bruna (2017); Draxler et al.
; Garipov et al.
; Li et al.
Mehmeti-Gopel et al.
However, the existence of spurious local minima for deep CNNs caused by convolutions has never been proved mathematically before.
In this paper, we prove that inﬁnite spurious local minima exist in the loss landscape of deep CNNs with squared loss or cross-entropy loss.
This is in contrast to the "no spurious local minima" property of deep linear networks.
The construction of spurious local minima in this paper is general and applies to practical dataset and CNNs containing two consecutive convolutional layers, which is satisﬁed by popular CNN architectures. The idea is to construct a local minimum θ at ﬁrst, and then construct another point θ (cid:48) in parameter space which has the same empirical risk as θ and there exist regions around θ (cid:48) with less empirical risks. However, the construction of spurious local minima for CNNs faces some technical challenges, and the construction for fully connected deep networks cannot be directly extended to CNNs. Our main contribution in this paper is to tackle these technical challenges.
In the construction of spurious local minima for fully connected deep ReLU networks (He et al.
Ding et al.
; Goldblum et al.
; Liu et al.
(2021)), in order to construct θ (cid:48) and perturb around it, data samples are split into some groups according to the inputs of a speciﬁc ReLU neuron such that each group will behave differently under the perturbation of network parameters so as to produce a lower risk.
This technique relies on data split and parameter perturbation, and cannot be directly applied to CNNs due to the following difﬁculties.
Every neuron in CNN feature maps has limited receptive ﬁeld that covers partial pixels in an input image (take image as an example), and hence the inputs to a ReLU neuron can be identical even for distinct samples, making them hard to distinguish.
This data split issue is further complicated by the nonlinear ReLU activations that truncate negative inputs, and the activation status can vary from place to place and from sample to sample.
Moreover, the ﬁlters and biases of CNNs are shared by all neurons in the same feature map, and thus adjusting the output of a ReLU neuron by perturbing these tied parameters will also affect other neurons in the same feature map.
We solve these challenges by developing some new techniques in this paper.
By taking limited receptive ﬁelds and possible distinct activation status for different locations and samples into account, we solve a combinatorial problem to show that a split of data samples is always possible somewhere in feature maps.
We then present a construction of CNN parameters ( θ (cid:48) ) that can be perturbed to achieve lower losses for general local minima θ .
Despite the parameters are tied, our construction can perturb the outputs of samples at a single neuron in feature maps without affecting other locations.
We also give a concrete example of spurious local minima for CNNs.
To our best knowledge, this is the ﬁrst work showing existence of spurious local minima in deep CNNs introduced by convolutional layers.
This paper is organized as follows.
Section 1.1 is related work.
Section 2 describes convolutional neural networks, and gives some notations used in this paper.
In section 3, our general results on spurious local minima are given with some discussions.
In section 4, we present an example of nontrivial spurious local minima for CNNs.
Section 5 presents experimental results to verify our theoretical ﬁndings.
Finally, conclusions are provided.
More lemmas, experimental details and all proofs are given in appendices.
1. R ELATED W ORK
For some neural networks and learning models, it has been shown that there exist no spurious local minima.
These models include deep linear networks ( Baldi & Hornik (1989); Kawaguchi (2016);
Lu & Kawaguchi (2017); Laurent & von Brecht (2018); Yun et al.
; Nouiehed & Razaviyayn (2018); Zhang (2019)), matrix completion and tensor decomposition (e.g., Ge et al.
(2016)), onehidden-layer networks with quadratic activation (Soltanolkotabi et al.
Du & Lee (2018)), deep linear residual networks (Hardt & Ma (2017)) and deep quadratic networks (Kazemipour et al.
Existence of spurious local minima for one-hidden-layer ReLU networks has been demonstrated, by constructing examples of networks and data samples, in Safran & Shamir (2018); Swirszcz et al.
; Zhou & Liang (2018); Yun et al.
; Ding et al.
; Sharifnassab et al.
; He et al.
; Goldblum et al.
(2020) etc.
For deep ReLU networks, He et al.
Ding et al.
; Goldblum et al.
Liu et al.
(2021) showed that spurious local minima exist for fully connected deep neural networks with some general loss functions.
For these spurious local minima, all ReLU neurons are active and deep neural networks are reduced to linear predictors.
Spurious local minima for CNNs are not treated in these works.
In comparison, we deal with spurious local minima for CNNs in this work, and the constructed spurious local minima can be nontrivial in which nonlinear predictors are generated and some ReLU neurons are inactive.
Du et al.
; Zhou et al. ; Brutzkus & Globerson (2017) showed the existence of spurious local minima for one-hidden-layer CNNs with a single non-overlapping ﬁlter, Gaussian input and squared loss.
In contrast, we discuss practical deep CNNs with multiple ﬁlters of overlapping receptive ﬁelds and arbitrary input, for both squared and cross-entropy loss.
Given non-overlapping ﬁlter and Gaussian input, the population risk with squared loss function can be formulated analytically with respect to the single ﬁlter w , which facilitates the analysis of loss landscape.
Thus, the techniques used in Du et al. ; Zhou et al. (2019); Brutzkus & Globerson (2017) cannot be extended to the general case of empirical risk with arbitrary input samples discussed in this paper. Nguyen & Hein (2018) showed that a sufﬁciently wide CNN (include a wide layer which has more neurons than the number of training samples followed by a fully connected layer) has a well-behaved loss surface with almost no bad local minima.
Liu (2022) explored spurious local minima for CNNs introduced by fully connected layers. Du et al.
Allen-Zhu et al.
(2019) explored the local convergence of gradient descent for sufﬁciently over-parameterized deep networks including CNNs.
P RELIMINARIES 2. N OTATIONS
We use M i, · and M ( i, · ) to denote the i th row of a matrix M , and use M · ,i and M ( · , i ) to denote the i th column of M .
M i,j and M ij are the ( i, j ) entry of M .
The i th component of a vector v is denoted as v i , v i or v ( i ) .
[ N ] is equivalent to { 1 , 2 , · · · , N } .
i : j means i, i + 1 , · · · , j . v
( i : end ) stands for the components of a vector v from the i th one to the last one.
1 n denotes a vector of size n whose components are all 1 s.
2. C ONVOLUTIONAL N EURAL N ETWORKS
A typical CNN includes some convolutional layers, pooling layers and fully connected layers.
Because spurious local minima caused by fully connected layers in CNNs can be treated in the same way as in fully connected networks, we will focus on spurious local minima caused by convolutional layers.
Convolutional layers can take advantage of the translational invariance inherent in data, and are deﬁned as follows.
Suppose the l th layer is a convolutional layer, neighboring neurons at layer ( l − 1) are grouped into patches for the convolution operation.
Let P l and s l be, respectively, the number of patches and the size of each patch in the l th layer, and denote by T l the number of convolutional ﬁlters (or the number of feature maps) in the l th layer.
Denote (cid:8) o l 1 , · · · , o lP l (cid:9) ∈
R s l as the set of patches in the l th layer. If there are multiple feature maps in layer ( l − 1) , a patch will include corresponding neighboring neurons from every feature map. The number of neurons in each feature map in the l th layer is P l − 1 .
The number of neurons in the l th layer is then n l = T l P l − 1 .
Given the p th ( p ∈ [ P l − 1 ]) patch o l − 1 p and the t th ( t ∈ [ T l ]) ﬁlter w lt ∈ R s l − 1 , the output of corresponding neuron in layer l is given as follows,
[Equation]
where h = ( t − 1) P l − 1 + p .
σ ( x ) = max (0 , x ) is the ReLU activation function.
There is a single bias b lt for each feature map.
For ease of presentation, we will assume that the strides for convolutions are equal to one and the input feature maps are zero padded such that for each convolutional layer, the sizes of input and output feature maps are equal.
The convolution operation can then be reformulated as a matrixvector product.
We explain this by considering the following one-dimensional example. Given a one-dimensional input o l − 1 = ( a, b, c, d, e ) T and a ﬁlter w {+
+} = w 11 , w 21 , w 31 ) T , the input becomes (0 , a, b, c, d, e, 0) T after padding with zeros.
Here, T l = 1 , s l − 1 = 3 , P l − 1 = 5 after zero-padding, n l = T l P l − 1 = 5 . The convolution operation is equivalent to o l = σ       ww
www    www    www   ww      abcd
e    +   
b lb lb lb lb l      .
One can set ( w 11 = 0 , w 21 = 1 , w 31 = 0) in 2 to forward o l − 1 unchanged (before adding bias and ReLU activation).
This is equivalent to setting the weight matrix as a identity one.
For the case of two-dimensional input, the input can be converted into a vector and the convolution operation can be reformulated as a matrix-vector product using similar idea.
We will use matrix-vector product to represent convolution operations.
Denote the weight matrix and bias vector of the l th layer as W l and b l , respectively, with W l ∈ R n l × n l − 1 and b l ∈ R n l , then the output of the l th layer is o l = σ ( W l o l − 1 + b l ) .
This matrix-vector product formulation can also be used to represent fully connected layers.
We only consider average pooling layers.
If layer l is a average pooling layer, each patch used in the pooling operation will include only the neighboring neurons within a single feature map in layer ( l − 1) .
Let P l − 1 be the number of patches in a single feature map of the ( l − 1) th layer, and hence the size of each feature map in layer l is P l − 1 .
For average pooling, the output of the p th neuron o l ( p ) in each feature map of layer l is computed by
[Equation]
where o l − 1 p ( i ) is the i th element in the p th patch of layer ( l − 1) within a feature map.
The average pooling operation is a linear operation and thus can can be represented by a matrix-vector product. The exact forms of parameter matrices for average pooling and fully connected operations will be given in Appendix A in detail.
Consider a training set { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . .
, ( x N , y N ) } , where x i ∈ R d x , y i ∈ R d y ( i ∈
[ N ]) are, respectively, the input and target output.
Let L be the number of layers in a CNN, and let o l,i be the output o l for the i th data sample (denote o 0 ,i := x i ).
The output vector of a CNN is
[Equation]
We introduce a diagonal matrix I l,i ∈
R n l × n l for each sample and each ReLU layer to represent the activation status of ReLU neurons, whose diagonal entries are I l,ik,k
=
if W l ( k, · ) o l − 1 ,i + b lk >and I L,ik,k = 0 otherwise.
Consequently, the output of a convolutional layer can be written as o l,i =
I l,i (
W l o l − 1 ,i + b l ) .
The CNN output can be written as
[Equation]
The empirical risk (training loss) is deﬁned as
[Equation]
where l is the loss function, such as the widely used cross-entropy loss for classiﬁcation problems.
Our main goal in this paper is to construct spurious local minima for the empirical risk function of deep CNNs given in (5).
S PURIOUS L OCAL M INIMA
In this section, we ﬁrst construct a general local minimum with parameters θ := (cid:0) W l , b l (cid:1)
L l =1 .
Then, we present another point θ (cid:48) in parameter space which has an equal empirical risk with θ .
We further perturb θ (cid:48) and show that empirical risk can be decreased.
Therefore, θ is a spurious local minimum in parameter space.
Our construction is general and can be applied to CNNs with two consecutive convolutional layers, which is easily satisﬁed by most practical CNN models, such as AlexNet (Krizhevsky et al.
VGG (Simonyan & Zisserman (2015)) and GoogleNet (Szegedy et al.
3. C ONSTRUCTION OF LOCAL MINIMA
We ﬁrst give a general construction of nontrivial local minima.
Given a local minimum for a subnetwork of a CNN, we show that it will induce a local minimum of the whole CNN if the subnetwork is embedded into the CNN appropriately.
Local minima for subnetworks of a CNN can be constructed using any method.
Our construction of spurious local minima is general and does not rely on the concrete form of local minima for subnetworks.
In section 4, we will give an explicit construction of exemplar nontrivial local minima for subnetworks.
Our construction of spurious local minima is general and will utilize two consecutive convolutional layers, whereever there are in the CNN.
For ease of presentation, ﬁrst we will assume that the last two hidden layers before the output layer (the L th layer, which is fully connected) are two consecutive convolutional layers.
The general case of having two consecutive convolutional layers in other places will be discussed in section 3.3.
When layers L − 1 and L − 2 are two consecutive convolutional layers, we will ﬁx the parameters of the last two convolutional layers in a way such that the output o L − 3 ,i is passed through layers L −and L − 1 unchanged.
Two feature maps in layers L − 2 and L − 1 , respectively, are reserved for the perturbation described later.
Without loss of generality, let them be the ﬁrst and second feature maps in these two layers, respectively.
Fig.1(a) shows the top layers of the CNN architeture with some associated parameters.
Note that the size of a single output feature map in the l th layer is P l − 1 , and P L − 2 =
P L − 3 =
P L − 4 after padding.
We have the following lemma to informally descibe the construction of local minima θ , and its formal version is given Lemma 6 in Appendix C.
[Figure]
Lemma 1.
(informal). Given a CNN whose numbers of ﬁlters in the last two convolutional layers satisfy T L − 1 ≥ 3 , T L − 2 ≥ 3 , the parameters W L ( · , 1 : 2 P L − 2 ) , W L − 1 , b L − 1 and W L − 2 , b L −of the last three layers are set to propagate o L − 3 ,i unchanged to the output neurons, and the ﬁrst and second feature maps in both layer L − 2 and layer L − 1 have no contributions to the ﬁnal output.
The remaining parameters, including W L ( · , 2 P L − 2 + 1 : end ) , b L and (cid:0) W l , b l (cid:1) L − 3 l =1 , constitute a subnetwork, and if they locally minimize the training loss when ﬁxing other parameters, then point θ := (cid:0) W l , b l (cid:1)
L l =1 is a local minimum in parameter space.
In this lemma, W L ( · , 1 : 2 P L − 2 ) means the weights of connections from output neurons to the ﬁrst two feature maps in layer L − 1 .
The identity forward propagations in layers L − 1 and L − 2 (see Fig.1(a)) can be implemented by setting corresponding submatrices in W L − 1 and W L − 2 to identity ones, which is equivalent to setting corresponding ﬁlters to the form of (0 , · · · , 1 , · · · , 0) T ( with a single nonzero entry in the middle) as explained in section 2.2, and setting biases b L − 1 , b L − 2 to zeros.
The embedding scheme to construct local minima or saddle points has also been used for fully connected networks by Fukumizu et al.
; Fukumizu & Amari (2000);
He et al.
; Liu et al.
; Zhang et al.
However, embedding for CNNs needs to tackle the problems of limited receptive ﬁelds, arbitrary activation status at different locations, and tied weights and biases. 3. C ONSTRUCTION OF SPURIOUS LOCAL MINIMA
Our construction of spurious local minima relies on the following assumption.
Assumption 1 .
At local minimum θ , the inputs to the ﬁnal fully connected layer are distinct for different samples, i.e., ∀ i, j ∈ [ N ] and i (cid:54) =
j, o L − 1 ,i (cid:54) =
o L − 1 ,j .
For practical CNNs and datasets with rich contents, the number of neurons n L − 1 input to the ﬁnal fully connected layer is big and the chance of o L − 1 ,i (note that o L − 1 ,i (cid:54)
=
since I L − 1 ,i (cid:54) =by the nondegenerate requirement in Lemma 6, the formal version of Lemma 1) being identical for different samples is very small, thus Assumption 1 is reasonable and enforces a very mild restriction on local minimum θ .
This assumption is used to prevent distinct samples from becoming indistinguishable when passing through ReLU layers and being truncated by them. For fully connected deep neural networks, (He et al. ; Liu et al.
(2021)) assumed that all data points are distinct, i.e., ∀ i, j ∈ [ N ] and i (cid:54) = j, x i (cid:54) = x j . Under this condition, we show in Lemma 7 in Appendix C that for practical datasets and CNNs there always exists local minimum θ for which Assumptionholds.
An exception is the neural collapse phenomenon that during the terminal phase of training, the features of the ﬁnal hidden layer tend to collapse to class feature means.
As a result, our construction of spurious local minima does not include those that may arise during the ending training phase when the loss is driving towards zero.
3.2. DATA SPLIT
In the following, we will show that given Assumption 1, there must exist at least one location in feature maps where the outputs of all samples can be split into two parts using a threshold such that the two parts will behave differently under perturbations.
This is not a easy task due to the following difﬁculties introduced by convolutional layers.
Firstly, the receptive ﬁeld of each neuron in a convolutional layer is smaller than the input image, thus at each location in feature maps, the effective inputs (pixels in the receptive ﬁeld) from distinct samples can be identical and indistinguishable.
Secondly, the activation status of hidden ReLU neurons can vary from place to place and from sample to sample.
Assumption 1 will help us to distinguish different samples.
Thirdly, the ﬁlters and biases are shared by all locations in a feature map, and one cannot perturb these parameters at a location without affecting other places.
We assume that training loss at θ satisﬁes R ( θ ) > 0 .
By Lemma 8 given in Appendix C, for training data that cannot be ﬁt by linear models (also assumed by He et al. ; Liu et al. and popular CNN architectures, there always exists point θ such that both Assumption 1 and R ( θ ) >hold.
Moreover, by Lemma 11 in Appendix E, for squared and cross-entropy loss function l , ∂l ( o ( x i ) , y i ) ∂ o cannot equal to zero for all samples if R ( θ ) > 0 , where o ( x i ) is the output function of input x i given by 4.
Without loss of generality, assume u i :
=
∂l ( o ( x i ) , y i )
∂o
(1) (cid:54) =
for some i ∈ [ N ] , where o (1) is the ﬁrst component of output vector o .
By optimality of θ := (cid:0) W l , b l (cid:1) L l =1 , we have ∂R∂b
[Equation]
We will split the outputs of all samples at layer L − 3 .
Given CNN parameters θ := (cid:0) W l , b l (cid:1) L l =1 and data samples { x 1 , x 2 , · · · , x N } , we consider the outputs o L − 3 ,i ( j ) ( i ∈ [ N ] , j ∈ [ M ]) in feature maps of layer L − 3 , where M is the number of neurons (locations) in all feature maps of layer L −used in Lemma 1.
For all j ∈ [ M ] , i ∈ [ N ] , we denote
[Equation]
v ji is the output at layer L − 3 before adding bias and ReLU activation.
Thus, at each location j ∈ [ M ] , there is a list ( v j 1 , v j 2 , · · · , v jN ) .
Let ˜ x
ji be the vector composed of pixels of x i in the receptive ﬁeld of the j th location.
˜ x ji can be identical for distinct samples, resulting in identical v ji s. Identical v ji s can also be resulted from ReLU activations that remove the difference between input samples.
In general, at each location j in the feature maps, ( v j 1 , v j 2 , · · · , v jN ) are organized in groups according to their magnitudes.
Assume there are g j groups in the list ( v j 1 , v j 2 , · · · , v jN ) .
Without loss of generality, we can re-index the samples in descending order of v ji and write the ordered list as
[Equation]
Let u = ( u 1 , u 2 , · · · , u N ) T and note that u (cid:54) = 0 and (cid:80) N i =1 u i = 0 .
In the following lemma, we show that there is at least one location where the split of data samples exists.
This is a combinatorial problem which needs to deal with multiple locations and multiple ordered groups at each location, and the groupings of samples can be arbitrary at each location.
Lemma 2.
Assume
that training loss R ( (cid:0) W l , b l (cid:1)
L l =1 ) > 0 for squared or cross-entropy loss.
Under Assumption 1, there exist some locations j ∈
[ M ] where the ordered list v j 1 = v j 2 = · · · v jm 1 > v jm 1 +1 = · · · v jm 2 > · · · > v jm gj − 1 +1 = · · · = v jm gj = v jN can be splitted into two parts (cid:110) v j 1 , v j 2 , · · · , v jn (cid:111) and (cid:110) v jn +1 , · · · , v jN (cid:111) by a threshold η such that
[Equation]
There may be more than one location where such data split exists, and we choose the one with the biggest threshold η for later usage.
Let h be the location with the biggest split threshold, and without loss of generality, assume it is located in the ﬁrst feature map of layer L − 3 (that is why we put a special emphasis on the ﬁrst and second feature maps in layers L − 2 and L − 1 in Lemma 6 and
Lemma 9).
Since η for the h th location is the biggest threshold among all locations j ∈
[ M ] , by 7, (cid:80) ni =1 u i (cid:54) = 0 are not satisﬁed at locations whose thresholds are less than η , thus we have [Equation] An Auxiliary Lemma At each location j ∈ [ M ] , samples are organized into groups according to v ji ( i ∈ [ N ]) . v ji s in each group are equal.
Let I jq be the set of indices of samples in the q th group of the j th location.
We have the following lemma to show that (cid:80) i ∈
I jq u i cannot equal to zero for all groups and all locations.
Lemma 3.
Under Assumption 1, for CNNs with squared or cross-entropy loss, if R ( (cid:0) W l , b l (cid:1) L l =1 ) > 0 , then there must exist some j ∈ [ M ] and q ∈ [ g j ] such that (cid:80) i ∈
I jq u i (cid:54) = 0 . Lemma 3 will be used by Lemma 2 to ﬁnd data split that satisﬁes (7). We will prove Lemmaby induction.
The number of groups g j at each location and the value of v ji for each group can be arbitrary.
We will prove that if Lemma 3 holds for m samples, then for all possible conﬁgurations of current groups and all ways of adding a new sample, it also holds for m + 1 samples.
3.2. C ONSTRUCTION OF θ (cid:48)
Given the CNN parameters θ :=
( W l , b l ) Ll =1 speciﬁed in Lemma 6 and the threshold η in Lemma 2, we will give the point θ (cid:48) :
= (cid:0) W (cid:48) l , b (cid:48) l (cid:1)
L l =1 in parameter space and show that training loss at θ (cid:48) is equal to that at θ .
We only change the parameters that are related with the ﬁrst and second feature maps in layers L − 2 and L − 1 , respectively, to obtain θ (cid:48) , and remaining parameters are ﬁxed.
The purpose of these parameter setting is to make R ( θ (cid:48) ) = R ( θ ) and enable different behaviors under perturbation of parameters for different parts of samples.
Different settings are designed depending on whether η ≥ − b L − 3or not.
The setting for the case of η ≥ − b L − 3is illustrated in Fig.1(b), and the setting for η < − b L − 3is shown in Fig.1(c). The following lemma informally describe the parameter settings and its formal version is given in Lemma 9 in Appendix C).
Lemma 4.
(informal).
Given the CNN parameters θ :=
( W l , b l ) Ll =1 speciﬁed in Lemma 1 (formally
Lemma 6 in Appendix C), let θ (cid:48) :
= (cid:0) W (cid:48) l , b (cid:48) l (cid:1)
L l =1 , and set W (cid:48)
L , b L , W (cid:48)
L − 1 , b L − 1 and W (cid:48)
L − 2 , b L −in a way such that the value w j · σ ( v ji + b L − 3)
+ b L 1 (the original output of the ﬁrst output neuron, contributed by location j in the ﬁrst feature map of layer L − 3 ) is not changed for every location j and each sample i ∈ [ N ] , no matter how big v ji is relative to η and b L − 3.
There are three possible paths through which a value σ ( v ji + b L − 3) can propagate, i.e., the paths that connect the ﬁrst, second, and third feature maps, respectively, in layers L − 2 and L − 1 (see Fig.1).
If η ≥ − b L − 3, a positive value of σ ( v ji + b L − 3) will pass through either the ﬁrst or the second path.
If η < − b L − 3, a positive value of σ ( v j i + b L − 3) will pass through all three paths and the outputs from the ﬁrst and second paths counteract.
The differentiation of paths will make it possible for different parts of samples to behave differently under perturbation of parameters.
Remaining parameters in θ keep ﬁxed.
Then, R ( θ (cid:48) )
=
R ( θ ) .
3.2. PERTURBATION OF θ (cid:48)
Next, we will show that by perturbing θ (cid:48) , we can decrease the training loss.
We demand that only the ﬁnal output contributed by location h in the ﬁrst feature map of layer L − 3 , where v j 1 , v j 2 , · · · , v jn > η and (cid:80) ni
=1 u i (cid:54) = 0 happens, will be affected by the perturbation so as to decrease the training loss.
The perturbation is then designed such that a single bias in the path each v ji ( with v ji > η ) passes through is perturbed. The ﬁnal output is not affected by locations other than h even with v ji > η due to (cid:80) ni =1 u i = 0 . The following lemma informally describe our parameter perturbation scheme and its formal version is given in Lemma 10 in Appendix C).
Lemma 5.
(informal). If η ≥ − b L − 3, perturb b (cid:48)
L − 2.
If η < − b L − 3, perturb b (cid:48)
L − 2.
Remaining parameters in θ (cid:48) keep ﬁxed.
Then under Assumption 1, a training loss lower than R ( θ (cid:48) ) is obtained under this perturbation.
3. M AIN R ESULTS AND DISCUSSION
Combining Lemma 1, Lemma 4 and Lemma 5, we have the following theorem.
Theorem 1.
Under Assumption 1, the local minima θ := (cid:0) W l , b l (cid:1)
L l =1 given in Lemma 1 are spurious if R ( θ ) > 0 for squared or cross-entropy loss.
Due to nonnegative homogeneity of ReLU activation, inﬁnitely many spurious local minima exist by scaling the parameters of different layers in θ . In the general case when the two consecutive convolutional layers we utilized are not located at the top and there are some convolutional, average pooling, or fully connected layers between them and the output layer, we can still construct spurious local minima using similar idea. The only difference is the setting of parameters for layers above the two consecutive convolutional layers, and we set them such that the output of the two consecutive convolutional layers are propagated unchanged (except pooling operations in possible subsequent pooling layers) to the ﬁrst fully connected layer, which plays the role of layer L in Lemmas 1 and 4, and then the output of the ﬁrst fully connected layer is forwarded unchanged to the output neurons.
For such constructed θ and θ (cid:48) , we can show that θ is still a local minimum, there is still R ( θ (cid:48) )
=
R ( θ ) and the empirical risk can be decreased by perturbing θ (cid:48) . The details will be given in Appendix D. The requirement of having two consecutive convolutional layers may be further relaxed, and we leave it to our future work.
Remark 1: Our construction of spurious local minima shows that for practical datasets and CNNs each local minimum of the subnetwork is associated with a spurious local minimum. Since the output of a CNN with ReLU activations is a piece-wise linear function, and from the perspective of ﬁtting data samples with piece-wise linear output (Liu (2022)), the local minima of subnetworks of a CNN (and consequently the spurious local minima of the CNN) may be common due to the abundance of different ﬁtting patterns.
Furthermore, as suggested by Xiong et al.
(2020), CNNs have more expressivity than fully connected NNs per parameter in terms of the number of linear regions produced.
The ability of producing more linear regions implies more ﬁtting patterns, thus we conjecture that CNNs are more likely to produce spurious local minima than fully connected NNs of the same size. We leave the exploration of these ideas to our future work.
A N E XAMPLE OF N ONTRIVIAL S PURIOUS L OCAL M INIMA
In this section, we will construct an example of nontrivial local minimum in which some neurons are inactive, which can serve as the subnetwork in Lemma 1.
By Theorem 1, the associated local minimum for the CNN that contains this subnetwork is spurious.
In comparison, in He et al.
Ding et al.
; Goldblum et al.
; Liu et al.
(2021) for deep ReLU networks, spurious local minima are trivial since all ReLU neurons are active and deep neural networks are reduced to linear predictors.
The idea is to split data samples into two groups using a hyperplane that is perpendicular to an axis, and then ﬁt each group of samples using different predictors.
We design a CNN architecture with appropriate parameters to generate these two predictors, and show that perturbing parameters will increase or at least maintain the loss.
Given data samples { x 1 , x 2 , · · · , x N } , we use a hyperplane with normal p to split the data samples.
Choose any index k ∈ [ d x ] , we set p = (0 , 0 , · · · , 1 , 0 , · · · , 0) T where only the k th component of p is 1.
Without loss of generality, assume the k th component of input x is in located in its ﬁrst channel.
Let I =
(cid:8) i | x ki = max j ∈
[ N ] x kj (cid:9) , i.e., I is the set of indices of samples having the largest k th component.
Choose one element i ∗ in set I , and denote by j ∗ the index of any sample with the largest x k among x kn ( n ∈ [ N ] , n / ∈ I ) , then we set
[Equation]
For any point x on the positive side of hyperplane p , we have
[Equation]
We ﬁt the two groups of samples in different ways using two subnetworks.
Concatenating the two subnetworks, we obtain a full CNN.
The following theorem gives an informal description of its architecture and parameters.
The formal description will be given in Appendix B. [Figure]
Theorem 2.
(informal).
Given a CNN with two subnetworks, its parameters θ 1 = (cid:0) W l , b l (cid:1)
L l =1 are set as follows. The parameters of the ﬁrst subnetwork are set such that the k th component of sample x i ( i ∈ I ) is forward propagated to the ﬁnal fully connected layer, and that of sample x i ( i / ∈ I ) is blocked.
The parameters of the second subnetwork are optimized to minimize the empirical loss for samples { x j | j ∈ [ N ] , j / ∈ I } .
Then, the point θ 1 is a local minimum in parameter space.
For this local minimum, the ReLU units in the ﬁrst subnetwork that correspond to the k th component of input are only activated by samples x j ( j ∈ I ) and are inactive for remaining samples.
Thus, the local minimum (cid:0) W l , b l (cid:1)
L l =1 is nontrivial for which different activation patterns exist and the resulting predictor is nonlinear.
When using the local minimum θ 1 constructed in Theorem 2 as the subnetwork of θ in Lemma 1, in order to further construct θ (cid:48) shown in Lemma 4, we need to make sure that Assumption 1 and R ( θ ) > 0 hold. With the same ideas as dicussed in section 3.2.1, we can construct the second subnetwork such that there always exists point θ 1 (and consequently θ ) for which both Assumption 1 and R ( θ ) >
0 hold for popular datasets and CNN architectures.
E XPERIMENTAL R ESULTS
We conduct experiments on CIFAR-10 image set to verify the correctness of Theorem 1 and Theorem 2.
We use two approaches to show the existence of spurious local minima.
The ﬁrst one is to visualize the loss landscape around local minima θ using the technique given in Li et al.
Given two random directions δ and η , we compute the empirical losses R ( θ + αδ + βη ) at grid points in the two-dimensional plane speciﬁed by δ and η , where ( α, β ) are the coodinates of each grid point.
Then, the level sets of empirical loss are depicted using empirical losses at these grid points.
The second approach is to compute the empirical losses around θ along many random directions and see whether losses lower than R ( θ ) exist.
Experimental details are given in Appendix F.
Fig.2 shows the results of the level set visualization approach.
Fig.2(a) and Fig.2(b) demonstrate that the local minimum θ constructed in Lemma 6 is spurious since R ( θ (cid:48) )
=
R ( θ ) and local minimum with loss lower than R ( θ (cid:48) ) exists.
Fig.2(c) shows that the point θ 1 constructed in Theorem 2 is a local minimum.
Fig.3 shows the results of random direction approach and will be given in Appendix F.
C ONCLUSION
We have proved that convolutional layers can introduce inﬁnite spurious local minima in the loss landscape of deep convolutional neural networks with squared loss or cross-entropy loss.
To show this, we developed new techniques to solve the challenges introduced by convolutional layers.
We solved a combinatorial problem to demonstrate that a split of outputs of data samples is always possible somewhere in feature maps.
In this combinatorial problem, we overcame the difﬁculty of arbitrary groupings of outputs of data samples caused by limited receptive ﬁelds and arbitrary activation status of hidden neurons.
We also solved the tied parameters problem, giving perturbations of ﬁlters and biases to decrease the training loss that affect only the output of a single neuron in the feature map. A P ARAMETER MATRICES FOR AVERAGE POOLING AND FULLY CONNECTED OPERATIONS
We give a one-dimensional example to show how to formulate the average pooling operation as a matrix-vector product. Given a one-dimensional input o l − 1 = ( a, b, c, d ) T , suppose the patches for pooling are non-overlapping, which is usually the case, and the patch size is 2. The output is then o l = ( 12 ( a + b ) , 12 ( c + d )) T , which is equivalent to the following matrix-vector product [Equation]
Thus, W l = 12 (cid:18) 1 1 0 0 0 0 1 1 (cid:19) .
Parameter matrix for average pooling operation in two-dimensional case can be obtained using similar idea by converting the feature maps in input layer into a vector, and the nonzero entries in each row of W l are not all adjacent.
For fully connected layers, there are no constraints on the entries of parameter matrix W l ∈ R n l × n l − 1 .
B A N E XAMPLE OF N ONTRIVIAL L OCAL M INIMA FOR CNN S
We ﬁt the two groups of samples separated by the hyperplane with normal p in different ways.
For samples { x j | j ∈ [ N ] , j / ∈ I } , we use a subnetwork with parameter matrix W 2 to ﬁt them, i.e., the output of this subnetwork is
[Equation]
where ˆ x j is the input augmented with scalar 1 (at the same time weight matrices are augmented to include biases (denoted as ˆ W l ) to make notations simple).
We use a subnetwork with parameter matrix W 1 to ﬁt data samples { x j | j ∈ I } , and W 1 is given by [Equation] ∈ [ d y ] . [Equation]
Then, the output is
[Equation]
{ λ i , i ∈ [ d y ] } are constants determined as follows.
If there is only one sample x i ∗ in set I with target output y i ∗ and the loss function is the squared loss, there must exist ˜ y
i ∗ such that l (˜ y i ∗ , y i ∗ )
= 0 .
Let o i ∗ =
˜
y i ∗ , then we have ˜ y
i ∗ ( j ) =
W 2 ( j, · )ˆ x i ∗ + λ j ( p · x i ∗
[Equation]
c p
[Equation]
, ∀ j ∈ [ d y ] , which leads to
[Equation]
If there are multiple elements in set I or the loss function is the cross-entropy loss, the parameters { λ j , j ∈ [ d y ] } are determined by
[Equation] Concatenating the two subnetworks, we obtain a full CNN. The following theorem gives its architecture and parameters (cid:0) W l , b l (cid:1) L l =1 , and shows that θ 1 := (cid:0) W l , b l (cid:1)
L l =1 is a local minimum.
For ease of presentation, we ignore pooling layers at ﬁrst.
Theorem 3.
(formal version of Theorem 2).
Given a CNN, assume the number of ﬁlters in each convolutional layer satisﬁes T l ≥ 2 ( l ∈
[ L − 1]) , and the size of each convolutional layer satisﬁes n l − P l − 1 ≥ d y ( l ∈
[ L − 1]) .
Its parameters are given as follows.
The weight matrix of the ﬁrst ﬁlter in the ﬁrst convolutional layer is
[Equation]
where I d P 0 × P 0 is the identity matrix of size P 0 × P 0 , P 0 is the number of patches in the input used for convolution.
The corresponding bias is
[Equation] [Equation]
The ﬁlters and biases of higher convolution layers are set as
[Equation]
to forward propagate the ﬁrst feature map of o 1 ( x i ) unchanged.
The subnetwork with parameter matrix W 2 is formed by the part of CNN that removes the ﬁrst ﬁlter and all activation units in each layer, and its output is given as follows.
For all j ∈ [ N ] and j / ∈ I , o
[Equation]
W 2 is optimized to minimize (cid:80) i ∈
[ N ] , i/ ∈
I
[Equation]
2 ˆ x i
[Equation]
The ﬁnal fully connected layer is set as
[Equation]
21) where only the k th component connecting to the ﬁrst subnetwork is nonzero and equals to λ i .
Sufﬁciently large positive constants
c l ( l ∈
[ L − 1]) are added to the biases of each convolutional layer in the second subnetwork such that the involved ReLU units are all activated.
The biases of top fully connected layer b Li ( i ∈ [ d y ]) are then substituted by b Li − ∆ o L ( i ) , where ∆ o L ( i ) is recursively computed as
[Equation]
All remaining parameters are set to zeros.
Then, the point θ 1 = (cid:0) W l , b l (cid:1)
L l =1 speciﬁed above is local minimum in parameter space.
If pooling layers are used in CNNs, we can adjust the parameters in Theorem 3 to reﬂect the effects of parameter matrices of pooing layers.
However, the adjusted (cid:0) W l , b l (cid:1)
L l =1 is still a local minimum.
The condition for the number of ﬁlters is used to contain two subnetworks.
The condition for the size of each convolutional layers is used to make sure θ 1 presented in Theorem 3 meets the full rank requirement.
Because the size of each convolutional layer satisﬁes n l − P l − 1 ≥ d y ( l ∈ [ L − 1]) , the matrix W 2 ∈ R d y × d x for the second predictor is full rank around θ 1 (see the proof of Theorem 3 in
Appendix G.9).
These conditions are easily satisﬁed by most CNNs used in practice.
C L EMMAS FOR C ONSTRUCTION OF SPURIOUS LOCAL MINIMA
The following Lemma 6 descibe the construction local minima θ , and it is the formal version of Lemma 1 in section 3.1. Lemma 6.
Given a CNN whose numbers of ﬁlters in the last two convolutional layers satisfy
T L − 1 ≥ 3 , T L − 2 ≥ 3 , the parameters of its last three layers are set as follows to propagate o L − 3 ( x ) to the output layer.
The ﬁrst and second feature maps in both layer L − 2 and layer L − 1 have no contributions to the ﬁnal output.
[Equation]
The remaining parameters, including W L ( · , 2 P L − 2 + 1 : end ) , b L and (cid:0) W l , b l (cid:1) L − 3 l =1 , constitute a subnetwork, and they locally minimize the following training loss when ﬁxing those parameters in (23),
[Equation]
Assume that ˆ I l,i (cid:54) = 0 for all l ∈
[ L − 1] and i ∈ [ N ] .
Then, the point θ := (cid:0) W l , b l (cid:1)
L l =1 obtained above is a local minimum in parameter space.
The requirement in Lemma 6 that ˆ I l,i (cid:54) =
for all l ∈ [ L − 1] and i ∈ [ N ] is used to prevent degenerate cases in which all ReLU neurons in a layer are deactivated for some samples.
The nondegenerate cases always exist since we can ﬁx sufﬁciently big biases such that all ReLU neurons are turned on and then train the subnetwork.
The following Lemma 7 shows that for practical datasets and popular CNN architectures, there always exists point θ in parameter space where Assumption 1 holds.
Lemma 7.
For CNNs whose number of feature maps in each convolutional and average pooling layer is greater than or equals to the number of input channels n 0 and width of each fully connected layer is bigger than or equals to that of output layer, and datasets in which all data points are distinct, i.e., ∀ i, j ∈ [ N ] and i (cid:54) =
j, x i (cid:54) =
x j , if the distinctness of samples is preserved after each pooling operation of CNNs when directly applied to the input, then there always exists point θ for which
Assumption 1 holds.
For popular CNN architectures, such as AlexNet and VGG used in CIFAR10 and ImageNet image classiﬁction tasks, the number of feature maps in each layer is greater than the number of input channels, and the output layer is indeed narrower than hidden fully connected layers.
For practical datasets, the distinctness of samples is usually preserved after several pooling operations.
Thus, the conditions in Lemma 7 are reasonable in practice.
Proof.
We consider the general case in which the two consecutive convolutional layers we utilized are not located at the top, and there are some convolutional, pooling, or fully connected layers between them and the ﬁnal output layer.
For CNNs whose number of feature maps in each convolutional and average pooling layer is greater than or equals to the number of input channels, we can set the parameters (cid:0) W l , b l (cid:1)
L l =1 such that each convolutional layer forward propagates its input unchanged using the ﬁrst n 0 feature maps and the remaining feature maps are set to zeros using zero ﬁlters and biases.
Also, since the width of each fully connected layer is bigger than or equals to that of output layer, we use the ﬁrst d y neurons in the ﬁrst fully connected layer to perform the fully connection operation, and subsequent fully connected layers forward propagate their inputs unchanged to the ﬁnal output neurons.
Sufﬁciently large biases are used in the ﬁrst convolutional layer and the ﬁrst fully connected layer such that all ReLU neurons in the ﬁrst n 0 feature maps of convolutional and pooling layers and the ﬁrst d y ReLU neurons in hidden fully connected layers are turned on, and their effects can be cancelled at the ﬁnal output layer (as done for the second subnetwork in 22 in Theorem 3).
By doing so, the training loss of the CNN can be expressed as R ( θ ) =
1 N (cid:80) Ni =1 l (cid:0) W fc W pool x i , y i (cid:1) , where W pool is a constant matrix that characterizes the average pooling operations from input to the ﬁrst fully connected layer, and W fc characterizes the computation in the ﬁrst fully connected layer.
The CNN has been reduced to a linear classiﬁer and the corresponding point θ in parameter space becomes a local minimum when minimizing 1 N (cid:80) Ni =1 l (cid:0) W fc W pool x i , y i (cid:1) with respect to W fc .
The only operation that can affect the distinctness of o L − 1 , i ( i ∈ [ N ]) at layer L − 1 (or more generally, the two consecutive convolutional layers used in the construction of spurious local minima) is then the pooling operation.
If the distinctness of samples x
i ( i ∈ [ N ]) is preserved after each pooling operation of the CNN when directly applied to x i (i.e., W pool x i ), then ∀ i, j ∈ [ N ] and i (cid:54)
=
j, o L − 1 ,i (cid:54) =
o L − 1 ,j , hence Assumption 1 holds for θ .
The following Lemma 8 shows that for training data that cannot be ﬁt by linear models and popular
CNN architectures, point θ in parameter space with training loss R ( θ ) > 0 always exists.
Lemma 8.
For training data that cannot be ﬁt by linear models, and CNNs whose number of feature maps in each convolutional and average pooling layer is greater than or equals to the number of input channels n 0 and width of each fully connected layer is bigger than or equals to that of output layer, there always exists a point θ = (cid:0) W l , b l (cid:1)
L l =1 in parameter space such that training loss R ( θ ) > 0 .
The point θ = (cid:0) W l , b l (cid:1)
L l =1 in Lemma 8 can be the same as that in Lemma 7.
Therefore, combining the two lemmas, there exists point θ where both Assumption 1 and R ( θ ) > 0 hold.
Most practical datasets, such as CIFAR10 and ImageNet, are complex and cannot be ﬁt by linear models.
Thus, the conditions in Lemma 8 are reasonable in practice.
Proof.
Let W (cid:63) ∈
R
d y ×
d x is a local minimizer of 1 N (cid:80) Ni =1 l ( W x i , y i ) , where l is the loss function.
For training data that cannot be ﬁt by linear models, we have 1 N (cid:80) Ni =1 l ( W (cid:63) x
i , y i ) > 0 .
We use the same parameter settings as in the proof of Lemma 7, and then the training loss of CNN is expressed as R ( θ )
=
1 N (cid:80) Ni =1 l (cid:0) W fc W pool x i , y i (cid:1) .
Since 1 N (cid:80) Ni =1 l ( W (cid:63) x i , y i ) =
min
[Equation]
, we have R
[Equation]
θ ) > 0 .
The following Lemma 9 descibe the construction the point θ (cid:48) , and it is the formal version of Lemma 4 in section 3.2.2.
Lemma 9.
Given the CNN parameters θ :=
( W l , b l ) Ll =1 speciﬁed in Lemma 6, let θ (cid:48) :
= (cid:0) W (cid:48) l , b (cid:48) l (cid:1)
L l =1 .
If η ≥ − b L − 3, set
W
[Equation]
[Equation]
{+L (1 , 2 P L − 2 + 1 : 3 P L − 2 ) = 
+}
T ,
W (cid:48)
L (2 : d y , 1 : 2 P L − 2 ) = 0 ,
W (cid:48)
L − 1 ( i, i ) = − 1 , i ∈
[ P L − 3 ] ;
W (cid:48)
L − 1 ( i, i )
=
+1 , i ∈ { P L − 3 + 1 : 2 P L − 3 } , b (cid:48)
L − 1= η + b L − 3, b (cid:48)
L − 1= η + b L − 3,
W (cid:48)
L − 2 ( i, i ) = − 1 , i ∈
[ P L − 4 ] ;
W (cid:48)
L − 2 ( P L − 3 + i, i ) =
+1 ,
i ∈
[ P L − 4 ] , b (cid:48)
[Equation]
3= − min i
[Equation]
[ N ] ,
[Equation]
[ P L
[Equation]
4 ] v ji , b (cid:48)
[Equation]
− 2= η + b (cid:48)
[Equation]
3; b (cid:48)
[Equation]
2=
[Equation]
η
[Equation]
b (cid:48)
[Equation]
3; b (cid:48)
[Equation]
1 = b L
[Equation]
P
[Equation]
(cid:88)
j
[Equation]
[Equation]
(cid:0) η
[Equation]
b
[Equation]
(cid:1) .
If η
[Equation]
set
[Equation]
( θ ) .
The purpose of introducing b (cid:48)
L − 3is to make σ ( v ji + b (cid:48)
L − 3) ≥ 0 for all i ∈ [ N ] , j ∈ [ P L − 4 ] .
Next, we show that by perturbing θ (cid:48) , we can decrease the training loss.
We use different perturbation schemes for different cases of η .
The following Lemma 10 descibe the perturbation scheme, and it is the formal version of Lemma 5 in section 3.2.3.
Lemma 10.
If η
[Equation]
−
[Equation]
L
[Equation]
, perturb
[Equation]
as
[Equation]
If η < − b L − 3, perturb b (cid:48)
L − 2=
[Equation]
−
[Equation]
as
[Equation]
Remaining parameters in θ (cid:48) keep ﬁxed.
Then under Assumption 1, a training loss lower than R ( θ (cid:48) ) is obtained under this perturbation with a proper sign of δb .
D C ONSTRUCTION OF SPURIOUS L OCAL M INIMA : THE GENERAL CASE
When the two consecutive convolutional layers we utilized are not located at the top, and there are some convolutional, pooling, or fully connected layers between the two consecutive convolutional layers used in our construction and the ﬁnal output layer, we can still construct spurious local minima using settings similar to Lemmas 6, 9 and 10.
We let the ﬁrst fully connected layer play the role of layer L in Lemmas 6 and 9, and set the parameters of layers above the two consecutive convolutional layers such that the output of the two consecutive convolutional layers are propagated unchanged (except the pooling operations in subsequent pooling layers) to the ﬁrst fully connected layer, whose output is then forwarded invariantly to the ﬁnal output neurons.
Similar to the proof of Lemma 7, this is always possible by utilizing the minimal widths among subsequent convolutional layers and fully connected layers, respectively.
Then, by reserving the ﬁrst two feature maps in layers starting from the two consecutive convolutional layers and minimizing the loss of the remaining subnetwork, we can obtain a local minimum θ .
By setting the parameters connected to the two consecutive convolutional layers as in Lemma 9, we can obtain θ (cid:48) .
The positive value of σ ( v j i +
b
L − 3) may pass through different paths as before for different location j .
If there are average pooling layers above the two consecutive convolutional layers, since average pooling operation is linear, the average poolings in feature maps of different paths are ﬁnally aggregated in the ﬁrst fully connected layer, producing an output equal to that of θ .
Therefore, there is still R ( θ (cid:48) )
=
R ( θ ) .
The perturbation scheme is the same as in Lemma 10, and the perturbation will persist after passing through the linear average pooling operations, thus the empirical risk can be decreased by perturbing θ (cid:48) as before.
E M ORE A UXILIARY L EMMAS
Denote u i :=
∂l ( o ( x i ) , y i )
∂o
(1) and o i :=
o L,i (1) = (cid:80) Mj
{+=
+}
w j σ (cid:16) v ji + b L − 3 t j (cid:17) + b L 1 , where b L − 3 t j is the bias associated with the feature map in which location j resides. We have the following lemma for squared and cross-entropy loss.
Lemma 11.
For convolutional neural networks with squared loss or cross-entropy loss, the followings can be achieved through perturbing W L (1 , · ) under Assumption 1, where o (cid:48)
i and u (cid:48)
i denote, respectively, o i and u i after perturbation.
o (cid:48)
i (cid:54) =
o (cid:48)
j if o i
=
o j
( i, j ∈ [ N ] , i (cid:54) =
j ) .
u (cid:48)
i (cid:54) =
if u i =
( i ∈ [ N ]) .
u (cid:48)
i (cid:54) = u (cid:48)
j or u (cid:48)
i + u (cid:48)
j (cid:54) =
if u i = u j or u i + u j = 0 , respectively, ( i, j ∈ [ N ] , i (cid:54) =
j ) .
u
= ( u 1 , u 2 , · · · , u N ) T (cid:54)
=
if R = 1 N (cid:80) Ni =1 l ( o i , y i ) > 0 .
Nonzero gaps between o (cid:48)
i s or u (cid:48)
i s can still exist after subsequent perturbations.
The maintenance of nonzero gaps between o (cid:48)
i s or u (cid:48)
i s guarantees that o (cid:48)
i (cid:54) =
o (cid:48)
j or u (cid:48)
i (cid:54) = u (cid:48)
j etc.
still hold after subsequent perturbations.
In the following, we absorb biases into weights and the CNN output is written as o i =
W L I L − 1 ,i W L − 1 · · · I 1 ,i W 1 x i .
Here, we omit the hat on augmented variables for notational simplicity, and the exact meaning of involved symbols is clear from context.
The following two lemmas will be used to prove u (cid:48)
i (cid:54) = u (cid:48)
j or u (cid:48)
i + u (cid:48)
j (cid:54) =
( i, j ∈ [ N ] , i (cid:54) =
j ) in Lemma 11 for cross-entropy loss.
Lemma 12.
For cross-entropy loss, if u i = u
j for two samples x i , x j ( i (cid:54) =
j ) and
I L − 1 ,i W L − 1 · · · W 1 x i = αI L − 1 ,j W L − 1 · · · I 1 ,j W 1 x j ( α (cid:54) = 0 , α (cid:54) = 1) , then under Assumption 1, u (cid:48) i (cid:54) = u (cid:48)
j can be achieved through perturbing W L (1 , · ) .
Lemma 13.
For cross-entropy loss, if u i + u j = 0 for two samples x i , x j ( i (cid:54) =
j ) and
I L − 1 ,i W L − 1 · · · W 1 x i = αI L − 1 ,j W L − 1 · · · I 1 ,j W 1 x j ( α (cid:54) = 0 , α (cid:54) = 1) , then under Assumption 1, u (cid:48)
i + u (cid:48)
j (cid:54) = 0 can be achieved through perturbing W L (1 , · ) .
F E XPERIMENTAL D ETAILS
We use CIFAR-10 image set to train CNNs, which consists of 10 classes and 50000 training images of size 32 × 32 .
The CNN used in our experiments has 7 convolutional layers, with the number of channels being 64 , 64 , 128 , 128 , 256 , 256 , 256 , respectively, and no pooling layers are used.
The convolution ﬁlters are all 3 × 3 .
Each convolutional layer is followed by a ReLU layer.
The subnetwork in Lemma 6 is trained by Adam optimizer with 150 epochs, with a learning rate of 0.001 and weight decay of 0.0005.
The subnetwork W 2 in Theorem 3 is trained by Adam optimizer with 500 epochs, with a learning rate of 0.0001 and weight decay of 0.0005.
Both subnetworks use the Kaiming initialization (He et al.
The biggest threshold η in Lemma 2 is found to be 6 . 8915 .
Fig.3 shows the results of random direction approach.
G M ISSING P ROOFS
For the proofs in the following sections, we assume that the perturbation of network parameters for a differentiable local minimum θ = (cid:0) W l , b l (cid:1) L l =1 is very small such that the activation patterns
I l,i ( l ∈
[ L − 1] , i ∈ [ N ]) will keep constant. [Figure]
G. P ROOF OF L EMMA
Proof.
When parameters are perturbed as (cid:0) W l [Equation] b l (cid:1)
L l =1 −→ (cid:0) W l + δW l , b l + δ b l (cid:1) L l =
[Equation] the output after perturbation is as follows,
[Equation]
where δF i
[Equation]
The training loss after perturbation is
[Equation]
Since for every sample the output of each layer is nondegenerate, when minimizing
[Equation]
end
[Equation] [Equation] [Equation]
ˆ W
[Equation]
ˆ
[Equation]
1 ,i ˆ W L − 1 ˆ I L − 2 ,i · · · ˆ W 1 ˆ x i , y i (cid:17) in (24), the space of o ( x i ) has been fully explored by ˆ W L ˆ I L − 1 ,i ˆ W L − 1 · · · ˆ W 1 in the neighborhood during minimization.
Therefore,
R (( W l + δW l , b l + δ b l ) L l =1 ) cannot be lower than R (( W l , b l ) L l =1 ) despite the perturbation in δF i , and (cid:0) W l , b l (cid:1)
L l =1 is thus a local minimum in parameter space.
G. P ROOF OF L EMMA
Proof.
At a location j ∈ [ M ] (more speciﬁcally, M := min ( n L − 1 − 2 P L − 2 , n L − 2 − 2 P L − 3 , n L − 3 ) ), if there exist some groups in the ordered list of v ji ( i ∈ [ N ]) such that (cid:80) i ∈
I jq u
i (cid:54) = 0 , and suppose G 1 = (cid:110) v jm q +1 , v jm q +2 , · · · , v jm q +1 (cid:111) is the ﬁrst such group in the ordered list, we can set η = 1 2 (cid:16) v j m q +1 +1 + v j m q +1 (cid:17) , which is the midpoint of the gap between group G 1 and the next group, and set n = m q +1 .
We then have (cid:80) ni
=1 u i =
(cid:80)
m q
i =1 u i + (cid:80) i ∈ G 1 u i =
(cid:80) i ∈
G 1 u i (cid:54) = 0 .
The requirements in (7) are satisﬁed and we have found a split for data samples.
If for every group q ∈ [ g j ] at location j , there is (cid:80) i ∈
I jq u i = 0 , we have to explore other locations in feature maps to see whether there exist such splits.
According to Lemma 3, (cid:80) i ∈ I j q u i = cannot happen for every group q and every location j , and there must exist some locations where (cid:80)
[Equation]
u i (cid:54) = 0 for some groups.
As a result, we can split data samples as before at these locations.
There may be more than one location where such data split exists, and we choose the one with the biggest threshold η and use it in the perturbation of training loss.
Let h be the location having the biggest split threshold, we then have
[Equation]
G. P ROOF OF L EMMA
Proof.
There are two possible cases for η : η ≥ − b L − 3or η < − b L − 3, which will be treated differently in the following.
In order to compare the training losses at θ and θ (cid:48) , we need to compute the output o (cid:48)
( x i ) : = o (cid:48) L,i obtained by θ (cid:48) .
We will give the computation of its ﬁrst component o (cid:48) 1 ( x i ) in detail.
We only need to compute those terms in o (cid:48) 1 ( x i ) that are affected by the parameter settings in (25) and (26), and denote by ˜ o (cid:48)
( x i ) the sum of such terms.
If η ≥ − b L − 3, only the ﬁrst and the second feature maps in layers L − 1 and L − 2 are involved.
Let o (cid:48)
L − 3 ,i :
=
σ ( W L − 3 (1 : P L − 4 , · ) o L − 4 ,i + b (cid:48)
L − 31 P L − 4 ) , where 1 P L − 4 is the vector of size P L −whose components are all 1s.
o (cid:48)
L − 3 ,i is the output of the ﬁrst feature map at layer L − 3 using the new bias b (cid:48)
L − 3.
Since W L − 3 ( j, · ) o L − 4 ,i + b (cid:48)
L − 3≥ 0 for all i ∈ [ N ] , j ∈ [ P L − 4 ] , there is o [Equation]
,i = W L
[Equation] 3 (1 : P L [Equation]
4 , · ) o [Equation] 4 ,i + b
[Equation] 1 P L − 4 .
Then, the output ˜ o (cid:48)
( x i ) is
[Equation]
For notational simplicity, denote W (cid:48)
L (1 , i ) (
i ∈
[ P L − 2 ]) as w i , and note that v ji :=
W [Equation] j
[Equation] [ P [Equation]
Using (25), we have ˜ o [Equation] i
[Equation]
where
[Equation] [Equation]
We also denote
[Equation] [Equation]
and thus the partial output obtained from θ is
[Equation]
There are three possible cases for the value of v ji .
Note that we already have η + b L − 
 [Equation]
0 if η ≥ − b L − 3.
v ji ≥ η
In this case, we have v ji − η ≥ {+
+} and v ji + b L − 3≥ v ji − η ≥ 0 , then
[Equation]
) 2) − b L − 3≤
[Equation] < η
[Equation]
We have − v ji + η > 0 .
By (34) and v ji + b L − 3≥ 0 , we then have
[Equation]
) 3) v j i < − b L − [-
-]3.
In this case, we have v ji < η , v ji + b L − 3< 0 , then z (cid:48)
[Equation]
[-= 
 = z j , ∀ j ∈-]ji
[Equation] [Equation]
In all three possible cases, we have obtained z (cid:48)
j = z j , ∀ j ∈
[ P L − 2 ] .
Therefore, ˜ o (cid:48) ( x i ) =
(cid:80) P L − 2 j =1 w j · z j + b L 1 = ˜ o
( x i ) , ∀ i ∈
[ N ] .
We can obtain o k ( x i ) =
o (cid:48) k
( x i ) ( ∀ i ∈
[ N ] , ∀ k ∈ [ d y ] , k (cid:54)
= 1) for remaining output components since W (cid:48)
L (2 : d y , 1 : 2 P L − 2 ) = 0 .
The outputs are thus not changed, so does the training loss.
We now discuss the case of η < − b L − 3.
Denote w i :=
W L (1 , 2 P L − 2 + i ) , ∀ i ∈
[ P L − 2 ] .
The output ˜ o (cid:48) ( x i ) that is affected by the parameter setting in (26) is as follows, [Equation]
[Equation]
Using (26)
[Equation]
we have
[Equation]
where
[Equation]
There are also three possible cases for the value of v ji .
Note that we already have η + b L − 3< 0 when η < − b L − 3.
1) v
ji ≥ − b L −
In this case, using v ji − η > 0 , v ji + b L − 3≥ 0 and (42), we have
[Equation]
η ≤ v [-ji < − b L − 
-]j i
< − b L − 3. Using v ji − η ≥ 0 , v ji + b L − 3< 0 , we have
[Equation]
v
ji < η Using v ji − η < 0 , v ji + b L −
[Equation]
< 0 , we have
[Equation]
Therefore, in all possible cases of η and v ji , the output o (cid:48) 1 ( x i ) =
o 1 ( x i ) , ∀ i ∈
[ N ] .
We have o k ( x i ) =
o (cid:48) k
( x i ) ( ∀ i ∈
[ N ] , ∀ k ∈ [ d y ] , k (cid:54)
= 1) for remaining output components.
As a result, the training loss satisﬁes R ( θ ) =
R ( θ (cid:48) ) .
G. P ROOF OF L EMMA
Proof.
We will show that by perturbing θ (cid:48) , the training loss can be decreased.
We use different perturbation schemes to show this for different cases of η .
If η ≥ − b L − 3, we only perturb b (cid:48)
L − {+
+}
[Equation]
as [Equation] Under this perturbation, the output of each sample is perturbed as follows according to different cases of v ji .
1) v ji ≥ η .
In this case, after perturbation, by modifying (34) we have
[Equation]
Since η is at the midpoint of a gap between adjacent v ji s (as indicated in the proof of Lemma 2), there is v ji > η , hence for sufﬁciently small perturbation δb, v ji − η + δb > 0 and v ji + b L − [-
 [Equation]-]3+ δb >hold.
Therefore,
[Equation]
≤ [Equation]
Since v ji − η + δb < 0 , the perturbation δb does not pass through the ReLU activation, [Equation]
[Equation]
In this case, v ji − η + δb < 0 , then
[Equation]
Combining the above three cases, it is found that the output o (cid:48) 1 ( x i ) is perturbed only if v ji > η .
Correspondingly, the perturbation of output is
[Equation]
where I () is the indicator function.
Other components of output do not change under this perturbation since W (cid:48)
L (2 : d y , 1 : 2 P L − 2 ) = 0 .
The training loss is perturbed as
[Equation]
here o ( x i ) =
o (cid:48) ( x i ) is used to get ∂l ( o
[Equation]
i
[Equation]
∂o=
[Equation]
o
[Equation]
x
[Equation]
Therefore, we have
[Equation]
For all samples x i with v ji > η ( ∀ j ∈
[ P L − 2 ]) , by Lemma 2, only at the h th neuron in the ﬁrst feature map of layer L − 3 , (cid:80) ni =1 u i (cid:54) =
and v ji > η ( ∀ i ∈ [ n ]) both hold.
At other locations in this feature map, because η is the biggest split threshold among all locations, even if v ji > η holds, there is (cid:80) i ∈ { 1 , 2 , ··· ,N | v ji >η } u i =
(see (8)).
We then have
[Equation]
Setting Sgn ( δb ) =
− Sgn (cid:0) w h · (cid:80) ni
=1 u i (cid:1) if w h (cid:54) = 0 , where Sgn () is the sign function, we obtain δR [Equation]
[Equation]
Therefore, the training loss is decreased by the perturbation, resulting in
R
[Equation]
R [Equation] [Equation] Thus, θ is a spurious local minimum.
If w h = 0 , we perturb it as well by w h → δw , then
[Equation]
Setting Sgn ( δb ) =
− Sgn ( δw (cid:80) ni
=1 u i ) can still decrease the training loss, and thus the conclusion that θ is a spurious local minimum still holds.
If η < − b L − 3, we only pertur b (cid:48) L − 2= − η − b (cid:48) L − 3as b (cid:48)
[Equation]
With this perturbation, the output of each sample is perturbed in three different cases according to the value of v ji .
1) v
ji ≥ − b L − 3.
In this case, by modifying (42) we have
[Equation]
[Equation]
In this case, since a gap exists between η and v ji when splitting samples with η in Lemma 2, we have η < v ji and consequently v ji − η + δb > 0 for sufﬁciently small δb .
Therefore,
[Equation]
Using v ji − η + δb < [Equation] we have
[Equation]
The perturbation δb has been blocked by ReLU units.
Only in the ﬁrst and second cases, the output o (cid:48) 1 ( x i ) is affected by the perturbation δb .
Thus, we have
[Equation]
The output is perturbed as
[Equation]
Remaining output components keep unchanged.
The training loss is perturbed as follows,
[Equation]
Because η is the biggest split threshold among all locations, only for the h th location, there is (cid:80) i ∈{ 1 , 2 , ··· ,N | v ji >η } u i (cid:54)
= 0 .
We then have
[Equation]
which is the same as (52).
By setting δb with appropriate sign, we can obtain δR (cid:48) < 0 and conclude that θ is a spurious local minimum.
If a perturbation of W L (1 , · ) is required to split data samples as shown in Lemma 11, the training loss at θ is perturbed as
[Equation]
due to the optimality condition
[Equation]
since θ is a local minimum.
The perturbation δθ only modiﬁes W L (1 , · ) and does not change other parameters.
Therefore, by starting from θ + δθ and setting the parameters of the last three layers as in Lemma 9, we can obtain a point θ (cid:48)(cid:48) with
R ( θ + δθ ) =
R ( θ (cid:48)(cid:48) ) .
Thus we have R ( θ ) =
R ( θ (cid:48)(cid:48) ) .
We then perturb the biases of θ (cid:48)(cid:48) as in Lemma 10 to obtain a loss lower than R ( θ ) , which shows that θ is a spurious local minimum.
G. P ROOF OF L EMMA
Proof.
We prove by induction.
We will often perturb the network parameters to tune o i or u i ( i ∈ [ N ]) , which does not change empirical loss, as shown in (64).
Let us ﬁrst consider some special cases.
G.5. S PECIAL C ASES : N = 1 OR M =
If there is only one data sample, i.e., N = 1 , there will be only one group at each location.
Then, by
Lemma 11, we have u = ( u 1 ) (cid:54) = 0 , thus Lemma 3 holds.
We then consider the case of only one location: M = 1 .
If there are some groups with (cid:80) i ∈ I
q u
i (cid:54) = 0 ,
Lemma 3 already holds.
If there exists a group consisting of a single sample with u i = 0 , according to Lemma 11, we can perturb network parameters to get u (cid:48) i (cid:54) =
and thus Lemma 3 holds.
Otherwise, all groups have multiple samples and there are (cid:80) i ∈ I
q u i =
for all groups, and by R ( θ ) > 0 and consequently u (cid:54) = 0 , there must exist some group k such that there exists i ∈ I 1 k and u i (cid:54) = 0 .
All samples in group k have identical outputs, and by Lemma 11, we can separate this sample with u i (cid:54) {+= 
 from+} other samples in this group by perturbation, i.e., o (cid:48)
i (cid:54) =
o (cid:48)
j ( ∀ j ∈ I 1 k , j (cid:54) = i ) .
By doing so, we can obtain a isolated sample with u (cid:48)
i (cid:54) = 0 , and thus Lemma 3 is satisﬁed.
G.5. I NDUCTION H YPOTHESIS
Now we discuss the cases for N ≥ 2 and M > 1 , and use induction to prove.
In order to give our induction hypothesis, we ﬁrst give some deﬁnitions.
For multiple locations ( M > 1 ), there are usually a number of groups at each location.
Recall that I jq denotes the set of indices of samples in the q th group of the j th location.
We can select one group with (cid:80) i ∈
I jqj u i (the sum of u
i s in this group) from each location j and form a sequence (cid:80) i ∈
I 1 q 1 u i , (cid:80) i ∈ I
2 q 2 u i , · · · , (cid:80) i ∈
I MqM u i .
If all elements in this sequence are equal and nonzero, i.e., (cid:80) i ∈ I 1 q 1 u i =
(cid:80) i ∈ I
2 q 2 u i = · · · = (cid:80) i ∈
I MqM u
i (cid:54) = 0 , we call it a complete and identical sequence of groups ( CI sequence ).
If all elements in a sequence are nonzero but not equal, or partial elements in this sequence equal to zero, we call it a non-CI sequence .
Given a CI sequence, after adding a sample u k into the group q j at each location j , if (cid:80) i ∈ I 1 q 1 u i =
(cid:80) i ∈ I
2 q 2 u i = · · · = (cid:80) i ∈
I MqM u
i = − u k , the sequence becomes (cid:80) i ∈ I 1 q 1 u i = 0 , (cid:80) i ∈ I 2 q 2 u i = 0 , · · · , (cid:80) i ∈
I MqM u i =
(now the set I jq j ( j ∈ [ M ]) contains the new sample index k ).
Therefore, if for all groups and all locations, there exists only one CI sequence, and there are (cid:80) i ∈
I jqj u i = 0 for all groups not in this CI sequence, then it is possible that for all j ∈ [ M ] and q ∈ [ g j ] , (cid:80) i ∈
I jq u i =after adding a new sample that can annihilate this CI sequence, thus Lemma 3 is violated.
In order to prove Lemma 3, it sufﬁces to prove that for every possible group conﬁguration groups with (cid:80) i ∈
I jq u
i (cid:54) = 0 exist after removing CI sequences.
Sometimes only CI sequences exist, like ( u i ) − ( u i ) − · · · − ( u i ) ( i ∈ [ N ]) , and when adding new samples, there is a risk of annihilating
such
CI sequences.
We can avoid such situations using perturbations of network parameters (this will be explained in sections G.5.3 and G.5.4).
Consequently, we give the following induction hypothesis.
For multiple locations, either non-CI sequences exist after removing each CI sequence (setting each element (cid:80) i ∈
I jqj
u i in it to zero), or the removal of CI sequences can be prevented.
By this induction hypothesis, after adding a new sample x m +1 into one existing group at each location, groups with (cid:80) i ∈
I jqj u
i (cid:54) = 0 still exist, and Lemma 3 holds accordingly.
As base cases, we will prove that this induction hypothesis holds for N = 2 .
We then prove that if it holds for N = m ( m ≥ 2) , it will also hold when adding a new sample x m +1 , as induction step.
G.5. B ASE C ASES : N =
M = 2 .
First, we discuss the cases of two locations.
When N = 2 , all possible group conﬁgurations are discussed in the following and we will prove that induction hypothesis holds for each case.
CASE 1.
Suppose the groups at two locations are
[Equation]
If u 1 + u 2 (cid:54) = 0 , then the sequence ( u 1 , u 2 ) − ( u 1 , u 2 ) (the ﬁrst element in the sequence is ( u 1 , u 2 ) from location 1, and the second one is ( u 1 , u 2 ) from location 2) is a CI sequence.
However, this
CI sequence cannot be vanished by adding a new sample since we can use perturbation of network parameters to prevent it from happening, described later in the induction step.
If u 1 + u 2 = 0 , by Lemma 11, we can perturb network parameters to obtain u (cid:48) 1 + u [Equation] 2 = 0 at ﬁrst.
CASE 2.
Suppose the groups at two locations are
[Equation]
where groups on the left have bigger v ji s than those on the right at each location.
By Lemma 11, we can perturb network parameters to obtain u (cid:48) [-
-]1 (cid:54)
= {+
+} and u (cid:48) [-
-]2 (cid:54) = 0 for squared loss and cross-entropy loss, then the sequences ( u (cid:48) 1 , u (cid:48) 2 ) − ( u (cid:48) 1 ) and ( u (cid:48) 1 , u (cid:48) 2 ) − ( u (cid:48) 2 ) are both non-CI sequences, thus the induction hypothesis holds.
If ( u 1 , u 2 ) in location 1 is separated as ( u (cid:48) 1 ) , ( u (cid:48) 2 ) after perturbation, then it can be treated by the following case 3.
CASE 3.
Suppose the groups at two locations are
[Equation]
or location 1 : [Equation] location 2 :
[Equation]
u 2 ) , ( u 1 ) . For these cases, by Lemma 11, if u 1 = u 2 , we can perturb the parameters to obtain u (cid:48) [-
-]1 (cid:54) =
u (cid:48) 2 for squared loss and cross-entropy loss, then the sequences ( u (cid:48) 1 ) − ( u (cid:48) 2 ) and ( u (cid:48) 2 ) − ( u (cid:48) 1 ) are both non-CI sequences.
The sequences ( u 1 ) − ( u 1 ) and ( u 2 ) − ( u 2 ) are CI sequences.
Again, by appropriate perturbations when adding new samples in the induction step, such CI sequences will not be removed and thus
Lemma 3 holds.
M ≥ 3 .
If there are more than two locations, we can select two locations as a subset, which must belong to one of the above three possible cases.
The full sequences of all locations contain those sequences in this two-location subset as sub-sequences, and consequently groups with (cid:80) i ∈
I jqj u
i (cid:54) =still exist in full sequences.
Therefore, induction hypothesis holds for M ≥ 3 .
G.5. I NDUCTION S TEP
At last, we prove that if the induction hypothesis holds for N = m samples, then it holds for
N = m + 1 samples.
Suppose sample x m +1 is to be added given current arrangement of groups.
Note that we can have u m +1 (cid:54) =
according to Lemma 11.
There are the following three possible situations.
1) x m +1 is added as a new group at every location.
This is equivalent to adding a new CI sequence.
CI and non-CI sequences formed by previous groups still exist, thus induction hypothesis holds.
2) x m +1 is inserted into an existing group at every location.
More speciﬁcally, this can be divided into the following four cases.
A ) x m +1 is inserted into an existing group satisfying (cid:80) i ∈
I jq u i =
at every location.
This is equivalent to adding a new CI sequence into existing groups.
Since CI or non-CI sequences still exist, the induction hypothesis holds
B ) x m +1 is inserted into existing CI or non-CI sequences.
If inserting x m +1 into an existing CI sequence, it will either be a new CI sequence, or annihilated by this addition.
By induction hypothesis, non-CI sequences may still exist, hence induction hypothesis holds again after inserting x m +1 .
Another possible situation is that CI sequences exist, such as ( u i ) − ( u i ) − · · · − ( u i ) ( i ∈
[ N ]) , and when adding new samples, there is a risk of annihilating such CI sequences, and if no remaining non-CI sequences exist, then no groups with (cid:80) i ∈
I jq u
i (cid:54) =exist.
We can perturb the outputs of samples to avoid such situation using a perturbation of network parameters.
For example, if u 1 + u 3 = 0 , then adding u 3 into the sequence ( u 1 ) − ( u 1 ) − · · · − ( u 1 ) will cause its disappearance.
We will have o 1 =
o 3 since sample 1 and sample 3 always appear in the same group at every location after the addition.
After perturbation that tunes o
and o 3 differently to get o (cid:48) 1 (cid:54) =
o (cid:48) 3 , as shown in Lemma 11, sample 1 and sample 3 then cannot appear in the same group at every location.
The annihilation of sequence ( u 1 ) − ( u 1 ) − · · · − ( u 1 ) is thus avoided and induction hypothesis holds.
If inserting x m +1 into an existing non-CI sequence, it will still be a non-CI sequence by deﬁnition.
Therefore, induction hypothesis holds.
C ) x m +1 is inserted into an existing CI sequence at some locations and into an existing non-CI sequence at remaining locations.
The involved CI sequence becomes a non -CI one, and the involved non-CI sequence may become a non-CI sequence, a CI sequence or a zero sequence.
Anyway, non-CI sequences still exist and cannot be removed by adding a sample, and thus induction hypothesis holds.
D ) x m +1 is inserted into an existing CI or non-CI sequence at some locations and into groups with (cid:80) i ∈
I jq u i = 0 at remaining locations.
In this case, non-CI sequences will exist after inserting x m +1 , thus induction hypothesis holds.
3) x m +1 is inserted into existing groups at some locations j ∈ J 1 , and added as new groups at remaining locations j ∈ J 2 .
Equivalently, we can regard new groups in J 2 (before adding x m +1 ) as existing groups with (cid:80) i ∈
I jq u i = 0 , thus the above case D) can be applied directly, and induction hypothesis is maintained
G.5. I NDUCTION C ONCLUSION
We have proved that if the induction hypothesis holds for N = m samples, then it holds for N = m + samples.
By induction hypothesis, there exists j ∈
[ M ] , q ∈ [ g j ] , (cid:80) i ∈
I jq u
i (cid:54) = 0 , and consequently a split of data samples somewhere in feature maps is always possible.
G. P ROOF OF L EMMA
Proof.
For cross-entropy loss, the training loss for a sample ( x i , y i ) is
[Equation]
where o i,j is the j th component of o i .
Its derivative is
[Equation] [Equation]
Let p i =
e − oi, 1 (cid:80) dyk =1 e − oi,k , we have u i =
p i − y i
[Equation]
[Equation]
Denote w T :=
W L (1 , · ) , o :
= o i, 1 .
Under the perturbation δ w , by Taylor expansion we have
[Equation]
[Equation]
Note that for o L − 1 ,i =
I
L − 1 ,i W L − 1 · · · I 1 ,i W 1 x i , we have o L − 1 ,i (1 : 2 P L − 2 )
=
( ∀ i ∈
[ N ]) due to the parameter setting in Lemma 6, thus perturbation δ w (1 : 2 P L − 2 ) has no effect in (72).
For cross-entropy loss, the ﬁrst and second order derivatives of u i are
[Equation]
respectively.
Let Q i := I L − 1 ,i W L − 1 · · · I 1 ,i W 1 , we now prove that if Q i x i = αQ
j x j ( α (cid:54) = 0 , α (cid:54) = 1) , then by setting δ w appropriately, one can always make u (cid:48)
i (cid:54) = u (cid:48)
j .
Note that ∂u
i ∂o
=
p
i − p
i (cid:54) = since p i (cid:54) = 0 and p i (cid:54) = 1 for ﬁnite inputs.
Also note the requirement I L − 1 ,i (cid:54) = 0 in Lemma 6, we have Q i x i (cid:54) = 0 , i ∈ [ N ] .
We can set δ w = (cid:15) Q i x i (cid:107) Q i x i (cid:107) such that ( Q i x i ) · δ w (cid:54) = 0 and ( Q j x j ) · δ w (cid:54) = 0 , where (cid:15) is a sufﬁciently small positive number.
By (72), if ∂u i ∂o (cid:54) = 1 α ∂u j ∂o , then ∂u
i ∂o
( Q i x i ) · δ w (cid:54) =
∂u j ∂o ( Q j x j ) · δ w , hence we have u (cid:48)
i (cid:54) = u (cid:48)
j up to the ﬁrst-order Taylor expansion.
If Q i x i = αQ j x j and ∂u i ∂o
=
α ∂u j ∂o , we need to consider the second-order term in (72), and if ∂ 2 u i ∂o 2 α 2 (cid:54) =
∂ 2 u j ∂o 2 , we have u (cid:48)
i (cid:54) = u (cid:48)
j up to the second-order Taylor approximation.
Otherwise, we have ∂u i ∂o
=
α ∂u j ∂o and ∂ 2 u
i ∂o 2 α 2 = ∂ 2 u j ∂o 2 , which result in
[Equation]
and
[Equation]
respectively.
We next show that (75) and (76) cannot both hold for each case of y i, 1 ∈ { 0 , 1 } , and thus we have u (cid:48)
i (cid:54) = u (cid:48)
j up to either the ﬁrst-order or the second-order Taylor approximation.
If u i = p i , u j = p j or u i = p i − 1 , u j = p
j − 1 , then u i = u
j implies p i = p j .
From (75) and α (cid:54) = 1 , contradiction is resulted and hence u (cid:48)
i (cid:54) = u (cid:48)
j .
If samples x i and x j have distinct labels, say without loss of generality, u i = p
i − 1 , u j = p j , then u i = u
j leads to p j = p i − 1 .
Substituting it into (75) and (76), we obtain
[Equation]
) and
[Equation]
respectively.
(77) and (78) can be transformed into
[Equation]
and
[Equation]
respectively.
By 1 − p i (cid:54) = 0 , (79) results in
[Equation]
Solving (80) and (81), we obtain α = − 1 , p i = 1 .
However, p i = 1 is impossible for ﬁnite inputs, therefore, we have u (cid:48)
i (cid:54) = u (cid:48)
j after the perturbation.
G. P ROOF OF L EMMA
Proof.
We now prove that if u i + u j = 0 and Q i x i = αQ
j x j ( α (cid:54) = 0 , α (cid:54) = 1) , then by setting δ w appropriately, one can always obtain u (cid:48)
i + u (cid:48)
j (cid:54) = 0 .
By (72), if ∂u i ∂o (cid:54) = − 1 α ∂u j ∂o , then ∂u
i ∂o ( Q i x i ) · δ w (cid:54) = − ∂u j ∂o ( Q j x j ) · δ w , hence u (cid:48)
i + u (cid:48)
j (cid:54) =
up to the ﬁrst-order Taylor expansion.
If Q i x i = αQ j x j and ∂u
i ∂o = − 1 α ∂u j ∂o , we need to consider the second-order term in (72), and if ∂ 2 u i ∂o 2 α 2 (cid:54) = − ∂ 2 u j ∂o 2 , we have u (cid:48)
i + u (cid:48)
j (cid:54) =
up to the second-order Taylor approximation.
Otherwise, we have ∂u i ∂o = − 1 α ∂u j ∂o and ∂ 2 u i ∂o
α 2 = − ∂ 2 u j ∂o 2 , which result in − ( p [Equation] = p [Equation]
− [Equation] j [Equation]
82) and [Equation] p j ) = 0 ,
[Equation] [Equation]
respectively.
We next show that (82) and (83) cannot both hold for every case of y i, 1 ∈ { 0 , 1 } .
If u i = p i , u j = p j , then u i + u j = 0 implies p j = − p
i .
Substituting it into (82) and (83) and with simple calculations, we can obtain
[Equation]
and
[Equation]
The solution is α = 1 , p i = 0 , which contradict α (cid:54) = 1 , p i (cid:54) = 0 .
If u i = p i − 1 , u j = p
j − 1 , then u i + u j = 0 implies p j = 2 − p i .
In this case, the solution to (82) and (83) is α = 1 , p i
= 1 , again contradiction is resulted.
If samples x i and x j have distinct labels, say without loss of generality, u i = p
i − 1 , u j = p j , then u i + u j = 0 leads to p j = 1 − p
i .
The solution to (82) and (83) will be α = − 1 and p i can be arbitrary.
However, α = − 1 implies Q i x i = − Q j x j .
Due to I L − 1 ,i (cid:54) = 0 by the requirement in Lemma 6, we have for all i ∈
[ N ] that all components of Q i x
i are nonnegative and some components of it must be positive, and thus Q i x i = − Q j x j leads to contradiction.
Therefore, α = − 1 is impossible.
In summary, for various cases of y i, 1 ∈ { 0 , 1 } , we all have u (cid:48)
i + u (cid:48)
j (cid:54) = 0 .
G. P ROOF OF L EMMA
Proof.
o (cid:48)
i (cid:54) =
o (cid:48)
j ( i, j ∈ [ N ] , i (cid:54) =
j ) for squared loss and cross-entropy loss.
We perturb W L (1 , · ) to make o (cid:48)
i (cid:54) =
o (cid:48)
j .
Under this perturbation, we have
[Equation]
Since o L − 1 ,i (cid:54) =
o L − 1 ,j by Assumption 1, we have Q i x i (cid:54) =
Q j x j .
Then, if o i
=
o j , we have o (cid:48)
i − o (cid:48)
j = ( Q i x i − Q j x j ) · δ w .
Let δ w =
(cid:15) ( Q i x i − Q j x j ) , where (cid:15) is a sufﬁciently small positive number, there is o (cid:48)
i − o (cid:48)
j = (cid:15) (cid:13)(cid:13) Q i x i − Q j x j (cid:13)(cid:13) 2 (cid:54) = 0 .
Thus, o (cid:48) i (cid:54) =
o (cid:48)
j .
2. Proof of u
[Equation] i [Equation]
[Equation]
[Equation] For squared loss, we perturb W L (1 , · ) to make u (cid:48) i (cid:54)[-= 
-]
{+= 
+} if u i = 0 .
With this perturbation, we have
[Equation]
We have u i
=
o i − y
i, 1 for squared loss l
( o i , y i )
= 12 (cid:107) o
i − y
i (cid:107) 2 , then
[Equation]
Setting δ w = (cid:15) ( Q i x i ) results in u (cid:48)
i = (cid:15) (cid:13)(cid:13)
Q i x i (cid:13)(cid:13) 2 .
By Q i x i (cid:54) =
due to the requirement I L − 1 ,i (cid:54) =in Lemma 6, we have u (cid:48)
i (cid:54) = 0 .
For cross-entropy loss function, the derivative u i is given in (70).
For ﬁnite inputs and y i, 1 ∈ { 0 , 1 } , we have [Equation] and
[Equation]
, thus u
[Equation]
3. Proof of u
[Equation]
[Equation]
j
[Equation] u [Equation]
i [Equation] For squared loss, we perturb W L (1 , · ) to make u (cid:48) i (cid:54) = u (cid:48)
j or u (cid:48)
i + u (cid:48)
j (cid:54) = 0 .
Under this perturbation, if u i = u j , by (87) we have u (cid:48)
i − u (cid:48)
j = ( Q i x i − Q j x j ) · δ w .
Let δ w =
(cid:15) ( Q i x i − Q j x j ) , there is u (cid:48)
i − u (cid:48)
j = (cid:15) (cid:13)(cid:13) Q i x i − Q j x j (cid:13)(cid:13) 2 (cid:54) = 0 .
Therefore, u (cid:48) i (cid:54) = u (cid:48)
j ( i, j ∈ [ N ] , i (cid:54) = j ) .
If u i + u j = 0 , we have u (cid:48)
i + u (cid:48)
j = ( Q i x i + Q j x j ) · δ w .
Note that Q
i x i (cid:54) =
− Q j x j due to
I L − 1 ,i (cid:54) =
and I L − 1 ,j (cid:54) = 0 by the requirement in Lemma 6.
Let δ w =
(cid:15) ( Q i x i + Q j x j ) , we then have u (cid:48)
i + u (cid:48)
j = (cid:15) (cid:13)(cid:13) Q i x i + Q j x j (cid:13)(cid:13) 2 (cid:54) = 0 .
Therefore, u (cid:48) i + u (cid:48)
j (cid:54) =
( i, j ∈ [ N ] , i (cid:54) = j ) .
For cross-entropy loss function, recall that w T :=
W L (1 , · ) and Q i :=
I L − 1 ,i W L − 1 · · · I 1 ,i W 1 .
Under perturbation δ w , by the ﬁrst-order Taylor expansion in (72), using (73) we have u (cid:48)
[Equation]
[Equation]
Note that Q i x i (cid:54) = 0 , Q j x j (cid:54) = 0 , and p i − p
i (cid:54) = 0 since p i (cid:54) = 0 and p i (cid:54) = 1 .
If Q i x i is not parallel to Q j x j , then ( p i − p 2 i )( Q i x i ) ± ( p j − p 2 j )( Q j x j ) (cid:54) = 0 .
Therefore, we have u (cid:48)
i (cid:54) = u (cid:48)
j or u (cid:48)
i + u (cid:48)
j (cid:54) =
( i, j ∈ [ N ] , i (cid:54) = j ) , respectively.
If Q i x i is parallel to Q j x j , i.e., Q i x i = αQ
j x j ( α (cid:54) = 0)) , we can obtain u (cid:48)
i (cid:54) = u (cid:48)
j or u (cid:48)
i + u (cid:48)
j (cid:54) =
( i, j ∈ [ N ] , i (cid:54) = j ) by Lemma 12 or Lemma 13, respectively.
α = 1 is not allowed by Assumption 1. u =
( u 1 , u 2 , · · · , u N ) T (cid:54)
{+= 
 if R = 1 N (cid:80) Ni =1 l ( o i y i >+}
For squared loss, u i
=
o i − y
i, 1 .
If o i − y i = 0 for every sample x i ( i ∈ [ N ]) , then the empirical loss R = 1 N (cid:80) N i =1 l ( o i , y i ) = 1 2 N (cid:80) N i =1 (cid:107) o
i − y
i (cid:107) 2 = 0 , contradicting the assumption R > 0 .
Therefore, there must be some nonzero components in vector ( o i − y i ) for some i ∈ [ N ] .
Without loss of generality, let some components in ( o 1 − y 1 , 1 , o 2 − y 2 , 1 , · · · , o N − y N, 1 ) be nonzero, hence u (cid:54)
= 0 .
For cross-entropy loss, according to (70), u i (cid:54) = {+
+}
( ∀ i ∈ [ N ]) for ﬁnite inputs, thus u = ( u 1 , u 2 , · · · , u N ) T (cid:54) = 0 .
Nonzero gaps between o (cid:48)
i s or u (cid:48) i
s can still exist after subsequent perturbations, for squared loss and cross-entropy loss.
We discuss the gaps between o (cid:48)
i s in detail.
Given some samples with identical outputs, say without loss of generality, o 1 =
o 2 = · · · = o s , by Lemma 11 we can use perturbation of parameters δ w T :=
δW L (1 , · ) to obtain o (cid:48) 1 (cid:54) =
o (cid:48) 2 , thus a gap exists between o (cid:48) 1 and o (cid:48) 2 .
Without loss of generality, suppose after the ﬁrst perturbation, the outputs of samples are ordered as o (cid:48)
1 > o (cid:48) 2 > · · · > o (cid:48)
n = o (cid:48) n +1 = · · · = o (cid:48) n + m > o (cid:48) n + m +1 > · · · > o (cid:48) s .
We now want to perturb the outputs of samples again to avoid o (cid:48)
n = o (cid:48)
n +1 using δ w (cid:48) .
Under the second perturbation δ w (cid:48) , the output of each sample is o (cid:48)(cid:48)
i = ( w (cid:48)
+ δ w (cid:48) )
· Q i x i := o (cid:48)
i + δ w (cid:48) ·
Q i x i .
The gaps between samples are changed to o (cid:48)(cid:48)
i − o (cid:48)(cid:48)
i +
=
o (cid:48)
i − o (cid:48)
i +
+ δ w (cid:48) ·
( Q i x i −
Q i +1 x
i +1 )
( i ∈ [ s − 1]) .
We set δ w (cid:48) small enough as follows to make sure the nonzero gaps in o (cid:48)
i − o (cid:48)
i +1 ( i ∈
[ s − 1]) do not disappear.
For the gap o (cid:48)(cid:48)
i − o (cid:48)(cid:48)
i +1 , we have o (cid:48)(cid:48)
i
[Equation]
[Equation] − 1]) .
We already have o (cid:48)
i − o (cid:48)
i +1 > 0 for nonzero gaps, and if δ w (cid:48) are small enough
such that
[Equation]
90) or equivalently
[Equation]
then we will have ∀ i ∈
[ s − 1] , o (cid:48)(cid:48)
i − o (cid:48)(cid:48)
i +1 > 0 .
Note that (cid:13)(cid:13)
Q
i x i − Q
i +1 x
i +1 (cid:13)(cid:13) (cid:54) = 0 .
The nonzero gaps between neighboring samples still exist.
Additional perturbations can be treated in the same spirit.
For a dataset with N samples, at most N perturbations are needed.
The total perturbation is obtained by adding the perturbations in all steps, i.e., δ w + δ w (cid:48) + · · · .
The gaps between u (cid:48)
i s can be treated similarly using small enough perturbations.
G. P ROOF OF T HEOREM
Proof.
The parameter matrix W 1 (1 : P 0 , 1 : P 0 ) of the ﬁrst convolutional layer can be implemented by a ﬁlter like w 11 = (0 , 0 , 1 , 0 , 0) (for s 0 = 5 ).
With the parameter setting in Theorem 3, the output of the k th neuron in the ﬁrst convolution layer is
[Equation]
Then, we have
[Equation]
93) due to (10).
The outputs of remaining neurons in the ﬁrst feature map of the ﬁrst convolutional layer do not matter since they will be annihilated later by W L .
Given x ki + c p < 0 ( ∀ i / ∈ I ) , we have
[Equation]
The ﬁlters and biases of higher convolutional layers ( l ∈ { 2 , 3 , · · · , L − 1 } ) for the ﬁrst predictor propagate o l (1 : P l − 1 ) (the outputs in the ﬁrst feature map of each layer) forward without changing them.
Sufﬁciently large positive constants
c l ( l ∈
[ L − 1]) are added to the biases of each layer such that the ReLU units in the second subnetwork are all activated and their outputs are not truncated.
The ﬁrst convolutional layer will be added c 1 to each of its output neuron, and the change of outputs of higher convolutional layers can be recursively computed as
[Equation]
The effect of introducing c l ( l ∈ [ L − 1]) is cancelled at the output layer by subtracting ∆ o L ( i ) from o L ( i ) ( i ∈ [ d y ]) .
This is equivalent to subtracting ∆ o L ( i ) from b Li .
By the parameter setting, the output of each sample is
[Equation]
where the expression of W 2 ˆ x i is given in (20).
This implies
[Equation]
and
[Equation]
if there is a single index in set I , where ˜ y
i is deﬁned in section 4.
If there are multiple elements in set
I ,
[Equation]
is minimized.
For samples { x i , i / ∈ I } , we have
[Equation]
We now show that for the CNN given in Theorem 3, θ 1 = (cid:0) W l , b l (cid:1)
L l =1 is a local minimum.
This can be seen by perturbing the parameters and showing the non-decreaseness of loss.
The training loss is
[Equation]
Under perturbation (cid:0) δW l , δ b l (cid:1) L l =1 , since there is a gap between x ki ( i ∈ [ N ]) and the threshold 12 (cid:0) x ki ∗ + x kj ∗ (cid:1) , the ReLU unit in (92) keeps its activation status for every sample when the perturbation of b
=
c p is sufﬁciently small.
Therefore, the output of each sample is still in the form of (96).
Accordingly, if there are multiple elements in set I ,
[Equation]
where δ o ( x i ) ( i ∈
I ) and δ o ( x i ) ( i / ∈
I ) are caused by the perturbations of parameters in both W 1 and W 2 .
However, since (cid:0) λ 1 , λ 2 , · · · , λ d y (cid:1) minimize (cid:80) i ∈ I l (cid:16) W
ˆ x i + (cid:0) x ki + c p (cid:1) (cid:0) λ 1 , λ 2 , · · · , λ d y (cid:1) T , y i (cid:17) , and the space of o ( x i ) has been fully explored by (cid:0) λ 1 , λ 2 , · · · , λ d y (cid:1) during minimization, we have
[Equation]
On the other hand, since the size of each convolutional layer satisﬁes n l − P l − 1 ≥ d y ( l ∈
[ L − 1]) , the matrix ˆ W L ( · , P L − 2 + 1 : n L − 1 ) ˆ W L − 1 ( P L − 2 + 1 : n L − 1 , P L − 3 + 1 : n L − 2 ) · · · ˆ W 1 ( P 0 + 1 : n 1 , · ) ∈ R d y ×
d x is full rank and can generate the full space of d y × d x matrices around θ , thus (cid:80) i ∈
[ N ] , i/ ∈
I l ( W 2 ˆ x i , y i ) is minimized by W 2 in (20).
Therefore,
[Equation]
Using (102) and (103),
[Equation]
For the case of single element in set I , R (cid:48) ≥ R can be shown easily due to (98).
Therefore, θ 1 = (cid:0) W l , b l (cid:1)
L l =1 is a local minimum.