T HE LOSS SURFACE AND EXPRESSIVITY OF DEEP CONVOLUTIONAL NEURAL NETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We analyze the expressiveness and loss surface of practical deep convolutional neural networks (CNNs) with shared weights and max pooling layers.
We show that such CNNs produce linearly independent features at a “wide” layer which has more neurons than the number of training samples.
This condition holds e g. for the VGG network.
Furthermore, we provide for such wide CNNs necessary and sufﬁcient conditions for global minima with zero training error.
For the case where the wide layer is followed by a fully connected layer we show that almost every critical point of the empirical loss is a global minimum with zero training error.
Our analysis suggests that both depth and width are very important in deep learning.
While depth brings more representational power and allows the network to learn high level features, width smoothes the optimization landscape of the loss function in the sense that a sufﬁciently wide network has a well-behaved loss surface with almost no bad local minima.
I NTRODUCTION
It is well known that the optimization problem for training neural networks can have exponentially many local minima (Auer et al., 1996; Safran & Shamir, 2016) and NP-hardness has been shown in many cases (Blum & Rivest., 1989; Sima, 2002;
Livni et al., 2014; Shamir, 2017; Shalev-Shwartz et al., 2017).
However, it has been empirically observed (Dauphin et al., 2014; Goodfellow et al., 2015) that the training of state-of-the-art deep CNNs (LeCun et al., 1990; Krizhevsky et al., 2012), which are often overparameterized, is not hampered by suboptimal local minima.
In order to explain the apparent gap between hardness results and practical performance, many interesting theoretical results have been recently developed (Andoni et al., 2014; Sedghi & Anandkumar, 2015; Janzamin et al., 2016; Haeffele & Vidal, 2015; Gautier et al., 2016; Brutzkus & Globerson, 2017; Soltanolkotabi, 2017; Soudry & Hoffer, 2017; Goel & Klivans, 2017; Du et al., 2017; Zhong et al., 2017; Tian, 2017; Li & Yuan, 2017) in order to identify conditions under which one can guarantee that local search algorithms like gradient descent converge to the globally optimal solution.
However, it turns out that these approaches are either not practical as they require e.g. knowledge about the data generating measure, or a modiﬁcation of network structure and objective, or they are for quite restricted network structures, mostly one hidden layer networks, and thus are not able to explain the success of deep networks in general.
For deep linear networks one has achieved a quite complete picture of the loss surface as it has been shown that every local minimum is a global minimum (Baldi & Hornik, 1988; Kawaguchi, 2016; Freeman & Bruna, 2017; Hardt & Ma, 2017;
Yun et al., 2017).
By randomizing the nonlinear part of a feedforward network with ReLU activation function and making some additional simplifying assumptions, Choromanska et al.
(2015a) can relate the loss surface of neural networks to a certain spin glass model.
In this model the objective of local minima is close to the global optimum and the number of bad local minima decreases quickly with the distance to the global optimum.
This is a very interesting result but is based on a number of unrealistic assumptions (Choromanska et al., 2015b).
More recently, Nguyen & Hein (2017) have analyzed deep fully connected networks with general activation functions and could show that almost every critical point is a global minimum if one layer has more neurons than the number of training points.
While this result holds for networks in practice, it requires a quite extensively overparameterized network.
In this paper we overcome the restriction of previous work in several ways.
This paper is one of the ﬁrst ones, which theoretically analyzes deep CNNs.
CNNs are of high practical interest as they learn very useful representations (Zeiler & Fergus, 2014; Mahendran & Vedaldi, 2015; Yosinski et al., 2015) with a small number of parameters.
We are only aware of Cohen & Shashua (2016) who study the expressiveness of CNNs with max-pooling layer and ReLU activation but with rather unrealistic ﬁlters (just 1 × 1 ) and no shared weights.
In our setting we allow as well max pooling and general activation functions.
Moreover, we can have an arbitrary number of ﬁlters and we study general convolutions as the ﬁlters need not be applied to regular structures like 3 × 3 but can be patch-based where the only condition is that all the patches have the size of the ﬁlter.
Convolutional layers, fully connected layers and max-pooling layers can be combined in almost arbitrary order.
We study in this paper the expressiveness and loss surface of such a CNN where one layer is wide, in the sense that it has more neurons than the number of training points.
While this assumption sounds at ﬁrst quite strong, we want to emphasize that the VGG network (Simonyan & Zisserman, 2015) and other CNNs, see Table 1, fulﬁll this condition.
We show that such wide CNNs produce linearly independent feature representations at the wide layer and thus are able to ﬁt the training data exactly (universal ﬁnite sample expressivity).
This is even true with probability one when all the parameters of the bottom layers (from input to the wide layer) are chosen randomly 1 .
We think that this explains partially the results of Zhang et al.
(2017) where they show experimentally for several CNNs that they are able to ﬁt random labels.
Moreover, we provide necessary and sufﬁcient conditions for global minima with zero squared loss and show for a particular class of CNNs that almost all critical points are globally optimal, which to some extent explains why such wide CNNs can be optimized so efﬁciently.
[Equation]
ONVOLUTIONAL N EURAL N ETWORKS
We ﬁrst introduce our notation and deﬁnition of CNNs.
Let N be the number of training samples and denote by X
=
[ x 1 , . .
, x N ] T ∈ R N × d , Y =
[ y 1 , . . .
, y N ] T ∈ R N × m the input resp.
output matrix for the training data ( x i , y i )
Ni =1 , where d is the input dimension and m the number of classes.
Let L be the number of layers of the network, where each layer is either a convolutional, maxpooling or fully connected layer.
The layers are indexed from k = 0 , 1 , . . .
, L which corresponds to input layer, 1st hidden layer, . . .
, and output layer.
Let n k be the width of layer k and f k : R d → R n k the function that computes for every input its feature vector at layer k .
The convolutional layer consists of a set of patches of equal length where every patch is a subset of neurons from the same layer.
Throughout this paper, we assume that the patches of every layer cover the whole layer, i.e. every neuron belongs to at least one of the patches, and that there are no patches that contain exactly the same subset of neurons.
This means that if one patch covers the whole layer then it must be the only patch of the layer.
Let P k and l k be the number of patches resp.
the size of each patch at layer k for every 0 ≤ k < L.
For every input x ∈ R d , let (cid:8) x 1 , . . .
, x P 0 (cid:9)
∈
R ldenote the set of patches at the input layer and (cid:110)
f 1 k ( x ) , . . .
f P k k ( x ) (cid:111) ∈
R l k the set of patches at layer k .
Each ﬁlter of the layer consists of the same set of patches.
We denote by T k
the number of convolutional ﬁlters and by W k =
[ w 1 k , . .
w T k k ] ∈ R l k − 1 × T k the corresponding parameter matrix of the convolutional layer k for every 1 ≤ k < L .
Each column of W k corresponds to one ﬁlter.
Furthermore, b k ∈ R n k denotes the bias vector and σ k : R → R the activation function for each layer.
Note that one can use the same activation function for all layers
but we use the general form to highlight the role of different layers.
In this paper, all functions are applied componentwise, and we denote by [ a ] the set of integers { 1 , 2 , . . .
, a } and by [ a, b ] the set of integers from a to b .
Deﬁnition 2.1 (Convolutional layer)
A layer k is called a convolutional layer if its output f k ( x ) ∈ R n k is deﬁned for every x ∈ R d as
[Equation]
(1) for every p ∈
[ P k − 1 ] , t ∈ [ T k ] , h :=
( p − 1) T k + t.
The value of each neuron at layer k is computed by ﬁrst taking the inner product between a ﬁlter of layer k and a patch at layer k − 1 , adding the bias and then applying the activation function.
The number of neurons at layer k is thus n k = T k P k − 1 , which we denote as the width of layer k .
Our deﬁnition of a convolutional layer is quite general as every patch can be an arbitrary subset of neurons of the same layer and thus covers most of existing variants in practice.
Deﬁnition 2.1 includes the fully connected layer as a special case by using P k − 1 = 1 , l k − 1 = n k − 1 , f 1 k − 1 ( x ) =
f k − 1 ( x ) ∈ R n k − 1 , T k = n k , W k ∈ R n k − 1 × n k , b k ∈ R n k .
Thus we have only one patch which is the whole feature vector at this layer.
Deﬁnition 2.2 (Fully connected layer)
A layer k is called a fully connected layer if its output f k ( x ) ∈ R n k is deﬁned for every x ∈ R d as
[Equation]
For some results we also allow max-pooling layers.
Deﬁnition 2.3 (Max-pooling layer)
A layer k is called a max-pooling layer if its output f k
[Equation]
x ) ∈ R n k is deﬁned for every x ∈ R d and p ∈
[ P k − 1 ] as
[Equation]
A max-pooling layer just computes the maximum element of every patch from the previous layer.
Since there are P k − 1 patches at layer k − 1 , the number of output neurons at layer k is n k =
P k − 1 .
Reformulation of Convolutional Layers: For each convolutional or fully connected layer, we denote by M k :
R l k − 1 × T k → R n k − 1 × n k the linear map that returns for every parameter matrix
W k ∈ R l k − 1 × T k the corresponding full weight matrix
U k
= M k ( W k ) ∈ R n k − 1 × n k .
For convolutional layers, U k can be seen as the counterpart of the weight matrix W k in fully connected layers.
We deﬁne U k
= M k ( W k )
=
W k if layer k is fully connected.
Note that the mapping M k depends on the patch structure of each convolutional layer k .
For example, suppose that layer k has two ﬁlters of length 3 , that is, W k
=
[ w 1 k , w 2 k ] = (cid:34) a d b e c f (cid:35) , and n k − 1 = 5 and patches given by a 1D-convolution with stride 1 and no padding then: U Tk =
M k ( W k ) T =   a b c  d e f   a b c  d e f   a b c  d e f 
 ,
The above ordering of the rows of U Tk of a convolutional layer is determined by Equation (1), in particular, the row index
h of U T k is calculated as h =
( p − 1) T k + t , which means for every given patch p one has to loop over all the ﬁlters t and compute the corresponding value of the output unit by taking the inner product of the h -th row of U
Tk with the whole feature vector of the previous layer.
We assume throughout this paper that that there is no non-linearity at the output layer.
By ignoring max-pooling layers for the moment, the feature maps f
k : R d → R n k can be written as f 0 ( x ) =
x, f k ( x ) =
σ k (cid:0) g k ( x ) (cid:1) , where
[Equation]
where g k : R d → R n k is the pre-activation output at layer k .
By stacking the feature vectors of layer k of all training samples, before and after applying the activation function, into a matrix, we deﬁne:
F k = [
[Equation]
k
[Equation] [Equation]
, f k
[Equation]
x N )]
[Equation]
N
[Equation]
n k , and
[Equation]
In this paper
[Equation]
we refer to F k as the output matrix at layer k .
It follows from above that
[Equation]
In this paper
[Equation]
we assume the following general condition on the structure of convolutional layers.
Assumption 2.4 (Convolutional Structure)
For every convolutional layer k , there exists at least one parameter matrix W k ∈
R l k − 1 × T k for which the corresponding weight matrix
U k
=
M k ( W k ) ∈ R n k − 1 × n k has full rank.
It is straightforward to see that Assumption 2.4 is satisﬁed if every neuron of a convolutional layer belongs to at least one patch and there are no identical patches.
Lemma 2.
If Assumption 2.4 holds, then for every convolutional layer k , the set of W k ∈ R l k − 1 × T k for which U k
= M k ( W k ) ∈ R n k − 1 × n k does not have full rank has Lebesgue measure zero.
Proof: Since U k
= M k ( W k ) ∈ R n k − 1 × n k and M k is a linear map, every entry of U k is a linear function of the entries of W k .
Let m = min( n k − 1 , n k ) , then the set of low rank matrices
U k is characterized by a system of equations where the (cid:0) max( n k − 1 ,n k ) m (cid:1) determinants of all m × m sub-matrices of U k are zero.
As the determinant is a polynomial in the entries of the matrix and thus a real analytic function, and the composition of analytic functions is again analytic, we get that each determinant is a real analytic function of W k .
By Assumption 2.4, there exists at least one W k such that one of these determinants is non-zero.
Thus by Lemma A.2, the set of W k where this determinant is zero has Lebesgue measure zero.
As all the submatrices need to have low rank in order that U k has low rank, we get that the set of W k where U k has low rank has Lebesgue measure zero.
(cid:3)
W IDE CNN S CAN LEARN LINEARLY INDEPENDENT FEATURES
In this section, we show that a class of standard CNN architectures with convolutional layers, fully connected layers and max-pooling layers plus standard activation functions like ReLU, sigmoid, softplus, etc are able to learn linearly independent features at any hidden layer if that layer has more neurons than the number of training samples.
Our assumption on training data is the following.
Assumption 3.1 (Training data)
The patches of different training samples are non-identical, that is, x pi (cid:54)
= x qj for every p, q ∈ [ P 0 ] , i, j ∈ [ N ] , i (cid:54) =
j.
Assumption 3.1 is quite weak, especially if the size of the input patches is larger.
If the assumption does not hold, one can add a small perturbation to the training samples: { x 1 + (cid:15) 1 , . . .
, x N + (cid:15) N } .
The set of { (cid:15) i } Ni =1 where Assumption 3.1 is not fulﬁlled for the new dataset has measure zero.
Moreover, { (cid:15) i } Ni =1 can be chosen arbitrarily small so that the inﬂuence of the perturbation is negligible.
Our main assumptions on the activation function of the hidden layers is the following.
Assumption 3.2 (Activation function)
The activation function σ is continuous, non-constant, and satisﬁes one of the following conditions:
• ∃ µ + , µ − ∈ R s.t.
lim t →−∞ σ k ( t ) = µ − and lim t →∞ σ k ( t ) = µ + and µ + µ − =• ∃ ρ 1 , ρ 2 , ρ 3 , ρ 4 ∈ R + s.t.
| σ ( t ) | ≤ ρ 1 e ρ 2 t for t < 0 and | σ ( t ) | ≤ ρ 3 t + ρ 4 for t ≥
Assumption 3.2 covers several standard activation functions.
Lemma 3.
The following activation functions satisfy Assumption 3.2:
[Equation]
The softplus function is a smooth approximation of ReLU.
It holds:
[Equation]
The ﬁrst main result of this paper is the following.
Theorem 3.4 (Linearly Independent Features) Let Assumption 3.1 hold for the training sample.
Consider a deep CNN architecture for which there exists some layer 1 ≤ k ≤ L − 1 such that
Layer 1 and layer k are convolutional or fully connected while all the other layers can be convolutional, fully connected or max-pooling 2.
The width of layer k is larger than the number of training samples, n k =
T k P k − 1 ≥ N 3.
( σ 1 , . . .
, σ k ) satisfy Assumption 3.
Then there exist a set of parameters of the ﬁrst k layers ( W l , b l ) kl =1 such that the set of feature vectors { f k ( x 1 ) , . . .
, f k ( x N ) } are linearly independent.
Moreover, ( W l , b l ) kl
=1 can be chosen in such a way that all the weight matrices
U l
=
M l ( W l ) have full rank for every 1 ≤ l ≤ k.
Theorem 3.4 implies that a large class of CNNs employed in practice with convolutional, fully connected and max-pooling layers and standard activation functions like ReLU, sigmoid or softplus can produce linearly independent features at any hidden layer if its width is larger than the number of training samples.
Figure 1 shows an example of a CNN architecture that satisﬁes the conditions of Theorem 3.4 at the ﬁrst convolutional layer ( i.e. k = 1 ).
Note that if a set of vectors is linearly independent then they are also linearly separable.
In this sense, Theorem 3.4 suggests that deep and wide CNNs can produce linearly separable features at every wide hidden layer.
Linear separability in neural networks has been recently studied by An et al.
(2015), where the authors show that a two-hidden-layer fully connected network with ReLU activation function can transform any training set to be linearly separable while approximately preserving the distances of the training data at the output layer.
Compared to An et al.
our Theorem 3.4 is derived for
CNNs with a wider range of activation functions.
Moreover, our result shows even linear independence of features which is stronger than linear separability.
Recently, Nguyen & Hein (2017) have shown a similar result for deep fully connected networks and analytic activation functions.
We note that, in contrast to fully connected networks, for CNNs the condition n k ≥ N of Theorem 3.4 does not necessarily imply that the network has a huge number of parameters as the layers k and k + 1 can be chosen to be convolutional.
In particular, the condition n k = T
k P
k − 1 ≥ N can be fulﬁlled by increasing the number of ﬁlters T k or by using a large number of patches P k − 1 (however
P k − 1 is upper bounded by n k ), which is however only possible if l k − 1 is small as otherwise our condition on the patches cannot be fulﬁlled.
Interestingly, such a architecture has been used in the
VGG-Net (Simonyan & Zisserman, 2015), where they use small 3 × 3 ﬁlters and minimal stride 1 in the ﬁrst layer and thus they fulﬁll the condition n k ≥ N for k = 1 , see Table 1, for ImageNet.
Also note that other state-of-the-art-networks fulﬁll the condition in Table 1.
Overall, Theorem 3.4 can be seen as a theoretical support for the usage of small ﬁlters and strides in practical CNN architectures as it increases the chance of achieving linear separability at early hidden layers in the network and also reduces the total number of training parameters.
The reason why linear separability helps will be discussed in Section 4 when we analyze the loss surface of the CNNs.
Note also that the condition n k ≥ N is a sufﬁcient condition but not necessary to prove our results.
In particular, we conjecture that linear separability might hold with far less number of neurons in practical applications.
One might ask now how difﬁcult it is to ﬁnd such parameters which generate linearly independent features at a hidden layer?
Our next result shows that once analytic 2 activation functions, e.g. sigmoid or softplus, are used at the ﬁrst k hidden layers of the network, the linear independence of features at layer k holds with probability 1 even if one draws the parameters of the ﬁrst k layers ( W l , b l ) k randomly on any open Lebesgue measurable subset of the parameter space with any distribution which is absolutely continuous w.r.t. the Lebesgue measure.
Theorem 3.5 Let Assumption 3.1 hold for the training samples.
Consider a deep CNN for which there exists some layer 1 ≤ k ≤ L − 1 such that
Every layer from 1 to k is convolutional or fully connected 2.
The width of layer k is larger than number of training samples, that is, n k =
T k P k − 1 ≥ N 3.
( σ 1 , . . .
, σ k ) are real analytic functions and satisfy Assumption 3.2.
Then the set of parameters of the ﬁrst k layers ( W l , b l ) kl =1 for which the set of feature vectors { f k ( x 1 ) , . .
, f k ( x N ) } of layer k are not linearly independent has Lebesgue measure zero.
Note that Theorem 3.5 is a much stronger statement than Theorem 3.4, as it shows that for almost all weight conﬁgurations one gets linearly independent features at the wide layer.
While Theorem 3.5 does not hold for the ReLU activation function as it is not an analytic function, we want to note again that one can approximate the ReLU function arbitrarily well using the softplus function (see
Equation 6), which is an analytic function for any α > 0 and thus Theorem 3.5 applies.
It is an open question if the result holds also for the ReLU activation function itself.
It is very interesting to note that Theorem 3.5 explains previous empirical observations.
In particular,
Czarnecki et al.
(2017) have shown empirically that linear separability is often obtained already in the ﬁrst few hidden layers of the trained networks.
This is done by attaching a linear classiﬁer probe (Alain & Bengio, 2016) to every hidden layer in the network after training the whole network with backpropagation.
The fact that Theorem 3.5 holds even if the parameters of the bottom layers up to the wide layer k are chosen randomly is also in line with recent empirical observations for CNN architectures that one has little loss in performance if the weights of the initial layers are chosen randomly without training (Jarrett et al., 2009; Saxe et al., 2011; Yosinski et al., 2014).
An application of Theorem 3.4 yields the following universal ﬁnite sample expressivity for CNNs.
In particular, a deep CNN architecture with scalar output can perfectly express the values of any scalar-valued function over a ﬁnite number of inputs as long as the width of the last hidden layer is larger than the number of training samples.
Corollary 3.6 (Universal Finite Sample Expressivity) Let Assumption 3.1 hold for the training samples.
Consider a standard CNN with scalar output which satisﬁes the conditions of Theorem 3.
at the last hidden layer k
= L − 1 .
Let f L : R d → R be the output of the network given as
[Equation]
where λ ∈ R n L − 1 is the weight vector of the last layer.
Then for every target output y ∈ R N , there exists (cid:8) λ, ( W l , b l ) L − 1 l =1 (cid:9) so that it holds f L ( x i ) =
y i for every i ∈ [ N ] .
Proof: Since the network satisﬁes the conditions of Theorem 3.4 for k =
L − 1 , there exists a set of parameters ( W l , b l ) L − 1 l =1 such that rank ( F L − 1 ) =
N. Let F L =
[ f L ( x 1 ) , . . .
, f L ( x N )]
T ∈ R N then it follows that F L = F L − 1 λ.
Pick λ =
F TL − 1 ( F L − 1 F TL − 1 ) − 1 y
then it holds F L = F L − 1 λ =
y.
(cid:3)
The expressivity of neural networks has been well-studied in the literature, in particular in the universal approximation theorems for one hidden layer networks (Cybenko, 1989;
Hornik et al., 1989).
Recently, many results have be shown why deep networks are superior to shallow networks in terms of expressiveness (Delalleau & Bengio, 2011; Telgarsky, 2016; 2015; Eldan & Shamir, 2016; Safran & Shamir, 2017; Yarotsky, 2016; Poggio et al., 2016; Liang & Srikant, 2017; Mhaskar & Poggio, 2016;
Montufar et al., 2014; Pascanu et al., 2014; Raghu et al., 2017).
While most of these results are derived for fully connected networks, it seems that Cohen & Shashua (2016) are the ﬁrst ones who study expressivity of CNNs.
In particular, they show that CNNs with max-pooling and ReLU units are universal in the sense that they can approximate any given function if the size of the networks is unlimited.
However, the number of convolutional ﬁlters in this result has to grow exponentially with the number of patches and they do not allow shared weights in their result, which is a standard feature of CNNs.
Our Corollary 3.6 shows universal ﬁnite sample expressivity, instead of universal function approximation, even for L = 2 and k = 1 , that is a single convolutional layer network can perfectly ﬁt the training data as long as the number of hidden units is not smaller than the number of samples.
To the best of our knowledge, this is the ﬁrst result on universal ﬁnite sample expressivity for a large class of practical CNNs.
For fully connected networks, universal ﬁnite sample expressivity has been studied by Zhang et al.
(2017); Nguyen & Hein (2017); Hardt & Ma (2017).
They show that a network with a single hidden layer with N hidden units can express any training set of size N .
While the number of training parameters of a single hidden layer CNN with N hidden units and scalar output is just 2 N + T 1 l 0 , where T 1 is the number of convolutional ﬁlters and l 0 is the length of each ﬁlter, it is Nd + 2 N for fully connected networks.
If we set the width of the hidden layer of the CNN as n 1 = T 1 P 0 = N
[Table]
in order to fulﬁll the condition of Corollary 3.
[Table]
then the number of training parameters of the CNN becomes 2 N + Nl 0 /P 0 , which is less than 3 N if l 0 ≤ P 0 compared to ( d + 2) N for the fully connected case.
In practice one almost always has l 0 ≤ P 0 as l 0 is typically a small integer and Pis on the order of the dimension of the input.
Thus, the number of parameters of the CNN to achieve universal ﬁnite sample expressivity is signiﬁcantly smaller than that of fully connected networks.
Obviously, in practice it is more important that the network generalizes rather than just ﬁtting the training data.
By using shared weights and sparsity structure, CNNs seem to implicitly regularize the model class in order to achieve good generalization performance.
Thus even though they can ﬁt also random labels or noise (Zhang et al., 2017) due to the universal ﬁnite sample expressivity shown in Corollary 3.6, they seem still to be able to generalize well (Zhang et al., 2017).
T HE L OSS S URFACE OF C ONVOLUTIONAL N EURAL N ETWORKS
In this section, we restrict our analysis to the use of least squares loss.
However, as we show later that the network can produce exactly the target output ( i.e. F L = Y ) for some choice of parameters, all our results can also be extended to any other loss function where the global minimum is attained at F L = Y , for instance the squared Hinge-loss analyzed in Nguyen & Hein (2017).
Let P denote the space of all parameters of the network.
The ﬁnal training objective Φ : P → R is given as
[Equation]
where F L is deﬁned as in (5), which is also the same as
[Equation]
where U l = M l ( W l ) for every 1 ≤ l ≤ L.
Our assumptions on the architecture of CNNs is given below.
Assumption 4.1 (CNN Architecture)
Every layer in the network is a convolutional layer or fully connected layer and the output layer is fully connected.
Moreover, there exists some hidden layer 1 ≤ k ≤ L − 1 such that the following holds:
• The width of layer k is larger than number of training samples, that is, n k = T k P k − 1 ≥ N • All the activation functions of the hidden layers ( σ 1 , . . .
, σ L − 1 ) satisfy Assumption 3.
• ( σ k +1 , . . .
, σ L − 1 ) are strictly increasing or strictly decreasing, and differentiable • The network is pyramidal from layer k
+ 1 till the output layer, that is, n k +1 ≥ . .
≥ n L
A typical example that satisﬁes Assumption 4.1 is the following (see Figure 1 for an illustration):
• The ﬁrst layer is a convolutional layer with n 1 = T 1 P 0 ≥ N • Every layer from layer 2 till the output layer is convolutional or fully connected • ( σ 1 , . . .
, σ k ) can be ReLU, sigmoid or softplus • ( σ k +1 , . . .
, σ L − 1 ) can be sigmoid or softplus • n 2 ≥ n 3 ≥ . . .
≥ n L
One can easily check that the above example satisﬁes Assumption 4.1 for k = 1 .
In the following, let us deﬁne for every 1 ≤ k ≤ L − 1 the subset S k ⊆ P such that
S k := (cid:8) ( W l , b l ) Ll =1 (cid:12)(cid:12) rank ( F k )
=
N and U l has full rank for every l ∈ [ k + 2 , L ] (cid:9) .
The set S k is the set of parameters where the feature vectors at layer k are linearly independent and all the weight matrices from layer k + 2 till the output layer have full rank.
In the following, we examine conditions for global optimality in S k .
It is important to note that S k covers almost the whole parameter space under an additional mild condition on the activation function.
Lemma 4.2 Let Assumption 3.1 hold for the training sample and let the CNN architecture satisfy
Assumption 4.1 for some layer 1 ≤ k ≤ L − 1 .
We assume further that the activation functions of the ﬁrst k layers ( σ 1 , . . .
, σ k ) are real analytic.
Then the set P \ S k has Lebesgue measure zero.
Proof: One can see that
[Equation]
By Theorem 3.5, it holds that the set (cid:8) ( W l , b l ) Ll =1 (cid:12)(cid:12) rank ( F k ) < N (cid:9) has Lebesgue measure zero.
Moreover, it follows from Lemma 2.5 that the set (cid:8) ( W l , b l ) Ll =1 (cid:12)(cid:12) U l has low rank for some layer l (cid:9) also has measure zero.
Thus, P \ S k has Lebesgue measure zero.
(cid:3)
In the next key lemma, we bound the objective function in terms of its gradient magnitude w.r.t.
the weight matrix of layer k for which n k ≥ N.
For every matrix A ∈ R m × n , let σ min ( A ) and σ max ( A ) denote the smallest and largest singular value of A .
Let (cid:107) A (cid:107) F = (cid:113)(cid:80) i,j A 2 ij , (cid:107) A (cid:107) min :=
min i,j | A ij | and (cid:107) A (cid:107) max := max i,j | A ij | .
From Equations (4), (5) and (7), it follows that Φ can be seen as a function of ( U l , b l ) Ll =1 , and thus we can use ∇ U k Φ .
If layer k is fully connected then U k
= M k
[Equation]
W k )
=
W k and thus ∇ U k Φ = ∇ W k Φ .
Otherwise, if layer k is convolutional then we note that ∇ U k
Φ is “not” the true gradient of the training objective because U k is not the true optimization parameter but
W k .
In this case, the true gradient of Φ w.r.t.
to the true parameter matrix W k which consists of convolutional ﬁlters can be computed via chain rule as
[Equation]
Please note that even though we write the partial derivatives with respect to the matrix elements, ∇ W k Φ resp.
∇ U k
Φ are the matrices of the same dimension as W k resp.
U k in the following.
Lemma 4.3 Consider a standard deep CNN which satisﬁes Assumption 4.1 for some hidden layer 1 ≤ k ≤ L − 1 .
Then it holds
[Equation]
and
[Equation]
Our next main result is motivated by the fact that empirically when training over-parameterized neural networks with shared weights and sparsity structure like CNNs, there seem to be no problems with sub-optimal local minima.
In many cases, even when training labels are completely random, local search algorithms like stochastic gradient descent can converge to a solution with almost zero training error (Zhang et al., 2017).
To understand better this phenomenon, we ﬁrst characterize in the following Theorem 4.4 the set of points in parameter space with zero loss, and then analyze in
Theorem 4.5 the loss surface for a special case of the network.
We emphasize that our results hold for standard deep CNNs with convolutional layers with shared weights and fully connected layers.
Theorem 4.4 (Necessary and Sufﬁcient Condition for Zero Training Error)
Let
Assumption 3.1 hold for the training sample and suppose that the CNN architecture satisﬁes Assumption 4.
for some hidden layer 1 ≤ k ≤ L − 1 .
Let Φ : P → R be deﬁned as in (7) .
Given any point ( W l , b l ) Ll =1 ∈ S k .
Then it holds that Φ (cid:16) ( W l , b l ) Ll =1 (cid:17) =
if and only if ∇ U k +1 Φ (cid:12)(cid:12)(cid:12) ( W l ,b l ) Ll =1 = 0 .
Proof: If Φ (cid:16) ( W l , b l ) Ll =1 (cid:17) =
then it follows from the upper bound of Lemma 4.3 that ∇ U
k +1 Φ = 0 .
Now, we suppose that ∇ U k +1 Φ = 0 .
Since ( W l , b l ) Ll =1 ∈ S k
it holds rank ( F k ) =
N and U l has full rank for every l ∈ [ k + 2 , L ] .
Thus it holds σ min ( F k ) > 0 and σ min ( U l ) > 0 for every l ∈ [ k + 2 , L ] .
Moreover, ( σ k +1 , . . .
, σ L − 1 ) have non-zero derivative by Assumption 4.1 and thus (cid:107) σ (cid:48)
l ( G l ) (cid:107) min > 0 for every l ∈ [ k + 1 , L − 1] .
This combined with the lower bound in Lemma 4.
leads to (cid:107) F L − Y (cid:107) F = 0 and thus Φ (cid:16) W l , b l ) Ll =1 (cid:17)
= 0 .
(cid:3)
Lemma 4.2 shows that the set of points which are not covered by Theorem 4.4 has just measure zero under a mild condition.
The necessary and sufﬁcient condition of Theorem 4.4 is rather intuitive as it requires the gradient of the training objective to vanish w.r.t.
the full weight matrix of layer k
+regardless of the architecture of this layer.
It turns out that if layer k + 1 is fully connected, then this condition is always satisﬁed at a critical point, in which case we obtain that every critical point in S k is a global minimum with exact zero training error.
This is shown in the next Theorem 4.5, where we consider a classiﬁcation task with m classes.
Z ∈ R m × m is the full rank class encoding matrix
e.g. the identity matrix and ( X, Y ) the training sample such that Y i : =
Z j : whenever x i belongs to class j for every i ∈ [ N ] , j ∈ [ m ] .
Theorem 4.5 (Loss Surface of CNNs) Let ( X, Y, Z ) be a training set for which Assumption 3. holds, the CNN architecture satisﬁes Assumption 4.1 for some hidden layer 1 ≤ k ≤ L − 1 , and layer k + 1 is fully connected.
Let Φ : P → R be deﬁned as in (7) .
Then
• Every critical point ( W l , b l ) Ll =1 ∈ S k is a global minimum with Φ (cid:16) ( W l , b l ) Ll =1 (cid:17) =
• There exist inﬁnitely many global minima ( W l , b l ) Ll =1 ∈ S k with Φ (cid:16) ( W l , b l ) Ll =1 (cid:17) =
Theorem 4.5 indicates that the loss surface for this type of CNNs has a rather simple structure in the sense that almost every critical point is a global minimum with zero training error.
It remains an interesting open problem if this result can be transferred to the case where layer k + 1 is also convolutional.
In any case whether layer k + 1 is fully connected or not, one might still assume that a solution with zero training error still exists.
However, note that Theorem 4.4 shows that at those points where the loss is zero, the gradient of Φ w.r.t.
U k +1 must be zero as well.
An interesting special case of Theorem 4.5 is when the network is fully connected, in which case all the results of
Theorem 4.5 hold without any modiﬁcations.
Corollary 4.6 (Loss Surface of Fully Connected Nets) Let ( X, Y, Z ) be a training set with nonidentical training samples,
i.e. x i (cid:54) =
x j for every i (cid:54) =
j and the fully connected network satisﬁes
Assumption 4.1 for some layer 1 ≤ k ≤ L − 1 .
Let Φ : P → R be deﬁned as in (7) .
Then the following holds
• Every critical point ( W l , b l ) Ll =1 ∈ S k is a global minimum with Φ (cid:16) ( W l , b l ) Ll =1 (cid:17) =
• There exist inﬁnitely many global minima ( W l , b l ) Ll =1 ∈ S k with Φ (cid:16) ( W l , b l ) Ll =1 (cid:17) =
Corollary 4.6 can be seen as a formal proof for the implicit assumption used in the recent work (Nguyen & Hein, 2017) that there exists a global minimum with zero training error for the class of fully connected, deep and wide networks.
C ONCLUSION
We have analyzed the expressiveness and loss surface of CNNs in realistic and practically relevant settings.
As state-of-the-art networks fulﬁll exactly or approximately the condition to have a sufﬁciently wide convolutional layer, we think that our results help to understand why current CNNs can be trained so effectively.
It would be interesting to discuss the loss surface for cross-entropy loss, which currently does not ﬁt into our analysis as the global minimum does not exist when the data is linearly separable.
A M ISSING P ROOFS OF S ECTIONA. P ROOF OF L EMMA 3.
• ReLU: It holds for every t < 0 that σ ( t ) = max(0 , t ) = 0 < e t , and for t ≥ 0 that σ ( t ) =
t < t + 1 .
Thus ReLU satisﬁes the second condition of Assumption 3.2.
• Sigmoid: It holds that lim t
[Equation]
lim
[Equation]
=
[Equation]
Thus σ satisﬁes the ﬁrst condition of Assumption 3.2.
• Softplus: Since 1 + e αt ≤ 2 e αt for every t ≥ 0 , it holds for every t ≥ 0 that
[Equation]
Moreover, since log(1+ t ) ≤ t for t > 0 , it holds log(1+ e αt ) ≤ e αt for every t ∈ R .
Thus it holds that 0 ≤ σ α ( t ) ≤ e αt α for every t < 0 .
Therefore σ α satisﬁes the second condition of Assumption 3.2 for ρ 1 = 1 /α, ρ 2 =
α, ρ 3 = 1 , ρ 4 = log(2) /α.
A. P ROOF OF T HEOREM 3.
To prove Theorem 3.4, we ﬁrst show that Assumption 3.1 can be transported from the input to the output layer.
Lemma A.1 Let Assumption 3.1 hold for the training sample.
Consider a standard deep CNN architecture which satisﬁes the following
The ﬁrst layer is either convolutional or fully connected while all the other layers can be convolutional, fully connected or max-pooling 2.
( σ 1 , . . .
, σ L ) are continuous and non-constant activation functions
Then for every layer 1 ≤ k ≤ L , there exist a set of parameters of the ﬁrst k layers ( W l , b l ) kl =1 such that it holds f pk ( x i ) (cid:54) = f
qk ( x j ) for every p, q ∈ [ P k ] , i, j ∈ [ N ] , i (cid:54) =
j. Moreover, ( W l , b l ) kl =1 can be chosen in such a way that, except for max-pooling layers, all the weight matrices
U l
=
M l ( W l ) have full rank for every 1 ≤ l ≤ k.
Proof: Since Assumption 3.1 holds for the training inputs, one can ﬁrst pick ( W 1 , b 1 ) such that
Assumption 3.1 holds at the ﬁrst layer.
Then given that Assumption 3.1 is satisﬁed at the ﬁrst layer, the selection of the parameters of the higher layers can be done similarly.
Since our deﬁnition of a convolutional layer includes fully connected layer as a special case, it is sufﬁcient to prove the result for the general convolutional structure.
Since the ﬁrst layer is a convolutional layer by our assumption, we denote by Q =
[ a 1 , . .
a T 1 ] ∈ R l 0 × T 1 a matrix that contains the set of convolutional ﬁlters of the ﬁrst layer.
Note here that there are T 1 ﬁlters, namely (cid:8) a 1 , . .
. , a T 1 (cid:9) , where each ﬁlter a t ∈ R l 0 for every t ∈
[ T 1 ] .
Let us deﬁne the set
S :=
(cid:8) Q ∈ R l 0 × T 1 (cid:12)(cid:12)
M 1 ( Q ) has low rank (cid:9) ∪
[Table]
Basically, S is the set of “true” parameter matrices of the ﬁrst layer where the corresponding weight matrix M 1 ( Q ) has low rank or there exists two patches of two different training samples that have the same inner product with some corresponding two ﬁlters.
By Assumption 3.1 it holds that x pi (cid:54) =
x qj for every p, q ∈ [ P 0 ] , i (cid:54) =
j , and thus the right hand side in the above formula of S is just the union of a ﬁnite number of hyperplanes which has Lebesgue measure zero.
For the left hand side, it follows from Lemma 2.5 that the set of Q for which M 1 ( Q ) does not have full rank has measure zero.
Thus the left hand side of S is a set of measure zero.
Since S is the union of two measure zero sets, it has also measure zero, and thus the complementary set R l 0 × T 1 \ S must be non-empty and we choose W 1 ∈ R l 0 × T 1 \ S.
Since σ 1 is a continuous and non-constant function, there exists an interval ( µ 1 , µ 2 ) such that σ 1 is bijective on ( µ 1 , µ 2 ) .
We select and ﬁx some matrix Q
=
[ a 1 , . .
. , a T 1 ] ∈ R l 0 × T 1 \ S and select some β ∈ ( µ 1 , µ 2 ) .
Let α > 0 be a free variable and W
= [ w 11 , . .
. , w T 1 1 ] where w t 1 denotes the t -th ﬁlter of the ﬁrst layer.
Let us pick
[Equation]
It follows that W 1 = αQ and thus M 1 ( W 1 ) =
M 1 ( αQ ) =
α M 1 ( Q ) as M 1 is a linear map by our deﬁnition.
Since Q / ∈ S by construction, it holds that M 1 ( W 1 ) has full rank for every α (cid:54) = 0 .
By
Deﬁnition 2.1, it holds for every i ∈ [ N ] ,
p ∈
[ P 0 ] , t ∈
[ T 1 ] , h = ( p − 1) T 1 + t that
[Equation]
Since β ∈ ( µ 1 , µ 2 ) , one can choose a sufﬁciently small positive α such that it holds α (cid:104) a, x pi (cid:105) + β ∈ ( µ 1 , µ 2 ) for every i ∈ [ N ] ,
p ∈
[ P 0 ] , t ∈
[ T 1 ] .
Under this construction, we will show that every entry of f 1 ( x i ) must be different from every entry of f 1 ( x j ) for i (cid:54) =
j. Indeed, let us compare f 1 ( x i ) h and f 1 ( x j ) v for some h
= ( p − 1) T 1 + t, v = ( q − 1) T 1 + t (cid:48)
and i (cid:54) =
j. It holds for sufﬁcient small α > 0 that
[Equation]
where the last inequality follows from three facts.
First, it holds (cid:104) a t , x pi (cid:105) (cid:54) =
(cid:68) a t
[Equation]
, x qj (cid:69) since Q / ∈ S.
Second, for the chosen α the values of the arguments of the activation function σ 1 lie within ( µ 1 , µ 2 ) .
Third, since σ 1 is bijective on ( µ 1 , µ 2 ) , it maps different inputs to different outputs.
Now, since the entries of f 1 ( x i ) and that of f 1 ( x j ) are already pairwise different from each other, their corresponding patches must be also different from each other no matter how the patches are organized in the architecture, that is,
[Equation]
Now, if the network has only one layer, i.e. L = 1 , then we are done.
Otherwise, we will prove via induction that this property can be translated to any higher layer.
In particular, suppose that one has already constructed ( W l , b l ) kl =1 so that it holds
[Equation]
This is true for k = 1 due to (8).
We will show below that (9) can also hold at layer k + 1 .
Case 1: Layer k + 1 is convolutional or fully connected
Since (9) holds for k by our induction assumption, it must hold that
[Equation]
which means Assumption 3.1 also holds for the set of features at layer k .
Thus one can follows the similar construction as done for layer 1 above by considering the output of layer k as input to layer k + 1 .
Then one obtains that there exist ( W k +1 , b k +1 )
where
U k +1 =
M k +
( W k +1 ) has full rank so that the similar inequality (8) now holds for layer k + 1 , which thus implies (9) holds for k + 1 .
By Deﬁnition 2.3, it holds n
[Equation]
= P k and one has for every
[Equation]
Since (9) holds at layer k by our induction assumption, every entry of every patch of f k ( x i ) must be different from every entry of every patch of f k ( x j ) for every i (cid:54) =
j , that is, ( f pk ( x i ))
r (cid:54) =
( f qk ( x j ))
s for every r,
s ∈
[ l k ] , p, q ∈ [ P k ] , i (cid:54) =
j. Therefore, their maximum elements cannot be the same, that is,
[Equation]
which proves (9) for layer k + 1 .
So far, we have proved that (9) holds for every 1 ≤ k ≤ L .
Thus it follows that for every layer k , there exists a set of parameters of the ﬁrst k layers for which the patches at layer k of different training samples are pairwise different from each other, that is, f pk ( x i ) (cid:54) =
f qk ( x j ) for every p, q ∈
[ P k ] , i (cid:54) =
j. Moreover, except for max-pooling layers, all the weight matrices up to layer k have been chosen to have full rank.
[Equation]
Proof of Theorem 3.4 Let A = F k =
[ f k ( x 1 ) T , . . .
, f k ( x N ) T ] ∈ R N × n k .
Since our deﬁnition of a convolutional layer includes fully connected layer as a special case, it is sufﬁcient to prove the result for convolutional structure.
By Theorem’s assumption, layer k is convolutional and thus it holds by Deﬁnition 2.1 that
[Equation]
for every i ∈ [ N ] , t ∈
[ T k ] , p ∈
[ P k − 1 ] and j = ( p, t ) :
= ( p − 1) T k + t ∈ [ n k ] .
In the following, we show that there exists a set of parameters of the network such that rank ( A ) =
N and all the weight matrices
U l = M l ( W l ) have full rank.
First, one observes that the subnetwork consisting of all the layers from the input layer till layer k − 1 satisﬁes the conditions of Lemma A.1.
Thus by applying Lemma A.1 to this subnetwork, one obtains that there exist ( W l , b l ) k − 1 l =1 for which all the matrices ( U l ) k − 1 l =1 , except for max-pooling layers, have full rank and it holds that f pk − 1 ( x i ) (cid:54) = f
qk − 1 ( x j ) for every p, q ∈ [ P k − 1 ] , i (cid:54) = j. The main idea now is to ﬁx the parameters of these layers and pick ( W k , b k ) such that U k
= M k ( W k ) has full rank and it holds rank ( A )
= N .
Let us deﬁne the set
[Equation]
From the above construction, it holds that f p k − 1 ( x i ) (cid:54) =
f p k − 1 ( x j ) for every p ∈
[ P k − 1 ] , i (cid:54) = j , and thus S is the union of a ﬁnite number of hyperplanes which thus has measure zero.
Let us denote by
Q =
[ a 1 , . .
. , a T k ] ∈ R l k − 1 × T k a parameter matrix that contains all the convolutional ﬁlters of layer k in its columns.
Pick a t ∈ R l k − 1 \ S for every t ∈ [ T k ] , so that it holds that U k
= M k ( Q ) has full rank.
Note here that such matrix Q always exists.
Indeed, Q is chosen from a positive measure set as its columns ( i.e. a t ) are picked from a positive measure set.
Moreover, the set of matrices Q for which M k ( Q ) has low rank has just measure zero due to Lemma 2.5.
Thus there always exists at least one matrix Q so that all of its columns do not belong to S and that M k ( Q ) has full rank.
In the rest of the proof, the value of matrix Q is ﬁxed.
Let α ∈ R be a free parameter.
Since σ k is a continuous and non-constant function, there exist a β ∈ R such that σ k ( β ) (cid:54) = 0 .
Let the value of β be ﬁxed as well.
We construct the convolutional ﬁlters W k =
[ w 1 k , . .
. , w T k k ] and the biases b k ∈ R n k of layer k as follows.
For every p ∈ [ P k − 1 ] , t ∈ [ T k ] , j = ( p, t ) , we deﬁne
[Equation]
It follows that W k = − αQ and thus U k
= M k ( W k ) =
− α M k ( Q ) as M k is a linear map.
Moreover, since M k ( Q ) has full rank by construction, it holds that U k has full rank for every α (cid:54) = 0 .
As α varies, we get a family of matrices
A ( α ) ∈ R N × n k where it holds for every i ∈ [ N ] , j = ( p, t ) ∈ [ n k ] that
[Equation]
Note that each row of A ( α ) corresponds to one training sample and that permutations of the rows of
A ( α ) do not change the rank of A ( α ) .
We construct a permutation γ of { 1 , 2 , . . .
, N } as follows.
For every j = 1 , 2 , . . .
, N , let ( p, t ) be the tuple determined by j (the inverse transformation for given j ∈ [ n k ] is p = (cid:108)
jT k (cid:109) and t = j − (cid:16) (cid:108)
jT k (cid:109) − 1 (cid:17) T k ) and deﬁne
[Equation]
Note that γ ( j ) is unique for every 1 ≤ j ≤ N since a t / ∈ S. It is clear that γ constructed as above is a permutation of { 1 , 2 , . . .
, N } since every time a different element is taken from the index set [ N ] .
From the deﬁnition of γ ( j ) , it holds that for every j = ( p, t ) ∈
[ N ] , i ∈
[ N ] , i > j that
[Equation]
We can relabel the training data according to the permutation so that w.l.o.g we can assume that γ is the identity permutation, that is, γ ( j ) =
j for every j ∈ [ N ] , in which case it holds for every j = ( p, t ) ∈ [ N ] , i ∈
[ N ] , i > j that
[Equation]
Under the above construction of ( W l , b l ) kl =1 , we are ready to show that there exist α for which rank ( A ( α ))
=
N. Since σ k satisﬁes Assumption 3.2, we consider the following cases.
Case 1 : There are ﬁnite constants µ + , µ − ∈ R s.t.
lim t →−∞ σ k ( t ) = µ − and lim t →∞ σ k ( t ) = µ + and µ + µ − = 0 .
Let us consider the ﬁrst case where µ − = 0 .
From (10) and (11) one obtains
[Equation]
where η ij is given for every i < j where j = ( p, t ) as
[Equation]
Note that η ij cannot be zero for i (cid:54) =
j because a t / ∈ S.
In the following, let us denote
[Equation]
: N as a sub-matrix of A ( α ) that consists of the ﬁrst N rows and columns.
By the
Leibniz-formula one has
[Equation]
where S N is the set of all N ! permutations of the set { 1 , . .
. , N } and γ is the identity permutation.
Now, one observes that for every permutation π (cid:54) =
γ , there always exists at least one component j where π ( j ) >
j in which case it follows from (12)
that
[Equation]
Since there are only ﬁnitely many such terms in (13), one obtains
[Equation]
where the last inequality follows from our choice of β .
Since det( A ( α ) 1: N, 1: N ) is a continuous function of α , there exists α 0 ∈ R such that for every α ≥ α 0 it holds det( A ( α ) 1: N, 1: N ) (cid:54) = 0 and thus rank ( A ( α ) 1: N, 1: N ) = N which further implies rank ( A ( α ))
=
N.
Thus the corresponding set of feature vectors { f k ( x 1 ) , . .
, f k ( x N ) } are linearly independent.
For the case where µ + = 0 , one can argue similarly.
The only difference is that one considers now the limit for α → −∞ .
In particular, (10) and (11) lead to
[Equation]
For every permutation π (cid:54) =
γ there always exists at least one component j where π ( j ) >
j , in which case it holds that
[Equation]
and thus it follows from the Leibniz formula that
[Equation]
Since det( A ( α ) 1: N, 1: N ) is a continuous function of α , there exists α 0 ∈ R such that for every α ≤ α 0 it holds det( A ( α ) 1: N, 1: N ) (cid:54)
=
and thus rank ( A ( α ) 1: N, 1: N ) =
N which further implies rank ( A ( α ))
=
N. Thus the set of feature vectors at layer k are linearly independent.
Case
: There are
positive constants ρ 1 , ρ 2 , ρ 3 , ρ 4 s.t.
| σ k ( t ) | ≤ ρ 1 e ρ 2 t for t < 0 and
|
σ k ( t ) | ≤ ρ 3 t + ρ 4 for t ≥
Our proof strategy is essentially similar to the previous case.
Indeed, for every permutation π (cid:54) =
γ there always exist at least one component j = ( p, t ) ∈ [ N ] where π ( j ) >
j in which case δ j := (cid:10) a t , f pk − 1 ( x j ) − f pk − 1 ( x π ( j ) ) (cid:11) <
due to (11).
For sufﬁciently large α > 0 , it holds that αδ j + β < 0 and thus one obtains from (10) that
[Equation]
If π ( j ) =
j then | A ( α ) π ( j ) j | = | σ k ( β ) | which is a constant.
For π ( j ) < j , one notices that δ j :=
(cid:10) a t , f pk − 1 ( x j ) − f pk − 1 ( x π ( j ) ) (cid:11) can only be either positive or negative as a t / ∈ S.
In this case, if δ j < 0 then it can be bounded by the similar exponential term as above for sufﬁciently large α .
Otherwise it holds αδ j + β > 0 for sufﬁciently large α >
and we get
[Equation]
Overall, for sufﬁciently large α > 0 , there must exist positive constants P, Q, R, S, T such that it holds for every π ∈ S N \ { γ } that
[Equation]
The upper bound goes to zero as α goes to ∞ .
This combined with the Leibniz formula from (13), we get lim α →∞ det( A ( α ) 1: N, 1: N ) =
σ k ( β ) N (cid:54) = 0 .
Since det( A ( α
) 1: N, 1: N ) is a continuous function of α , there exists α 0 ∈ R such that for every α ≥ α 0 it holds det( A ( α ) 1: N, 1: N ) (cid:54) = 0 and thus rank ( A ( α ) 1: N, 1: N ) =
N which implies rank ( A ( α ))
=
N. Thus the set of feature vectors at layer k are linearly independent.
Overall, we have shown that there always exist ( W l , b l ) kl =1 such that the set of feature vectors { f k ( x 1 ) , . . .
, f k ( x N ) } at layer k are linearly independent.
Moreover, ( W l , b l ) k l =1 have been chosen so that all the weight matrices
U l
=
M l ( W l ) , except for max-pooling layers, have full rank for every 1 ≤ l ≤ k.
A. P ROOF OF T HEOREM 3.
To prove Theorem 3.5, the following key property of real analytic functions is required.
Lemma A.2 Nguyen (2015); Mityagin (2015)
If f : R n → R is a real analytic function which is not identically zero then the set { x ∈ R n | f ( x ) = 0 } has Lebesgue measure zero.
Any linear function is real analytic and the set of real analytic functions is closed under addition, multiplication and composition, see e.g. Prop. 2.2.2 and Prop. 2.2.8 in Krantz & Parks (2002).
As we assume that all the activation functions of the ﬁrst k layers are real analytic, we get that the function f k is a real analytic function of ( W l , b l ) kl =1 as it is the composition of real analytic functions.
Now, we recall from our deﬁnition that F k
=
[ f k ( x 1 ) , . . .
, f k ( x N )] T ∈ R N × n k is the output matrix at layer k for all training samples.
One observes that the set of low rank matrices
F k can be characterized by a system of equations such that all the (cid:0) n k N (cid:1) determinants of all N × N sub-matrices of F k are zero.
As the determinant is a polynomial in the entries of the matrix and thus an analytic function of the entries and composition of analytic functions are again analytic, we conclude that each determinant is an analytic function of the network parameters of the ﬁrst k layers.
By Theorem 3.4, there exists at least one set of parameters of the ﬁrst k layers such that one of these determinant functions is not identically zero and thus by Lemma A.2, the set of network parameters where this determinant is zero has Lebesgue measure zero.
But as all submatrices need to have low rank in order that rank ( F k ) <
N , it follows that the set of parameters where rank ( F k ) < N has
Lebesgue measure zero.
B M ISSING P ROOFS OF S ECTIONB. P ROOF OF L EMMA 4.
To prove Lemma 4.3, we ﬁrst derive standard backpropagation in Lemma B.1.
In the following we use the Hadamard product ◦ , which for A, B ∈ R m × n is deﬁned as A ◦ B ∈ R m × n with ( A ◦ B ) ij =
A ij B ij .
Let δ kj ( x i ) =
∂
Φ ∂g kj ( x i ) be the derivative of Φ w.r.t.
the value of unit j at layer k evaluated at a single sample x i .
We arrange these vectors for all training samples into a single matrix ∆ k
=
[ δ k : ( x 1 ) , . . .
, δ k : ( x N )] T ∈ R N × n k .
The following lemma is a slight modiﬁcation of
Lemma 2.1 in (Nguyen & Hein, 2017) for which we provide the proof for completeness.
Lemma B.1 Given some hidden layer 1 ≤ k ≤ L − 1 .
Let ( σ k +1 , . . .
, σ L − 1 ) be differentiable.
Then the following hold:
[Equation]
∇ U l Φ = F Tl − 1 ∆ l , ∀ k + 1 ≤ l ≤ L
Proof: 1.
By deﬁnition, it holds for every i ∈ [ N ] , j ∈ [ n L ] that
[Equation]
and hence, ∆ L = F L − Y.
For every k + 1 ≤ l ≤ L − 1 , the output of the network for a single training sample can be written as the composition of differentiable functions ( i.e. the outputs of all layers from l + 1 till the output layer), and thus the chain rule yields for every i ∈ [ N ] , j ∈ [ n l ] that
[Equation]
) s ( x i ) ∂g
( l +1)
s ( x i )
∂f lj ( x i ) ∂f lj ( x i ) ∂g
lj ( x i ) =
n l +1 (cid:88) s =
δ ( l +1) s ( x i )
( U l +1 )
js σ (cid:48)
( g lj ( x i ))
= n l +1 (cid:88) s =
(∆ ( l +1) ) is ( U l
+
Tsj σ (cid:48)
(( G l ) ij ) and hence ∆ l = (∆ l +1 U T l +1 ) ◦ σ (cid:48) ( G l ) .
For every k + 1 ≤ l ≤ L − 1 , r ∈ [ n l − 1 ] , s ∈ [ n l ] , one has
[Equation]
and hence ∇ U l Φ
= F Tl − 1 ∆ l
[Equation]
The following straightforward inequalities are also helpful to prove Lemma 4.3.
Let λ min ( · ) and λ max ( · ) denotes the smallest and largest eigenvalue of a matrix.
Lemma B.2 Let A ∈ R m × n with m ≥ n .
Then it holds σ max ( A ) (cid:107) x (cid:107) 2 ≥ (cid:107)
Ax (cid:107) 2 ≥ σ min ( A ) (cid:107) x (cid:107)for every x ∈ R n .
Proof: Since m ≥ n , it holds that σ min ( A ) =
(cid:112) λ min ( A T A )
= (cid:113) min x T
A T Ax x T x = min
[Equation]
Ax (cid:107) 2 (cid:107) x (cid:107)and thus σ min ( A ) ≤ (cid:107) Ax (cid:107) 2 (cid:107) x (cid:107) 2 for every x ∈ R n .
Similarly, it holds σ max ( A ) =
(cid:112) λ max ( A T A ) =
(cid:113)
max x T
A T Ax x T x = max (cid:107) Ax (cid:107) 2 (cid:107) x (cid:107) 2 and thus σ max ( A ) ≥ (cid:107)
Ax (cid:107) 2 (cid:107) x
[Equation]
2 for every x ∈ R n .
(cid:3)
Lemma B.3 Let A ∈ R m × n , B ∈ R n × p with m ≥ n .
Then it holds σ max ( A ) (cid:107) B (cid:107) F ≥ (cid:107) AB (cid:107) F ≥ σ min ( A ) (cid:107) B (cid:107) F .
Proof: Since m ≥ n , it holds that λ min ( A T A ) =
σ min ( A ) 2 and λ max ( A T A ) = σ max ( A ) 2 .
Thus we have (cid:107) AB (cid:107) 2 F = tr( B T A T AB ) ≥ λ min ( A T A )
tr( B T B ) =
σ min ( A ) 2 (cid:107) B (cid:107) 2 F .
Similarly, it holds (cid:107) AB
[Equation]
(cid:3)
Proof of Lemma 4.
We ﬁrst prove the lower bound.
Let I m denotes an m -by- m identity matrix and ⊗ the Kronecker product.
From Lemma B.1 it holds ∇ U
k +
Φ = F TL ∆ k +1 and thus
[Equation]
It follows that
[Equation]
where the inequality follows from Lemma B.2 for the matrix ( I n k +1 ⊗ F Tk ) ∈ R n k n k +
×
Nn k +1 with n k ≥ N by Assumption 4.1.
Using Lemma B.1 again, one has
[Equation]
where the last inequality follows from Lemma B.3 for the matrices U k +2 ∈ R n k +1 × n k +2 and ∆ Tk + with n k +1 ≥ n k +2 by Assumption 4.1.
By repeating this for (cid:107) ∆ k +2 (cid:107) F , . . .
, (cid:107) ∆ L − 1 (cid:107) F , one gets (cid:107) ∆ k +1 (cid:107) F ≥ (cid:16)
L − 1 (cid:89)
[Equation]
=
[Equation]
From (14), (15), one obtains
[Equation]
which proves the lower bound.
The proof for upper bound is similar.
Indeed one has
[Equation]
16) where the inequality follows from Lemma B.2 Now
[Equation]
one has
[Equation]
where the last inequality follows from Lemma B.
[Equation]
By repeating this chain of inequalities for (cid:107) ∆ k +2 (cid:107) F , . . .
, (cid:107) ∆ L − 1 (cid:107) F , one obtains:
[Equation]
From (16), (17), one obtains that
[Equation]
which proves the upper bound.
B. P ROOF OF T HEOREM 4.
Since layer k + 1 is fully connected, it holds at every critical point in S k that ∇ W k +1 Φ =
= ∇ U
k +1 Φ .
This combined with Theorem 4.4 yields the result.
One basically needs to show that there exist ( W l , b l ) Ll =1 such that it holds: Φ (cid:16) ( W l , b l ) Ll =1 (cid:17)
= 0 , rank ( F k ) =
N and U l = M l ( W l ) has full rank ∀ l ∈
[ k + 2 , L ]
Note that the last two conditions are fulﬁlled by the fact that ( W l , b l ) Ll =1 ∈ S k .
By Assumption 4.1, the subnetwork consisting of the ﬁrst k layers satisﬁes the condition of
Theorem 3.4.
Thus by applying Theorem 3.4 to this subnetwork, one obtains that there exist ( W l , b l ) kl =1 such that rank ( F k )
=
N .
In the following, we ﬁx these layers and show how to pick ( W l , b l ) Ll = k +1 such that F L = Y.
The main idea now is to make the output of all the training samples of the same class become identical at layer k +
and thus they will have the same network output.
Since there are only m classes, there would be only m distinct outputs for all the training samples at layer L − 1 .
Thus if one can make these m distinct outputs become linearly independent at layer L − 1 then there always exists a weight matrix
W L that realizes the target output Y as the last layer is just a linear map by assumption.
Moreover, we will show that, except for max-pooling layers, all the parameters of other layers in the network can be chosen in such a way that all the weight matrices ( U l ) Ll
= k
+ achieve full rank.
Our proof details are as follows.
Case 1: k = L −
It holds rank ( F L − 1 ) =
N. Pick b L = 0 and W L = F TL − 1 ( F L − 1 F TL − 1 ) − 1 Y .
Since the output layer is fully connected, it follows from Deﬁnition 2.2 that
F L = F L − 1 W L + 1 N b TL =
Y. Since rank ( F k )
=
N and the full rank condition on ( W l ) Ll =
k +2 is not active when k = L − 1 , it holds that ( W l , b l ) Ll =1 ∈ S k which ﬁnishes the proof.
Case 2: k = L −
It holds rank ( F L − 2 ) =
N. Let A ∈ R m × n L − 1 be a full row rank matrix
such that A ij ∈ range( σ L − 1 ) .
Note that n L − 1 ≥ n L = m due to Assumption 4.1.
Let D ∈ R N × n L − be a matrix satisfying D
i : =
A j : whenever x i belongs to class j for every i ∈ [ N ] , j ∈ [ m ] .
By construction, F L − 2 has full row rank, thus we can pick b L − 1 = 0 , W L − 1 =
F TL − 2 ( F L − 2 F TL − 2 ) − 1 σ − 1 L − 1 ( D ) .
Since layer L − 1 is fully connected by assumption, it follows from Deﬁnition 2.2 that F L − 1 =
σ L − 1 ( F L − 2 W L − 1 + 1 N b TL − 1 )
=
D and thus ( F L − 1 ) i : =
D i : =
A j : whenever x i belongs to class j .
So far, our construction of the ﬁrst L − 1 layers has led to the fact that all the training samples belonging to the same class will have identical output at layer L − 1 .
Since A has full row rank by construction, we can pick for the last layer
b
L = 0 , W L =
A T ( AA T ) − 1 Z where Z ∈ R m × m is our class embedding matrix with rank ( Z ) =
m. One can easily check that AW L =
Z and that F L = F L − 1 W L + 1 N b TL =
F L − 1 W L where the later follows from
Deﬁnition 2.2 as the output layer is fully connected.
Now one can verify that F L = Y.
Indeed, whenever x i belongs to class j one has
[Equation]
Moreover, since rank ( F k ) =
N and rank ( W L )
=
rank ( A T ( AA T ) − 1 Z )
= m , it holds that ( W l , b l ) Ll =1 ∈ S k .
Therefore, there exists ( W l , b l ) Ll =1 ∈ S k with Φ (cid:16) ( W l , b l ) Ll =1 (cid:17) = 0 .
Case 3: k ≤ L −
It holds rank ( F k ) =
N. Let E ∈ R m × n k +1 be any matrix
such that E ij ∈ range( σ k +1 ) and E ip (cid:54) =
E jq for every 1 ≤ i (cid:54) =
j ≤ N, 1 ≤ p, q ≤ n k
+
Let D ∈ R N × n k +1 satisﬁes
D i : =
E j : for every x i from class j. Pick b k +1 = 0 , W k +
=
F Tk ( F k F Tk ) − 1 σ − 1 k +
( D ) .
Note that the matrix is invertible as F k has been chosen to have full row rank.
Since layer k + 1 is fully connected by our assumption, it follows from Deﬁnition 2.2 that F k +1 =
σ k +1 ( F k W k +
+ 1 N b Tk +1 ) =
σ k +
( F k W k +1 )
=
D and thus it holds
[Equation]
for every x i from class j .
So far, our construction has led to the fact that all training samples belonging to the same class have identical output at layer k + 1 .
The idea now is to see E as a new training data matrix of a subnetwork consisting of all layers from k
+ 1 till the output layer L .
In particular, layer k + 1 can be seen as the input layer of this subnetwork and similarly, layer
L can be seen as the output layer.
Moreover, every row of E ∈ R m × n k +1 can be seen as a new training sample to this subnetwork.
One can see that this subnet together with the training data matrix E satisfy the conditions of Theorem 3.4 at the last hidden layer L − 1 .
In particular, it holds that
• The rows of E ∈ R m × n k +1 are componentwise different from each other, and thus the input patches must be also different from each other, and thus E satisﬁes Assumption 3. • Every layer from k +1 till L − 1 is convolutional or fully connected due to Assumption 4.
• The width of layer L − 1 is larger than the number of samples due to Assumption 4.1, that is, n L − 1 ≥ n L = m • ( σ k +2 , . . .
, σ L − 1 ) satisfy Assumption 3.2 due to Assumption 4.
By applying Theorem 3.4 to this subnetwork and training data E , we obtain that there must exist ( W l , b l ) L − 1 l = k +2 for which all the weight matrices ( U l ) L − 1 l = k +2 have full rank such that the set of corresponding m outputs at layer L − 1 are linearly independent.
In particular, let A ∈ R m × n L − 1 be the corresponding outputs of E through this subnetwork then it holds that rank ( A )
=
m. Intuitively, if one feeds E j : as an input at layer k
+ 1 then one would get A j : as an output at layer L − 1 .
This combined with (18) leads to the fact that if one now feeds ( F k +1 ) i : =
E j : as an input at layer k +
then one would get at layer L − 1 the output ( F L − 1 ) i : =
A j : whenever x i belongs to class j .
Last, we pick b L = 0 , W L =
A T ( AA T ) − 1 Z.
It follows that AW L = Z.
Since the output layer L is fully connected, it holds from Deﬁnition 2.2 that F L = F L − 1 W L + 1 N b TL =
F L − 1 W L .
One can verify now that F L = Y. Indeed, for every sample x i from class j it holds that
[Equation]
Overall, we have shown that Φ = 0 .
In addition, it holds rank ( F k ) =
N from the construction of the ﬁrst k layers.
All the matrices ( U l ) L − 1 l = k +2 have full rank by the construction of the subnetwork from k + 1 till L .
Moreover, U L = W L also has full rank since rank ( W L ) =
rank ( A T ( AA T ) − 1 Z )
= m.
Therefore it holds ( W l , b l ) Ll =1 ∈ S k .
B. P ROOF OF C OROLLARY 4.
It is clear that the network structure satisﬁes the conditions of Theorem 4.4 as every layer is fully connected.
Moreover, the input patches also satisfy Assumption 3.1 because every input patch is simply one training sample in this case but since there are no identical training samples, one derives that the input patches must be different from each other.
Since all the conditions of Theorem 4.4 are met, the application of Theorem 4.4 yields the result.