Original: O N THE LOSS LANDSCAPE OF A CLASS OF DEEP NEURAL NETWORKS WITH NO BAD LOCAL VALLEYS
Revised: O N THE LOSS LANDSCAPE OF A CLASS OF DEEP NEURAL NETWORKS WITH NO BAD LOCAL VALLEYS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We identify a class of over-parameterized deep neural networks with standard activation functions and cross-entropy loss which provably have no bad local valley, in the sense that from any point in parameter space there exists a continuous path on which the cross-entropy loss is non-increasing and gets arbitrarily close to zero.
Revised: We identify a class of over-parameterized deep neural networks with standard activation functions and cross-entropy loss which provably have no bad local valley, in the sense that from any point in parameter space there exists a continuous path on which the cross-entropy loss is non-increasing and gets arbitrarily close to zero.

Original: This implies that these networks have no sub-optimal strict local minima.
Revised: This implies that these networks have no sub-optimal strict local minima.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: It has been empirically observed in deep learning (Dauphin et al., 2014; Goodfellow et al., 2015) that the training problem of over-parameterized 1 deep CNNs (LeCun et al., 1990; Krizhevsky et al., 2012) does not seem to have a problem with bad local minima.
Revised: It has been empirically observed in deep learning (Dauphin et al., 2014; Goodfellow et al., 2015) that the training problem of over-parameterized 1 deep CNNs (LeCun et al., 1990; Krizhevsky et al., 2012) does not seem to have a problem with bad local minima.

Original: In many cases, local search algorithms like stochastic gradient descent (SGD) frequently converge to a solution with zero training error even though the training objective is known to be non-convex and potentially has many distinct local minima (Auer et al., 1996; Safran & Shamir, 2018).
Revised: In many cases, local search algorithms like stochastic gradient descent (SGD) frequently converge to a solution with zero training error even though the training objective is known to be non-convex and potentially has many distinct local minima (Auer et al., 1996; Safran & Shamir, 2018).

Original: This indicates that the problem of training practical over-parameterized neural networks is still far from the worst-case scenario where the problem is known to be NP-hard (Blum & Rivest., 1989; Sima, 2002;
Revised: This indicates that the problem of training practical over-parameterized neural networks is still far from the worst-case scenario where the problem is known to be NP-hard (Blum & Rivest., 1989; Sima, 2002;

Original: Livni et al., 2014; ShalevShwartz et al., 2017).
Revised: Livni et al., 2014; ShalevShwartz et al., 2017).

Original: A possible hypothesis is that the loss landscape of these networks is“wellbehaved” so that it becomes amenable to local search algorithms like SGD and its variants.
Revised: A possible hypothesis is that the loss landscape of these networks is“wellbehaved” so that it becomes amenable to local search algorithms like SGD and its variants.

Original: As not all neural networks have a well-behaved loss landscape, it is interesting to identify sufﬁcient conditions on their architecture so that this is guaranteed.
Revised: As not all neural networks have a well-behaved loss landscape, it is interesting to identify sufﬁcient conditions on their architecture so that this is guaranteed.

Original: In this paper our motivation is to come up with such a class of networks in a practically relevant setting, that is we study multi-class problems with the usual empirical cross-entropy loss and deep (convolutional) networks and almost no assumptions on the training data, in particular no distributional assumptions.
Revised: In this paper our motivation is to come up with such a class of networks in a practically relevant setting, that is we study multi-class problems with the usual empirical cross-entropy loss and deep (convolutional) networks and almost no assumptions on the training data, in particular no distributional assumptions.

Original: Thus our results directly apply to the networks which we use in the experiments.
Revised: Thus our results directly apply to the networks which we use in the experiments.

Original: [Figure]
Revised: [Figure]

Original: Our contributions.
Revised: Our contributions.

Original: We identify a family of deep networks with skip connections to the output layer whose loss landscape has no bad local valleys (see Figure 1 for an illustration).
Revised: We identify a family of deep networks with skip connections to the output layer whose loss landscape has no bad local valleys (see Figure 1 for an illustration).

Original: Our setting is for the empirical loss and there are no distributional assumptions on the training data.
Revised: Our setting is for the empirical loss and there are no distributional assumptions on the training data.

Original: Moreover, we study directly the standard cross-entropy loss for multi-class problems.
Revised: Moreover, we study directly the standard cross-entropy loss for multi-class problems.

Original: There are little assumptions on the network structure which can be arbitrarily deep and can have convolutional layers (weight sharing) and skip-connections between hidden layers.
Revised: There are little assumptions on the network structure which can be arbitrarily deep and can have convolutional layers (weight sharing) and skip-connections between hidden layers.

Original: From a practical perspective, one can generate an architecture which fulﬁlls our conditions by taking an existing CNN architecture and then adding skip-connections from a random subset of N neurons ( N is the number of training samples), possibly from multiple hidden layers, to the output layer (see Figure 2 for an illustration).
Revised: From a practical perspective, one can generate an architecture which fulﬁlls our conditions by taking an existing CNN architecture and then adding skip-connections from a random subset of N neurons ( N is the number of training samples), possibly from multiple hidden layers, to the output layer (see Figure 2 for an illustration).

Original: For these networks we show that there always exists a continuous path from any point in parameter space on which the loss is non-increasing and gets arbitrarily close to zero.
Revised: For these networks we show that there always exists a continuous path from any point in parameter space on which the loss is non-increasing and gets arbitrarily close to zero.

Original: We note that this implies the loss landscape has no strict local minima, but theoretically non-strict local minima can still exist.
Revised: We note that this implies the loss landscape has no strict local minima, but theoretically non-strict local minima can still exist.

Original: Beside that, we show that the loss has also no local maxima.
Revised: Beside that, we show that the loss has also no local maxima.

Original: Beside the theoretical analysis, we show in experiments that despite achieving zero training error, the aforementioned class of neural networks generalize well in practice when trained with SGD whereas an alternative training procedure guaranteed to achieve zero training error has signiﬁcantly worse generalization performance and is overﬁtting.
Revised: Beside the theoretical analysis, we show in experiments that despite achieving zero training error, the aforementioned class of neural networks generalize well in practice when trained with SGD whereas an alternative training procedure guaranteed to achieve zero training error has signiﬁcantly worse generalization performance and is overﬁtting.

Original: Thus we think that the presented class of neural networks offer an interesting test bed for future work to study the implicit bias/regularization of SGD.
Revised: Thus we think that the presented class of neural networks offer an interesting test bed for future work to study the implicit bias/regularization of SGD.

Original:  D ESCRIPTION OF NETWORK ARCHITECTURE
Revised:  D ESCRIPTION OF NETWORK ARCHITECTURE

Original: We consider a family of deep neural networks which have d input units, H hidden units, m output units and satisfy the following conditions:
Revised: We consider a family of deep neural networks which have d input units, H hidden units, m output units and satisfy the following conditions:

Original: Every hidden unit of the ﬁrst layer can be connected to an arbitrary subset of input units.
Revised: Every hidden unit of the ﬁrst layer can be connected to an arbitrary subset of input units.

Original: Every hidden unit at higher layers can take as input an arbitrary subset of hidden units from (multiple) lower hidden layers.
Revised: Every hidden unit at higher layers can take as input an arbitrary subset of hidden units from (multiple) lower hidden layers.

Original: Any subgroup of hidden units lying on the same layer can have non-shared or shared weights, in the later case their number of incoming units have to be equal.
Revised: Any subgroup of hidden units lying on the same layer can have non-shared or shared weights, in the later case their number of incoming units have to be equal.

Original: There exist N hidden units which are connected to the output nodes with independent weights ( N denotes the number of training samples).
Revised: There exist N hidden units which are connected to the output nodes with independent weights ( N denotes the number of training samples).

Original: The output of every hidden unit j in the network, denoted as f j : R d → R , is given as
Revised: The output of every hidden unit j in the network, denoted as f j : R d → R , is given as

Original: [Equation]
Revised: [Equation]

Original: where x ∈ R
Revised: where x ∈ R

Original: d is an input vector of the network, σ j : R → R is the activation function of unit j , b j ∈ R is the bias of unit j , and u k → j ∈ R the weight from unit k to unit j .
Revised: d is an input vector of the network, σ j : R → R is the activation function of unit j , b j ∈ R is the bias of unit j , and u k → j ∈ R the weight from unit k to unit j .

Original: This deﬁnition covers a class of deep fully connected and convolutional neural networks with an additional condition on the number of connections to the output layer.
Revised: This deﬁnition covers a class of deep fully connected and convolutional neural networks with an additional condition on the number of connections to the output layer.

Original: In particular, while conventional architectures have just connections from the last hidden layer to the output, we require in our setting that there must exist at least N neurons, “regardless” of their hidden layer, that are connected to the output layer.
Revised: In particular, while conventional architectures have just connections from the last hidden layer to the output, we require in our setting that there must exist at least N neurons, “regardless” of their hidden layer, that are connected to the output layer.

Original: Essentially, this means that if the last hidden layer of a traditional network has just
Revised: Essentially, this means that if the last hidden layer of a traditional network has just

Original: L < N neurons then one can add connections from N − L neurons in the hidden layers below it to the output layer so that the network fulﬁlls our conditions.
Revised: L < N neurons then one can add connections from N − L neurons in the hidden layers below it to the output layer so that the network fulﬁlls our conditions.

Original: Similar skip-connections have been used in DenseNet (Huang et al., 2017) which are different from identity skip-connections as used in ResNets (He et al., 2016).
Revised: Similar skip-connections have been used in DenseNet (Huang et al., 2017) which are different from identity skip-connections as used in ResNets (He et al., 2016).

Original: In Figure 2 we illustrate a network with and without skip connections to the output layer which is analyzed in this paper.
Revised: In Figure 2 we illustrate a network with and without skip connections to the output layer which is analyzed in this paper.

Original: We note that several architectures like DenseNets Huang et al.
Revised: We note that several architectures like DenseNets Huang et al.

Original: (2017) already have skip-connections between hidden layers in their original architecture, whereas our special skip-connections go from hidden layers directly to the output layer.
Revised: (2017) already have skip-connections between hidden layers in their original architecture, whereas our special skip-connections go from hidden layers directly to the output layer.

Original: As our framework allow both kinds to exist in the same network (see
Revised: As our framework allow both kinds to exist in the same network (see

Original: Figure 2 for an example), we would like to separate them from each other by making the convention that in the following skip-connections, if not stated otherwise, always refer to ones which connect hidden neurons to output neurons.
Revised: Figure 2 for an example), we would like to separate them from each other by making the convention that in the following skip-connections, if not stated otherwise, always refer to ones which connect hidden neurons to output neurons.

Original: We denote by d the dimension of the input and index all neurons in the network from the input layer to the output layer as 1 , 2 , . . .
Revised: We denote by d the dimension of the input and index all neurons in the network from the input layer to the output layer as 1 , 2 , . . .

Original: , d, d + 1 , . . .
Revised: , d, d + 1 , . . .

Original: , d + H, d + H + 1 , . . .
Revised: , d + H, d + H + 1 , . . .

Original: , d + H + m which correspond to d input units, H hidden units and m output units respectively.
Revised: , d + H + m which correspond to d input units, H hidden units and m output units respectively.

Original: As we only allow directed arcs from lower layers to upper layers, it follows that k < j for every k → j. Let N be the number of training samples.
Revised: As we only allow directed arcs from lower layers to upper layers, it follows that k < j for every k → j. Let N be the number of training samples.

Original: Suppose that there are M hidden neurons which are directly connected to the output with independent weights where it holds N ≤ M ≤ H. Let { p 1 , . .
Revised: Suppose that there are M hidden neurons which are directly connected to the output with independent weights where it holds N ≤ M ≤ H. Let { p 1 , . .

Original: . , p M } with p j ∈ { d + 1 , . .
Revised: . , p M } with p j ∈ { d + 1 , . .

Original: , d + H } be the set of hidden units which are directly connected to the output units.
Revised: , d + H } be the set of hidden units which are directly connected to the output units.

Original: Let in( j ) be the set of incoming nodes to unit j and u j = [ u k → j ] k ∈ in( j ) the weight vector of the j -th unit.
Revised: Let in( j ) be the set of incoming nodes to unit j and u j = [ u k → j ] k ∈ in( j ) the weight vector of the j -th unit.

Original: Let U =
Revised: Let U =

Original: ( u d +1 , . .
Revised: ( u d +1 , . .

Original: u d + H , b d +1 , . . .
Revised: u d + H , b d +1 , . . .

Original: , b d + H ) denote the set of all weights and biases of all hidden units in the network.
Revised: , b d + H ) denote the set of all weights and biases of all hidden units in the network.

Original: Let V ∈ R M ×
Revised: Let V ∈ R M ×

Original: m be the weight matrix which connects the M hidden neurons to the m output units of the network.
Revised: m be the weight matrix which connects the M hidden neurons to the m output units of the network.

Original: An important quantity in the following is the matrix Ψ ∈ R N × M deﬁned as
Revised: An important quantity in the following is the matrix Ψ ∈ R N × M deﬁned as

Original: [Equation]
Revised: [Equation]

Original: As Ψ depends on U , we write Ψ U or Ψ( U ) as a function of U .
Revised: As Ψ depends on U , we write Ψ U or Ψ( U ) as a function of U .

Original: Let G ∈ R N ×
Revised: Let G ∈ R N ×

Original: m
Revised: m

Original: be the output of the network for all training samples.
Revised: be the output of the network for all training samples.

Original: In particular, G ij is the value of the j -th output neuron for training sample x i .
Revised: In particular, G ij is the value of the j -th output neuron for training sample x i .

Original: It follows from our deﬁnition that
Revised: It follows from our deﬁnition that

Original: [Equation]
Revised: [Equation]

Original: Let ( x i , y i )
Revised: Let ( x i , y i )

Original: Ni =1 be the training set where y i denotes the target class for sample x i .
Revised: Ni =1 be the training set where y i denotes the target class for sample x i .

Original: In this paper we want to analyze the commonly used cross-entropy loss given as
Revised: In this paper we want to analyze the commonly used cross-entropy loss given as

Original: [Equation]
Revised: [Equation]

Original: The cross-entropy loss is bounded from below by zero but this value is not attained.
Revised: The cross-entropy loss is bounded from below by zero but this value is not attained.

Original: In fact the global minimum of the cross-entropy loss need not exist e.g. if a classiﬁer achieves zero training error then by upscaling the function to inﬁnity one can drive the loss arbitrarily close to zero.
Revised: In fact the global minimum of the cross-entropy loss need not exist e.g. if a classiﬁer achieves zero training error then by upscaling the function to inﬁnity one can drive the loss arbitrarily close to zero.

Original: Due to this property, we do not study the global minima of the cross-entropy loss but the question if and how one can achieve zero training error.
Revised: Due to this property, we do not study the global minima of the cross-entropy loss but the question if and how one can achieve zero training error.

Original: Moreover, we note that sufﬁciently small cross-entropy loss implies zero training error as shown in the following lemma.
Revised: Moreover, we note that sufﬁciently small cross-entropy loss implies zero training error as shown in the following lemma.

Original: Lemma 2.

Revised: Lemma 2.

Original: If Φ( U, V ) < log(2) N , then the training error is zero.
Revised: If Φ( U, V ) < log(2) N , then the training error is zero.

Original: Proof: We note that if Φ( U, V ) < log(2) N , then it holds due to the positivity of the loss,
Revised: Proof: We note that if Φ( U, V ) < log(2) N , then it holds due to the positivity of the loss,

Original: [Equation]
Revised: [Equation]

Original: This implies that for all i = 1 , . . .
Revised: This implies that for all i = 1 , . . .

Original: , N ,
Revised: , N ,

Original: [Equation]
Revised: [Equation]

Original: In particular: max k (cid:54) =
Revised: In particular: max k (cid:54) =

Original: y i e G
Revised: y i e G

Original: ik − G iyi < 1 and thus max k (cid:54) =
Revised: ik − G iyi < 1 and thus max k (cid:54) =

Original: y i G ik − G iy
Revised: y i G ik − G iy

Original: i < 0 for all i = 1 , . .
Revised: i < 0 for all i = 1 , . .

Original: , N which implies the result.
Revised: , N which implies the result.

Original: (cid:3)
Revised: (cid:3)

Original:  M AIN RESULT
Revised:  M AIN RESULT

Original: The following conditions are required for the main result to hold.
Revised: The following conditions are required for the main result to hold.

Original: Assumption 3.
Revised: Assumption 3.

Original: All activation functions { σ d +1 , . . .
Revised: All activation functions { σ d +1 , . . .

Original: , σ d + H } are real analytic and strictly increasing 2.
Revised: , σ d + H } are real analytic and strictly increasing 2.

Original: Among M neurons { p 1 , . . .
Revised: Among M neurons { p 1 , . . .

Original: , p M } which are connected to the output units, there exist
Revised: , p M } which are connected to the output units, there exist

Original: N ≤ M neurons, say w.l.o.g. { p 1 , . .
Revised: N ≤ M neurons, say w.l.o.g. { p 1 , . .

Original: . , p N } , such that one of the following conditions hold:
Revised: . , p N } , such that one of the following conditions hold:

Original: • For every 1 ≤ j ≤ N : σ p j is bounded and lim t →−∞ σ p j ( t ) =
Revised: • For every 1 ≤ j ≤ N : σ p j is bounded and lim t →−∞ σ p j ( t ) =

Original: • For every 1 ≤ j ≤ N : σ p j is the softplus activation (3) , and there exists a backward path from p j to the ﬁrst hidden layer s.t. on this path there is no neuron which has skip-connections to the output or shared weights with other skip-connection neurons.
Revised: • For every 1 ≤ j ≤ N : σ p j is the softplus activation (3) , and there exists a backward path from p j to the ﬁrst hidden layer s.t. on this path there is no neuron which has skip-connections to the output or shared weights with other skip-connection neurons.

Original: The input patches of different training samples are distinct.
Revised: The input patches of different training samples are distinct.

Original: In particular, let n 1 be the number of units in the ﬁrst hidden layer and denote by S i for i
Revised: In particular, let n 1 be the number of units in the ﬁrst hidden layer and denote by S i for i

Original: ∈
Revised: ∈

Original: [ d + 1 , d + n 1 ]
Revised: [ d + 1 , d + n 1 ]

Original: their
Revised: their

Original: input support, then for all r (cid:54) =
Revised: input support, then for all r (cid:54) =

Original: s ∈ [ N ] , and i ∈
Revised: s ∈ [ N ] , and i ∈

Original: [ d + 1 , d + n 1 ] , it holds x r | S
Revised: [ d + 1 , d + n 1 ] , it holds x r | S

Original: i (cid:54) =
Revised: i (cid:54) =

Original: x s | S i .
Revised: x s | S i .

Original: The ﬁrst condition of Assumption 3.1 is satisﬁed for softplus , sigmoid , tanh , etc, whereas the second condition is fulﬁlled for sigmoid and softplus .
Revised: The ﬁrst condition of Assumption 3.1 is satisﬁed for softplus , sigmoid , tanh , etc, whereas the second condition is fulﬁlled for sigmoid and softplus .

Original: For softplus activation function (smooth approximation of ReLU),
Revised: For softplus activation function (smooth approximation of ReLU),

Original: [Equation]
Revised: [Equation]

Original: we require an additional assumption on the network architecture.
Revised: we require an additional assumption on the network architecture.

Original: The third condition is always satisﬁed for fully connected networks if the training samples are distinct.
Revised: The third condition is always satisﬁed for fully connected networks if the training samples are distinct.

Original: For CNNs, this condition means that the corresponding input patches across different training samples are distinct.
Revised: For CNNs, this condition means that the corresponding input patches across different training samples are distinct.

Original: This could be potentially violated if the ﬁrst convolutional layer has very small receptive ﬁelds.
Revised: This could be potentially violated if the ﬁrst convolutional layer has very small receptive ﬁelds.

Original: However, if this condition is violated for the given training set then after an arbitrarily small random perturbation of all training inputs it will be satisﬁed with probability 1.
Revised: However, if this condition is violated for the given training set then after an arbitrarily small random perturbation of all training inputs it will be satisﬁed with probability 1.

Original: Note that the M neurons which are directly connected to the output units can lie on different hidden layers in the network.
Revised: Note that the M neurons which are directly connected to the output units can lie on different hidden layers in the network.

Original: Also there is no condition on the width of every individual hidden layer as long as the total number of hidden neurons in the network is larger than N so that our condition M ≥ N is feasible.
Revised: Also there is no condition on the width of every individual hidden layer as long as the total number of hidden neurons in the network is larger than N so that our condition M ≥ N is feasible.

Original: Overall, we would like to stress that Assumption 3.1 covers a quite large class of interesting network architectures but nevertheless allows us to show quite strong results on their empirical loss landscape.
Revised: Overall, we would like to stress that Assumption 3.1 covers a quite large class of interesting network architectures but nevertheless allows us to show quite strong results on their empirical loss landscape.

Original: The following key lemma shows that for almost all U , the matrix Ψ( U ) has full rank.
Revised: The following key lemma shows that for almost all U , the matrix Ψ( U ) has full rank.

Original: Lemma 3.2 Under Assumption 3.1, the set of U such that Ψ( U ) has not full rank N has Lebesgue measure zero.
Revised: Lemma 3.2 Under Assumption 3.1, the set of U such that Ψ( U ) has not full rank N has Lebesgue measure zero.

Original: Proof: (Proof sketch)
Revised: Proof: (Proof sketch)

Original: Due to space limitation, we can only present below a proof sketch.
Revised: Due to space limitation, we can only present below a proof sketch.

Original: We refer the reader to the appendix for the detailed proof.
Revised: We refer the reader to the appendix for the detailed proof.

Original: The proof consists of two main steps.
Revised: The proof consists of two main steps.

Original: First, we show that there exists U s.t.
Revised: First, we show that there exists U s.t.

Original: the submatrix Ψ 1: N, 1: N has full rank.
Revised: the submatrix Ψ 1: N, 1: N has full rank.

Original: By Assumption 3.1, all activation functions are real analytic, thus the determinant of Ψ 1: N, 1: N is a real analytic function of the network parameters which Ψ depends on.
Revised: By Assumption 3.1, all activation functions are real analytic, thus the determinant of Ψ 1: N, 1: N is a real analytic function of the network parameters which Ψ depends on.

Original: By the ﬁrst result, this determinant function is not identically zero, thus Lemma A.1 shows that the set of U for which Ψ has not full rank has Lebesgue measure zero.
Revised: By the ﬁrst result, this determinant function is not identically zero, thus Lemma A.1 shows that the set of U for which Ψ has not full rank has Lebesgue measure zero.

Original: Now we sketch the proof for the ﬁrst step, that is to ﬁnd a U s.t.
Revised: Now we sketch the proof for the ﬁrst step, that is to ﬁnd a U s.t.

Original: Ψ has full rank.
Revised: Ψ has full rank.

Original: By Assumption 3.1.3, one can always choose the weight vectors of the ﬁrst hidden layer so that every neuron at this layer has distinct values for different training samples.
Revised: By Assumption 3.1.3, one can always choose the weight vectors of the ﬁrst hidden layer so that every neuron at this layer has distinct values for different training samples.

Original: For higher neurons, we set their initial weight vectors to be unit vectors with exactly one 1 and 0 elsewhere.
Revised: For higher neurons, we set their initial weight vectors to be unit vectors with exactly one 1 and 0 elsewhere.

Original: Note that the above construction of weights can be easily done so that all the neurons from the same layer and with the same number of incoming units can have shared/unshared weights according to our description of network architecture in Section 2.
Revised: Note that the above construction of weights can be easily done so that all the neurons from the same layer and with the same number of incoming units can have shared/unshared weights according to our description of network architecture in Section 2.

Original: Let c ( j ) be the neuron below j s.t.
Revised: Let c ( j ) be the neuron below j s.t.

Original: u c ( j ) →
Revised: u c ( j ) →

Original: j = 1 .
Revised: j = 1 .

Original: To ﬁnd U , we are going to scale up each weight vector u j by a positive scalar α j .
Revised: To ﬁnd U , we are going to scale up each weight vector u j by a positive scalar α j .

Original: The idea is to show that the determinant of Ψ 1: N, 1: N is non-zero for some positive value of { α j } .
Revised: The idea is to show that the determinant of Ψ 1: N, 1: N is non-zero for some positive value of { α j } .

Original: The biases can be chosen in such a way that the following holds for some β ∈ R ,
Revised: The biases can be chosen in such a way that the following holds for some β ∈ R ,

Original: [Equation]
Revised: [Equation]

Original: where f c ( p j ) ( x ) = σ c ( p j ) ( α c ( p j ) σ c ( c ( p j )) ( . . .
Revised: where f c ( p j ) ( x ) = σ c ( p j ) ( α c ( p j ) σ c ( c ( p j )) ( . . .

Original: f
Revised: f

Original: q j ( x ) . . . )) with q j being the index of some neuron in the ﬁrst hidden layer.
Revised: q j ( x ) . . . )) with q j being the index of some neuron in the ﬁrst hidden layer.

Original: Note that by our construction the value of unit q j is distinct at different training samples, and thus it follows from the strict monotonic property of activation functions from Assumption 3.1 and the positivity of (cid:8) α c ( p j ) , . . .
Revised: Note that by our construction the value of unit q j is distinct at different training samples, and thus it follows from the strict monotonic property of activation functions from Assumption 3.1 and the positivity of (cid:8) α c ( p j ) , . . .

Original: (cid:9)
Revised: (cid:9)

Original: that f c ( p j ) ( x i ) (cid:54) =
Revised: that f c ( p j ) ( x i ) (cid:54) =

Original: f c ( p j ) ( x j ) for every i (cid:54) =
Revised: f c ( p j ) ( x j ) for every i (cid:54) =

Original: j .
Revised: j .

Original: Next, we show that the set of training samples can be re-ordered in such a way that it holds f c ( p j ) ( x i ) <
Revised: Next, we show that the set of training samples can be re-ordered in such a way that it holds f c ( p j ) ( x i ) <

Original: f c ( p j ) ( x j ) for every i > j .
Revised: f c ( p j ) ( x j ) for every i > j .

Original: Note that this re-ordering does not affect the rank of Ψ .
Revised: Note that this re-ordering does not affect the rank of Ψ .

Original: Now, the intuition is that if one let α p j go to inﬁnity then Ψ ij converges to zero for i > j because it holds for all activations from Assumption 3.1 that lim t →−∞ σ p j ( t ) = 0 .
Revised: Now, the intuition is that if one let α p j go to inﬁnity then Ψ ij converges to zero for i > j because it holds for all activations from Assumption 3.1 that lim t →−∞ σ p j ( t ) = 0 .

Original: Thus the determinant of Ψ 1:1 , 1: N converges to (cid:81) Ni =

Revised: Thus the determinant of Ψ 1:1 , 1: N converges to (cid:81) Ni =

Original: σ p
Revised: σ p

Original: j
Revised: j

Original: ( β ) which can be chosen to be non-zero by a predeﬁned value of β , in which case Ψ will have full rank.
Revised: ( β ) which can be chosen to be non-zero by a predeﬁned value of β , in which case Ψ will have full rank.

Original: The detailed proof basically will show how to choose the speciﬁc values of { α j } so that all the above criteria are met.
Revised: The detailed proof basically will show how to choose the speciﬁc values of { α j } so that all the above criteria are met.

Original: In particular, it is important to make sure that the weight vectors of two neurons j, j (cid:48) from the same layer will be scaled by the same factor
Revised: In particular, it is important to make sure that the weight vectors of two neurons j, j (cid:48) from the same layer will be scaled by the same factor

Original: α j = α j (cid:48) as we want to maintain any potential weight-sharing conditions.
Revised: α j = α j (cid:48) as we want to maintain any potential weight-sharing conditions.

Original: The choice of activation functions from the second condition of Assumption 3.1 basically determines how the values of α should be chosen.
Revised: The choice of activation functions from the second condition of Assumption 3.1 basically determines how the values of α should be chosen.

Original: (cid:3)
Revised: (cid:3)

Original: While we conjecture that the result of Lemma 3.2 holds for softplus activation function without the additional condition as mentioned in Assumption 3.1, the proof of this is considerably harder for such a general class of neural networks since one has to control the output of neurons with skip connection from different layers which depend on each other.
Revised: While we conjecture that the result of Lemma 3.2 holds for softplus activation function without the additional condition as mentioned in Assumption 3.1, the proof of this is considerably harder for such a general class of neural networks since one has to control the output of neurons with skip connection from different layers which depend on each other.

Original: However, please note that the condition is also not too restrictive as it just might require more connections from lower layers to upper layers but it does not require that the network is wide.
Revised: However, please note that the condition is also not too restrictive as it just might require more connections from lower layers to upper layers but it does not require that the network is wide.

Original: Before presenting our main result, we ﬁrst need a formal deﬁnition of bad local valleys.
Revised: Before presenting our main result, we ﬁrst need a formal deﬁnition of bad local valleys.

Original: Deﬁnition 3.

Revised: Deﬁnition 3.

Original: The α -sublevel set of Φ is deﬁned as L α =
Revised: The α -sublevel set of Φ is deﬁned as L α =

Original: { ( U, V )
Revised: { ( U, V )

Original: | Φ( U, V ) < α } .
Revised: | Φ( U, V ) < α } .

Original: A local valley is deﬁned as a connected component of some sublevel set L α .
Revised: A local valley is deﬁned as a connected component of some sublevel set L α .

Original: A bad local valley is a local valley on which the loss function Φ cannot be made “arbitrarily small”.
Revised: A bad local valley is a local valley on which the loss function Φ cannot be made “arbitrarily small”.

Original: Intuitively, a typical example of a bad local valley is a small neighborhood around a sub-optimal strict local minimum.
Revised: Intuitively, a typical example of a bad local valley is a small neighborhood around a sub-optimal strict local minimum.

Original: We are now ready to state our main result.
Revised: We are now ready to state our main result.

Original: Theorem 3.4 The following holds under Assumption 3.1:
Revised: Theorem 3.4 The following holds under Assumption 3.1:

Original: There exist uncountably many solutions with zero training error.
Revised: There exist uncountably many solutions with zero training error.

Original: The loss landscape of Φ does not have any bad local valley.
Revised: The loss landscape of Φ does not have any bad local valley.

Original: There exists no suboptimal strict local minimum. 4.
Revised: There exists no suboptimal strict local minimum. 4.

Original: There exists no local maximum.
Revised: There exists no local maximum.

Original: Proof:
Revised: Proof:

Original: By Lemma 3.2 the set of U such that Ψ( U ) has not full rank N has Lebesgue measure zero.
Revised: By Lemma 3.2 the set of U such that Ψ( U ) has not full rank N has Lebesgue measure zero.

Original: Given U such that Ψ has full rank, the linear system Ψ( U ) V
Revised: Given U such that Ψ has full rank, the linear system Ψ( U ) V

Original: = Y has for every possible target output matrix Y ∈ R N × m at least one solution V .
Revised: = Y has for every possible target output matrix Y ∈ R N × m at least one solution V .

Original: As this is possible for almost all U , there exist uncountably many solutions achieving zero training error.
Revised: As this is possible for almost all U , there exist uncountably many solutions achieving zero training error.

Original: Let C be a non-empty, connected component of some α -sublevel set L α for α > 0 .
Revised: Let C be a non-empty, connected component of some α -sublevel set L α for α > 0 .

Original: Suppose by contradiction that the loss on C cannot be made arbitrarily small, that is there exists an (cid:15) > 0 such that Φ( U, V ) ≥ (cid:15) for all ( U, V ) ∈ C , where (cid:15) < α .
Revised: Suppose by contradiction that the loss on C cannot be made arbitrarily small, that is there exists an (cid:15) > 0 such that Φ( U, V ) ≥ (cid:15) for all ( U, V ) ∈ C , where (cid:15) < α .

Original: By deﬁnition, L α can be written as the pre-image of an open set under a continuous function, that is L α
Revised: By deﬁnition, L α can be written as the pre-image of an open set under a continuous function, that is L α

Original: =
Revised: =

Original: Φ − 1 ( { a | a < α } ) , and thus L α must be an open set (see Proposition A.2).
Revised: Φ − 1 ( { a | a < α } ) , and thus L α must be an open set (see Proposition A.2).

Original: Since C is a non-empty connected component of L α , C must be an open set as well, and thus C has
Revised: Since C is a non-empty connected component of L α , C must be an open set as well, and thus C has

Original: non-zero Lebesgue measure.
Revised: non-zero Lebesgue measure.

Original: By Lemma 3.2 the set of U where Ψ( U ) has not full rank has measure zero and thus C must contain a point ( U, V ) such that Ψ( U ) has full rank.
Revised: By Lemma 3.2 the set of U where Ψ( U ) has not full rank has measure zero and thus C must contain a point ( U, V ) such that Ψ( U ) has full rank.

Original: Let Y be the usual zero-one one-hot encoding of the target network output.
Revised: Let Y be the usual zero-one one-hot encoding of the target network output.

Original: As Ψ( U ) has full rank, there always exist V ∗ such that Ψ( U ) V ∗ =
Revised: As Ψ( U ) has full rank, there always exist V ∗ such that Ψ( U ) V ∗ =

Original: Y t ∗ , where t ∗ = log (cid:0) m − 1 e (cid:15) 2 − 1 (cid:1) Note that the loss of ( U, V ∗ ) is
Revised: Y t ∗ , where t ∗ = log (cid:0) m − 1 e (cid:15) 2 − 1 (cid:1) Note that the loss of ( U, V ∗ ) is

Original: [Equation]
Revised: [Equation]

Original: As the cross-entropy loss Φ( U, V ) is convex in V and Φ( U, V ) <
Revised: As the cross-entropy loss Φ( U, V ) is convex in V and Φ( U, V ) <

Original: α we have for the line segment V ( λ )
Revised: α we have for the line segment V ( λ )

Original: = λV + (1 − λ ) V ∗ for λ ∈ [0 , 1] ,
Revised: = λV + (1 − λ ) V ∗ for λ ∈ [0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: Thus the whole line segment is contained in L α and as C is a connected component it has to be contained in C .
Revised: Thus the whole line segment is contained in L α and as C is a connected component it has to be contained in C .

Original: However, this contradicts the assumption that for all ( U, V ) ∈ C it holds Φ( U, V ) ≥ (cid:15) .
Revised: However, this contradicts the assumption that for all ( U, V ) ∈ C it holds Φ( U, V ) ≥ (cid:15) .

Original: Thus on every connected component C of L α the training loss can be made arbitrarily close to zero and thus the loss landscape has no bad valleys.
Revised: Thus on every connected component C of L α the training loss can be made arbitrarily close to zero and thus the loss landscape has no bad valleys.

Original: Let ( U 0 , V 0 ) be a strict suboptimal local minimum, then there exists r > 0 such that Φ( U, V ) >
Revised: Let ( U 0 , V 0 ) be a strict suboptimal local minimum, then there exists r > 0 such that Φ( U, V ) >

Original: Φ( U 0 , V 0 ) > 0 for all ( U, V ) ∈ B
Revised: Φ( U 0 , V 0 ) > 0 for all ( U, V ) ∈ B

Original: (( U 0 , V 0 ) , r ) \ { ( U 0 , V 0 ) } where B ( · , r ) denotes a closed ball of radius r .
Revised: (( U 0 , V 0 ) , r ) \ { ( U 0 , V 0 ) } where B ( · , r ) denotes a closed ball of radius r .

Original: Let α = min
Revised: Let α = min

Original: ( U,V ) ∈ ∂B (cid:0) ( U 0 ,V 0 ) ,r (cid:1) Φ( U, V ) which exists as Φ is continuous and the boundary ∂B (cid:0) ( U 0 , V 0 ) , r (cid:1) of B (cid:0) ( U 0 , V 0 ) , r (cid:1) is compact.
Revised: ( U,V ) ∈ ∂B (cid:0) ( U 0 ,V 0 ) ,r (cid:1) Φ( U, V ) which exists as Φ is continuous and the boundary ∂B (cid:0) ( U 0 , V 0 ) , r (cid:1) of B (cid:0) ( U 0 , V 0 ) , r (cid:1) is compact.

Original: Note that α > Φ( U 0 , V 0 ) as ( U 0 , V 0 ) is a strict local minimum.
Revised: Note that α > Φ( U 0 , V 0 ) as ( U 0 , V 0 ) is a strict local minimum.

Original: Consider the sub-level set
Revised: Consider the sub-level set

Original: D =
Revised: D =

Original: L α +Φ( U 0 ,V 0).
Revised: L α +Φ( U 0 ,V 0).

Original: As Φ( U 0 , V 0 )
Revised: As Φ( U 0 , V 0 )

Original: <
Revised: <

Original: α +Φ( U 0 ,V 0 )it holds ( U 0 , V 0 )
Revised: α +Φ( U 0 ,V 0 )it holds ( U 0 , V 0 )

Original: ∈ D. Let E be the connected component of D which contains ( U 0 , V 0 ) ,
Revised: ∈ D. Let E be the connected component of D which contains ( U 0 , V 0 ) ,

Original: that is, ( U 0 , V 0 ) ∈
Revised: that is, ( U 0 , V 0 ) ∈

Original: E ⊆ D .
Revised: E ⊆ D .

Original: It holds
Revised: It holds

Original: E ⊂ B (cid:0) ( U 0 , V 0 ) , r (cid:1) as Φ( U, V ) <
Revised: E ⊂ B (cid:0) ( U 0 , V 0 ) , r (cid:1) as Φ( U, V ) <

Original: α +Φ( U 0 ,V 0 )<
Revised: α +Φ( U 0 ,V 0 )<

Original: α for all ( U, V ) ∈ E .
Revised: α for all ( U, V ) ∈ E .

Original: Moreover, Φ( U, V ) ≥ Φ( U 0 , V 0 ) > 0 for all ( U, V ) ∈ E and thus Φ can not be made arbitrarily small on a connected component of a sublevel set of Φ and thus E would be a bad local valley which contradicts 3.3.2.
Revised: Moreover, Φ( U, V ) ≥ Φ( U 0 , V 0 ) > 0 for all ( U, V ) ∈ E and thus Φ can not be made arbitrarily small on a connected component of a sublevel set of Φ and thus E would be a bad local valley which contradicts 3.3.2.

Original: Suppose by contradiction that ( U, V ) is a local maximum.
Revised: Suppose by contradiction that ( U, V ) is a local maximum.

Original: Then the Hessian of Φ is negative semi-deﬁnite.
Revised: Then the Hessian of Φ is negative semi-deﬁnite.

Original: However, as principal submatrices of negative semi-deﬁnite matrices are again negative semi-deﬁnite, then also the Hessian of Φ w.r.t V must be negative semideﬁnite.
Revised: However, as principal submatrices of negative semi-deﬁnite matrices are again negative semi-deﬁnite, then also the Hessian of Φ w.r.t V must be negative semideﬁnite.

Original: However, Φ is always convex in V and thus its Hessian restricted to V is positive semi-deﬁnite.
Revised: However, Φ is always convex in V and thus its Hessian restricted to V is positive semi-deﬁnite.

Original: The only matrix which is both p.s.d.
Revised: The only matrix which is both p.s.d.

Original: and n.s.d.
Revised: and n.s.d.

Original: is the zero matrix.
Revised: is the zero matrix.

Original: It follows that ∇ 2 V Φ( U, V )
Revised: It follows that ∇ 2 V Φ( U, V )

Original: = 0 .
Revised: = 0 .

Original: One can easily show that ∇ 2 V : j Φ =
Revised: One can easily show that ∇ 2 V : j Φ =

Original: N (cid:88) i =
Revised: N (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: From Assumption 3.1 it holds that there exists j ∈ [ N ] s.t.
Revised: From Assumption 3.1 it holds that there exists j ∈ [ N ] s.t.

Original: σ p j is strictly positive, and thus some entries of Ψ i : must be strictly positive.
Revised: σ p j is strictly positive, and thus some entries of Ψ i : must be strictly positive.

Original: Moreover, one has
Revised: Moreover, one has

Original: [Equation]
Revised: [Equation]

Original: It follows that some entries of ∇ 2 V : j Φ must be strictly positive.
Revised: It follows that some entries of ∇ 2 V : j Φ must be strictly positive.

Original: Thus ∇ 2 V : j Φ cannot be identically zero, leading to a contradiction.
Revised: Thus ∇ 2 V : j Φ cannot be identically zero, leading to a contradiction.

Original: Therefore Φ has no local maximum.
Revised: Therefore Φ has no local maximum.

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.4 shows that there are inﬁnitely many solutions which achieve zero training error, and the loss landscape is nice in the sense that from any point in the parameter space there exists a continuous path that drives the loss arbitrarily close to zero (and thus a solution with zero training error) on which the loss is non-increasing.
Revised: Theorem 3.4 shows that there are inﬁnitely many solutions which achieve zero training error, and the loss landscape is nice in the sense that from any point in the parameter space there exists a continuous path that drives the loss arbitrarily close to zero (and thus a solution with zero training error) on which the loss is non-increasing.

Original: While the networks are over-parameterized, we show in the next Section 4 that the modiﬁcation of standard networks so that they fulﬁll our conditions leads nevertheless to good generalization performance, often even better than the original network.
Revised: While the networks are over-parameterized, we show in the next Section 4 that the modiﬁcation of standard networks so that they fulﬁll our conditions leads nevertheless to good generalization performance, often even better than the original network.

Original: We would like to note that the proof of
Revised: We would like to note that the proof of

Original: Theorem 3.4 also suggests a different algorithm to achieve zero training error: one initializes all weights, except the weights to the output layer, randomly (e.g. Gaussian weights), denoted as U , and then just solves the linear system Ψ( U )
Revised: Theorem 3.4 also suggests a different algorithm to achieve zero training error: one initializes all weights, except the weights to the output layer, randomly (e.g. Gaussian weights), denoted as U , and then just solves the linear system Ψ( U )

Original: V =
Revised: V =

Original: Y to obtain the weights V to the output layer.
Revised: Y to obtain the weights V to the output layer.

Original: Basically, this algorithm uses the network as a random feature generator and ﬁts the last layer directly to achieve zero training error.
Revised: Basically, this algorithm uses the network as a random feature generator and ﬁts the last layer directly to achieve zero training error.

Original: The algorithm is successful with probability 1 due to Lemma 3.2.
Revised: The algorithm is successful with probability 1 due to Lemma 3.2.

Original: Note that from a solution with zero training error one can drive the cross-entropy loss to zero by upscaling to inﬁnity but this does not change the classiﬁer.
Revised: Note that from a solution with zero training error one can drive the cross-entropy loss to zero by upscaling to inﬁnity but this does not change the classiﬁer.

Original: We will see, that this simple algorithm shows bad generalization performance and overﬁtting, whereas training the full network with SGD leads to good generalization performance.
Revised: We will see, that this simple algorithm shows bad generalization performance and overﬁtting, whereas training the full network with SGD leads to good generalization performance.

Original: This might seem counterintuitive as our networks have more parameters than the original networks but is inline with recent observations in Zhang et al.
Revised: This might seem counter-intuitive as our networks have more parameters than the original networks but is inline with recent observations in Zhang et al.

Original: (2017) that state-of-the art networks, also heavily over-parameterized, can ﬁt even random labels but still generalize well on the original problem.
Revised: (2017) that state-of-the art networks, also heavily over-parameterized, can ﬁt even random labels but still generalize well on the original problem.

Original: Due to this qualitative difference of SGD and the simple algorithm which both are able to ﬁnd solutions with zero training error, we think that our class of networks is an ideal test bed to study the implicit regularization/bias of SGD, see e.g. Soudry et al.
Revised: Due to this qualitative difference of SGD and the simple algorithm which both are able to ﬁnd solutions with zero training error, we think that our class of networks is an ideal test bed to study the implicit regularization/bias of SGD, see e.g. Soudry et al.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The main purpose of this section is to investigate the generalization ability of practical neural networks with skip-connections added to the output layer to fulﬁll Assumption 3.1.
Revised: The main purpose of this section is to investigate the generalization ability of practical neural networks with skip-connections added to the output layer to fulﬁll Assumption 3.1.

Original: Datasets.
Revised: Datasets.

Original: We consider MNIST and CIFAR10 datasets.
Revised: We consider MNIST and CIFAR10 datasets.

Original: MNIST contains 5 .
Revised: MNIST contains 5 .

Original: 5 × 10 4 training samples and 10 4 test samples, and CIFAR10 has 5 × 10 4 training samples and 10 4 test samples.
Revised: 5 × 10 4 training samples and 10 4 test samples, and CIFAR10 has 5 × 10 4 training samples and 10 4 test samples.

Original: We do not use any data pre-processing  in all of our experiments.
Revised: We do not use any data pre-processing nor data-augmentation in all of our experiments.

Original: For every experiment on CIFAR10 described below, we consider both settings with and without data-augmentation.

Original: For data-augmentation, we follow the procedure as described in Zagoruyko & Komodakis (2016) by considering random crops of size 32 × 32 after 4 pixel padding on each side of the training images and random horizontal ﬂips with probability 0 .

Original: Network architectures.
Revised: Network architectures.

Original: For MNIST, we use a plain CNN architecture with 13 layers, denoted as
Revised: For MNIST, we use a plain CNN architecture with 13 layers, denoted as

Original: CNN13 (see Table 3 in the appendix for more details about this architecture).
Revised: CNN13 (see Table 3 in the appendix for more details about this architecture).

Original: For CIFAR10 we use
Revised: For CIFAR10 we use

Original: VGG11, VGG13, VGG16 (Simonyan & Zisserman, 2015) and DenseNet121 (Huang et al., 2017).
Revised: VGG11, VGG13, VGG16 (Simonyan & Zisserman, 2015) and DenseNet121 (Huang et al., 2017).

Original: As the VGG models were originally proposed for ImageNet and have very large fully connected layers, we adapted these layers for CIFAR10 by reducing their width from 4096 to 128 .
Revised: As the VGG models were originally proposed for ImageNet and have very large fully connected layers, we adapted these layers for CIFAR10 by reducing their width from 4096 to 128 .

Original: For each given network, we create the corresponding skip-networks by adding skip-connections to the output so that our condition M ≥ N from the main theorem is satisﬁed.
Revised: For each given network, we create the corresponding skip-networks by adding skip-connections to the output so that our condition M ≥ N from the main theorem is satisﬁed.

Original: In particular, we aggregate all neurons of all the hidden layers in a pool and randomly choose from there a subset of N neurons to be connected to the output layer (see e.g. Figure 2 for an illustration).
Revised: In particular, we aggregate all neurons of all the hidden layers in a pool and randomly choose from there a subset of N neurons to be connected to the output layer (see e.g. Figure 2 for an illustration).

Original: As existing network architectures have a large number of feature maps per layer, the total number of neurons is often very large compared to number of training samples, thus it is easy to choose from there a subset of N neurons to connect to the output.
Revised: As existing network architectures have a large number of feature maps per layer, the total number of neurons is often very large compared to number of training samples, thus it is easy to choose from there a subset of N neurons to connect to the output.

Original: In the following, we test both sigmoid and softplus activation function ( γ = 20 ) for each network architecture and their skip-variants.
Revised: In the following, we test both sigmoid and softplus activation function ( γ = 20 ) for each network architecture and their skip-variants.

Original: We use the standard cross-entropy loss and train all models with SGD+Nesterov momentum for 300 epochs.
Revised: We use the standard cross-entropy loss and train all models with SGD+Nesterov momentum for 300 epochs.

Original: The initial learning rate is set to 0 .
Revised: The initial learning rate is set to 0 .

Original: 1 for
Revised: 1 for

Original: Densenet121 and 0 .
Revised: Densenet121 and 0 .

Original: 01 for the other architectures.
Revised: 01 for the other architectures.

Original: Following Huang et al.
Revised: Following Huang et al.

Original: (2017), we also divide the learning rate by 10 after 50% and 75% of the total number of training epochs.
Revised: (2017), we also divide the learning rate by 10 after 50% and 75% of the total number of training epochs.

Original: Note that we do not use any explicit regularization like weight decay or dropout.
Revised: Note that we do not use any explicit regularization like weight decay or dropout.

Original: The main goal of our experiments is to investigate the inﬂuence of the additional skip-connections to the output layer on the generalization performance.
Revised: The main goal of our experiments is to investigate the inﬂuence of the additional skip-connections to the output layer on the generalization performance.

Original: We report the test accuracy for the original models and the ones with skip-connections to the output layer.
Revised: We report the test accuracy for the original models and the ones with skip-connections to the output layer.

Original: For the latter one we have two different algorithms: standard SGD for training the full network as described above (SGD) and the randomized procedure (rand).
Revised: For the latter one we have two different algorithms: standard SGD for training the full network as described above (SGD) and the randomized procedure (rand).

Original: The latter one uses a slight variant of the simple algorithm described at the end of the last section: randomly initialize the weights of the network U up to the output layer by drawing each of them from a truncated Gaussian distribution with zero mean and variance 2 d where d is the number of weight parameters and the truncation is done after ± 2 standard deviations (standard keras initialization), then use SGD to optimize the weights V for a linear classiﬁer with ﬁxed features Ψ( U ) which is a convex optimization problem.
Revised: The latter one uses a slight variant of the simple algorithm described at the end of the last section: randomly initialize the weights of the network U up to the output layer by drawing each of them from a truncated Gaussian distribution with zero mean and variance 2 d where d is the number of weight parameters and the truncation is done after ± 2 standard deviations (standard keras initialization), then use SGD to optimize the weights V for a linear classiﬁer with ﬁxed features Ψ( U ) which is a convex optimization problem.

Original: Note that the rand algorithm cannot be used with data augmentation in a straightforward way and thus we skip it for this part.

Original: Our experimental results are summarized in Table 1 for MNIST and Table 2 for CIFAR10.
Revised: Our experimental results are summarized in Table 1 for MNIST and Table 2 for CIFAR10.

Original: For skip-models, we report mean and standard deviation over 8 random choices of the subset of N neurons connected to the output.[Table] [Table]
Revised: For skip-models, we report mean and standard deviation over 8 random choices of the subset of N neurons connected to the output.

Original: Discussion of results.
Revised: Discussion of results.

Original: First of all, we note that adding skip connections to the output improves the test accuracy in almost all networks (with the exception of Densenet121) when the full network is trained with SGD.
Revised: First of all, we note that adding skip connections to the output improves the test accuracy in almost all networks (with the exception of Densenet121) when the full network is trained with SGD.

Original: In particular, for the sigmoid activation function the skip connections allow for all models except Densenet121 to get reasonable performance whereas training the original model fails.
Revised: In particular, for the sigmoid activation function the skip connections allow for all models except Densenet121 to get reasonable performance whereas training the original model fails.

Original: This effect can be directly related to our result of Theorem 3.4 that the loss landscape of skip-networks has no bad local valley and thus it is not difﬁcult to reach a solution with zero training error (see the appendix for more detailed discussions).
Revised: This effect can be directly related to our result of Theorem 3.4 that the loss landscape of skip-networks has no bad local valley and thus it is not difﬁcult to reach a solution with zero training [Table] [Table] data-augmentation).

Original: The exception is Densenet121 which gets already good performance for the sigmoid activation function for the original model.
Revised: For each original model A, A-skip denotes the corresponding skip-model in which a subset of N hidden neurons “randomly selected” from the hidden layers are connected to the output units. For Densenet121, these neurons are randomly chosen from the ﬁrst dense block. The names in open brackets (rand/SGD) specify how the networks are trained: rand ( U is randomized and ﬁxed while V is learned with SGD), SGD (both U and V are optimized with SGD).

Revised: Additional experimental results with data-augmentation are shown in Table 5 in the appendix.

Revised: error (see Section F in the appendix for more detailed discussions on this issue, as well as Section

Revised: E for a visual example which shows why the skip-models can succeed while the original models fail).

Revised: The exception is Densenet121 which gets already good performance for the sigmoid activation function for the original model.

Original: We think that the reason is that the original Densenet121 architecture has already quite a lot of skip-connections between the hidden layers which thus improves the loss surface already so that the additional connections added to the output units are not necessary anymore.
Revised: We think that the reason is that the original Densenet121 architecture has already quite a lot of skip-connections between the hidden layers which thus improves the loss surface already so that the additional connections added to the output units are not necessary anymore.

Original: The second interesting observation is that we do not see any sign of overﬁtting for the SGD version even though we have increased for all models the number of parameters by adding skip connections to the output layer and we know from Theorem 3.4 that for all the skip-models one can easily achieve zero training error.
Revised: The second interesting observation is that we do not see any sign of overﬁtting for the SGD version even though we have increased for all models the number of parameters by adding skip connections to the output layer and we know from Theorem 3.4 that for all the skip-models one can easily achieve zero training error.

Original: This is in line with the recent observation of Zhang et al.
Revised: This is in line with the recent observation of Zhang et al.

Original: (2017) that modern heavily over-parameterized networks can ﬁt everything (random labels, random input) but nevertheless generalize well on the original training data when trained with SGD.
Revised: (2017) that modern heavily over-parameterized networks can ﬁt everything (random labels, random input) but nevertheless generalize well on the original training data when trained with SGD.

Original: This is currently an active research area to show that SGD has some implicit bias (Neyshabur et al., 2017; Brutzkus et al., 2018; Soudry et al., 2018) which leads to a kind of regularization effect similar to the linear least squares problem where SGD converges to the minimum norm solution.
Revised: This is currently an active research area to show that SGD has some implicit bias (Neyshabur et al., 2017; Brutzkus et al., 2018; Soudry et al., 2018) which leads to a kind of regularization effect similar to the linear least squares problem where SGD converges to the minimum norm solution.

Original: Our results conﬁrm that there seems an implicit bias as we see a strong contrast to the (rand) results obtained by using the network as a random feature generator and just ﬁtting the last layer which also leads to solutions with zero training error with probability 1 as shown in Lemma 3.2 and the proof of Theorem 3.4.
Revised: Our results conﬁrm that there is an implicit bias as we see a strong contrast to the (skip-rand) results obtained by using the network as a random feature generator and just ﬁtting the connections to the output units ( i.e. V ) which also leads to solutions with zero training error with probability 1 as shown in Lemma 3.2 and the proof of Theorem 3.4.

Original: For this version we see that the test accuracy gets worse as one is moving from simpler networks (VGG11) to more complex ones (VGG16 and Densenet121) which is a sign of overﬁtting.
Revised: For this version we see that the test accuracy gets worse as one is moving from simpler networks (VGG11) to more complex ones (VGG16 and Densenet121) which is a sign of overﬁtting.

Original: Thus we think that our class of networks is also an interesting test bed to understand the implicit regularization effect of SGD.
Revised: Thus we think that our class of networks is also an interesting test bed to understand the implicit regularization effect of SGD.

Original: It seems that SGD selects from the inﬁnite pool of solutions with zero training error one which generalizes well, whereas the randomized feature generator selects one with much worse generalization performance.
Revised: It seems that SGD selects from the inﬁnite pool of solutions with zero training error one which generalizes well, whereas the randomized feature generator selects one with much worse generalization performance.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: In the literature, many interesting theoretical results have been developed on the loss surface of neural networks Yu & Chen (1995); Haeffele & Vidal (2017); Choromanska et al.
Revised: In the literature, many interesting theoretical results have been developed on the loss surface of neural networks Yu & Chen (1995); Haeffele & Vidal (2017); Choromanska et al.

Original: ; Kawaguchi (2016);
Revised: ; Kawaguchi (2016);

Original: Safran & Shamir (2016); Hardt & Ma (2017);
Revised: Safran & Shamir (2016); Hardt & Ma (2017);

Original: Yun et al.
Revised: Yun et al.

Original: ; Lu & Kawaguchi (2017); Venturi et al.
Revised: ; Lu & Kawaguchi (2017); Venturi et al.

Original: ; Liang et al.
Revised: ; Liang et al.

Original: (2018b)
Revised: (2018b)

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: ; Nouiehed & Razaviyayn (2018).
Revised: ; Nouiehed & Razaviyayn (2018).

Original: The behavior of SGD for the minimization of training objective has been also analyzed for various settings (Andoni et al., 2014; Sedghi & Anandkumar, 2015; Janzamin et al., 2016; Gautier et al., 2016; Brutzkus &
Revised: The behavior of SGD for the minimization of training objective has been also analyzed for various settings (Andoni et al., 2014; Sedghi & Anandkumar, 2015; Janzamin et al., 2016; Gautier et al., 2016; Brutzkus &

Original: Globerson, 2017; Soltanolkotabi, 2017; Soudry & Hoffer, 2017; Zhong et al., 2017; Tian, 2017; Du et al., 2018; Wang et al., 2018) to name a few.
Revised: Globerson, 2017; Soltanolkotabi, 2017; Soudry & Hoffer, 2017; Zhong et al., 2017; Tian, 2017; Du et al., 2018; Wang et al., 2018) to name a few.

Original: Most of current results are however limited to shallow networks (one hidden layer), deep linear networks and/or making simplifying assumptions on the architecture or the distribution of training data.
Revised: Most of current results are however limited to shallow networks (one hidden layer), deep linear networks and/or making simplifying assumptions on the architecture or the distribution of training data.

Original: An interesting recent exception is Liang et al.
Revised: An interesting recent exception is Liang et al.

Original: (2018a) where they show that for binary classiﬁcation one neuron with a skip-connection to the output layer and exponential activation function is enough to eliminate all bad local minima under mild conditions on the loss function.
Revised: (2018a) where they show that for binary classiﬁcation one neuron with a skip-connection to the output layer and exponential activation function is enough to eliminate all bad local minima under mild conditions on the loss function.

Original: More closely related in terms of the setting are (Nguyen & Hein, 2017; 2018) where they study the loss surface of fully connected and convolutional networks if one of the layers has more neurons than the number of training samples for the standard multi-class problem.
Revised: More closely related in terms of the setting are (Nguyen & Hein, 2017; 2018) where they study the loss surface of fully connected and convolutional networks if one of the layers has more neurons than the number of training samples for the standard multi-class problem.

Original: However, the presented results are stronger as we show that our networks do not have any suboptimal local valley or strict local minima and there is less over-parameterization if the number of classes is small.
Revised: However, the presented results are stronger as we show that our networks do not have any suboptimal local valley or strict local minima and there is less over-parameterization if the number of classes is small.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have identiﬁed a class of deep neural networks whose loss landscape has no bad local valleys.
Revised: We have identiﬁed a class of deep neural networks whose loss landscape has no bad local valleys.

Original: While our networks are over-parameterized and can easily achieve zero training error, they generalize well in practice when trained with SGD.
Revised: While our networks are over-parameterized and can easily achieve zero training error, they generalize well in practice when trained with SGD.

Original: Interestingly, a simple different algorithm using the network as random feature generator also achieves zero training error but has signiﬁcantly worse generalization performance.
Revised: Interestingly, a simple different algorithm using the network as random feature generator also achieves zero training error but has signiﬁcantly worse generalization performance.

Original: Thus we think that our class of models is an interesting test bed for studying the implicit regularization effect of SGD.
Revised: Thus we think that our class of models is an interesting test bed for studying the implicit regularization effect of SGD.

Original: A M ATHEMATICAL T OOLS
Revised: A M ATHEMATICAL T OOLS

Original: In the proof of Lemma 3.2 we make use of the following property of analytic functions.
Revised: In the proof of Lemma 3.2 we make use of the following property of analytic functions.

Original: Lemma A.1 (Nguyen, 2015; Mityagin, 2015)
Revised: Lemma A.1 (Nguyen, 2015; Mityagin, 2015)

Original: If f : R n → R is a real analytic function which is not identically zero then the set { x ∈ R n | f ( x ) = 0 } has Lebesgue measure zero.
Revised: If f : R n → R is a real analytic function which is not identically zero then the set { x ∈ R n | f ( x ) = 0 } has Lebesgue measure zero.

Original: We recall the following standard result from topology (see e.g. Apostol (1974), Theorem 4.23, p. 82), which is used in the proof of Theorem 3.4.
Revised: We recall the following standard result from topology (see e.g. Apostol (1974), Theorem 4.23, p. 82), which is used in the proof of Theorem 3.4.

Original: Proposition A.2 Let f : R m → R n be a continuous function.
Revised: Proposition A.2 Let f : R m → R n be a continuous function.

Original: If U ⊆ R n is an open set then f − 1 ( U ) is also open.
Revised: If U ⊆ R n is an open set then f − 1 ( U ) is also open.

Original: B P ROOF OF L EMMA 3. 
Revised: B P ROOF OF L EMMA 3.

Original: Proof: We assume w.l.o.g. that { p 1 , . . .
Revised: Proof: We assume w.l.o.g. that { p 1 , . . .

Original: , p N } is a subset of the neurons with skip connections to the output layer and satisfy Assumption 3.1.
Revised: , p N } is a subset of the neurons with skip connections to the output layer and satisfy Assumption 3.1.

Original: In the following, we will show that there exists a weight conﬁguration U such that the submatrix Ψ 1: N, 1: N has full rank.
Revised: In the following, we will show that there exists a weight conﬁguration U such that the submatrix Ψ 1: N, 1: N has full rank.

Original: Using then that the determinant is an analytic function together with Lemma A.1, we will conclude that the set of weight conﬁgurations U such that Ψ has not full rank has Lebesgue measure zero.
Revised: Using then that the determinant is an analytic function together with Lemma A.1, we will conclude that the set of weight conﬁgurations U such that Ψ has not full rank has Lebesgue measure zero.

Original: We remind that all the hidden units in the network are indexed from the ﬁrst hidden layer till the higher layers as d + 1 , . . .
Revised: We remind that all the hidden units in the network are indexed from the ﬁrst hidden layer till the higher layers as d + 1 , . . .

Original: , d + H.
Revised: , d + H.

Original: For every hidden neuron j ∈ [ d + 1 , d + H ] , u j denotes the associated weight vector
Revised: For every hidden neuron j ∈ [ d + 1 , d + H ] , u j denotes the associated weight vector

Original: [Equation]
Revised: [Equation]

Original: where in( j ) = the set of incoming units to unit j.
Revised: where in( j ) = the set of incoming units to unit j.

Original: Let n 1 be the number of units of the ﬁrst hidden layer.
Revised: Let n 1 be the number of units of the ﬁrst hidden layer.

Original: For every neuron j from the ﬁrst hidden layer, let us deﬁne the pre-activation output g j ,
Revised: For every neuron j from the ﬁrst hidden layer, let us deﬁne the pre-activation output g j ,

Original: [Equation]
Revised: [Equation]

Original: Due to Assumptions 3.1 (condition 3), we can always choose the weights { u d +1 , . .
Revised: Due to Assumptions 3.1 (condition 3), we can always choose the weights { u d +1 , . .

Original: u d + n 1 } so that the output of every neuron in the ﬁrst layer is distinct for different training samples, that is g j ( x i ) (cid:54) =
Revised: u d + n 1 } so that the output of every neuron in the ﬁrst layer is distinct for different training samples, that is g j ( x i ) (cid:54) =

Original: g j ( x i (cid:48) ) for every j ∈ [ d + 1 , d + n 1 ] and i (cid:54) = i (cid:48) .
Revised: g j ( x i (cid:48) ) for every j ∈ [ d + 1 , d + n 1 ] and i (cid:54) = i (cid:48) .

Original: For every neuron j ∈ [ d + n 1 + 1 , d + H ] in the higher layers we choose the weight vector
Revised: For every neuron j ∈ [ d + n 1 + 1 , d + H ] in the higher layers we choose the weight vector

Original: u j such that it has exactly one 1 and 0 elsewhere.
Revised: u j such that it has exactly one 1 and 0 elsewhere.

Original: According to our deﬁnition of network in Section 2, the weight vectors of neurons of the same layer need not have the same dimension, but any subgroup of these neurons can still have shared weights as long as the dimensions among them agree.
Revised: According to our deﬁnition of network in Section 2, the weight vectors of neurons of the same layer need not have the same dimension, but any subgroup of these neurons can still have shared weights as long as the dimensions among them agree.

Original: Thus the above choice of u is always possible.
Revised: Thus the above choice of u is always possible.

Original: In the following, let c ( j ) denote the neuron below j such that u c ( j ) →
Revised: In the following, let c ( j ) denote the neuron below j such that u c ( j ) →

Original: j = 1 .
Revised: j = 1 .

Original: This leads to
Revised: This leads to

Original: [Equation]
Revised: [Equation]

Original: Let α :=
Revised: Let α :=

Original: ( α d +1 , . .
Revised: ( α d +1 , . .

Original: . , α d + H ) be a tuple of positive scalars.
Revised: . , α d + H ) be a tuple of positive scalars.

Original: Let β ∈ R such that σ p j ( β ) (cid:54) = 

Revised: Let β ∈ R such that σ p j ( β ) (cid:54) =

Original: for every j ∈ [ N ] .
Revised: for every j ∈ [ N ] .

Original: We consider a family of conﬁgurations of network parameters of the form ( α j u j , b j ) d + H j = d
Revised: We consider a family of conﬁgurations of network parameters of the form ( α j u j , b j ) d + H j = d

Original: +1 , where the biases are chosen as
Revised: +1 , where the biases are chosen as

Original: [Equation]
Revised: [Equation]

Original: Note that the assignment of biases can be done via a forward pass through the network.
Revised: Note that the assignment of biases can be done via a forward pass through the network.

Original: By the above choice of biases and our deﬁnition of neurons in Section 2, we have
Revised: By the above choice of biases and our deﬁnition of neurons in Section 2, we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: One notes that the output of every skip-connection neuron p j is given by the ﬁrst equation if p j lies on the ﬁrst layer and by the second equation if p j lies on higher layers.
Revised: One notes that the output of every skip-connection neuron p j is given by the ﬁrst equation if p j lies on the ﬁrst layer and by the second equation if p j lies on higher layers.

Original: In the following, to reduce notational complexity we make a convention that: f c ( p j ) =
Revised: In the following, to reduce notational complexity we make a convention that: f c ( p j ) =

Original: g p j for every p j lies on the ﬁrst layer.
Revised: g p j for every p j lies on the ﬁrst layer.

Original: This allows us to use the second equation for every skip-connection neuron, that is,
Revised: This allows us to use the second equation for every skip-connection neuron, that is,

Original: [Equation]
Revised: [Equation]

Original: Now, since α > 0 and all activation functions are strictly increasing by Assumption 3.1, one can easily show from the above recursive deﬁnitions that if p j is a skip-connection neuron which does not lie on the ﬁrst hidden layer then one has the relation: f c ( p j ) ( x i ) <
Revised: Now, since α > 0 and all activation functions are strictly increasing by Assumption 3.1, one can easily show from the above recursive deﬁnitions that if p j is a skip-connection neuron which does not lie on the ﬁrst hidden layer then one has the relation: f c ( p j ) ( x i ) <

Original: f c ( p j ) ( x j ) if and only if g q j ( x i ) <
Revised: f c ( p j ) ( x j ) if and only if g q j ( x i ) <

Original: g q j ( x j ) , where q j is some neuron in the ﬁrst hidden layer.
Revised: g q j ( x j ) , where q j is some neuron in the ﬁrst hidden layer.

Original: This means if one sorts the elements of the set (cid:8) f c
Revised: This means if one sorts the elements of the set (cid:8) f c

Original: ( p j ) ( x 1 ) , . . .
Revised: ( p j ) ( x 1 ) , . . .

Original: , f c ( p j ) ( x N ) (cid:9) in increasing order then for every positive tuple α , the order is fully determined by the corresponding order of (cid:8) g q j ( x 1 ) , . . .
Revised: , f c ( p j ) ( x N ) (cid:9) in increasing order then for every positive tuple α , the order is fully determined by the corresponding order of (cid:8) g q j ( x 1 ) , . . .

Original: , g q j ( x N ) (cid:9) for some neuron q j in the ﬁrst layer.
Revised: , g q j ( x N ) (cid:9) for some neuron q j in the ﬁrst layer.

Original: Note that this order can be different for different neurons q j in the ﬁrst layer, and thus can be different for different skip-connection neurons p j .
Revised: Note that this order can be different for different neurons q j in the ﬁrst layer, and thus can be different for different skip-connection neurons p j .

Original: Let π be a permutation such that it holds for every j = 1 , 2 , . .
Revised: Let π be a permutation such that it holds for every j = 1 , 2 , . .

Original: N
Revised: N

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: It follows from above that π is fully determined by the values of g at the ﬁrst layer.
Revised: It follows from above that π is fully determined by the values of g at the ﬁrst layer.

Original: By deﬁnition one has f c ( p j ) ( x π i ) <
Revised: By deﬁnition one has f c ( p j ) ( x π i ) <

Original: f c ( p j ) ( x π j ) for every i > j. Since π is independent of every positive tuple α and fully determined by the values of g , it can be ﬁxed in the beginning.
Revised: f c ( p j ) ( x π j ) for every i > j. Since π is independent of every positive tuple α and fully determined by the values of g , it can be ﬁxed in the beginning.

Original: One can assume w.l.o.g. that π is the identity permutation as otherwise one can reorder the training samples according to π so that the rank of Ψ does not change.
Revised: One can assume w.l.o.g. that π is the identity permutation as otherwise one can reorder the training samples according to π so that the rank of Ψ does not change.

Original: Thus it holds for every α > 0 that
Revised: Thus it holds for every α > 0 that

Original: [Equation]
Revised: [Equation]

Original: Now, we are ready to show that there exists a positive tuple α for which Ψ has full rank.
Revised: Now, we are ready to show that there exists a positive tuple α for which Ψ has full rank.

Original: We consider two cases of the activation functions of skip-connection neurons as stated in Assumption 3.1:
Revised: We consider two cases of the activation functions of skip-connection neurons as stated in Assumption 3.1:

Original: • In the ﬁrst case, the activation functions σ p j : R → R for every j ∈ [ N ] are strictly increasing, bounded and lim t →−∞ σ p j ( t ) = 0 .
Revised: • In the ﬁrst case, the activation functions σ p j : R → R for every j ∈ [ N ] are strictly increasing, bounded and lim t →−∞ σ p j ( t ) = 0 .

Original: In the following, let l ( j ) denote the layer index of the hidden unit j .
Revised: In the following, let l ( j ) denote the layer index of the hidden unit j .

Original: For every hidden unit j ∈ { d + 1 , . . .
Revised: For every hidden unit j ∈ { d + 1 , . . .

Original: , d + H } we set α j to be the maximum of certain bounds (explained later in (10)) associated to all skip-connection neurons p k lying on the same layer, that is,
Revised: , d + H } we set α j to be the maximum of certain bounds (explained later in (10)) associated to all skip-connection neurons p k lying on the same layer, that is,

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) > 0 is an arbitrarily small constant which will be speciﬁed later.
Revised: where (cid:15) > 0 is an arbitrarily small constant which will be speciﬁed later.

Original: There are a few remarks we want to make for Eq.
Revised: There are a few remarks we want to make for Eq.

Original: (9) before proceeding with our proof.
Revised: (9) before proceeding with our proof.

Original: First, the second term in (9) can be empty if there is no skip-connection unit p k which lies on the same layer as unit j , in which case α j is simply set to 1 .
Revised: First, the second term in (9) can be empty if there is no skip-connection unit p k which lies on the same layer as unit j , in which case α j is simply set to 1 .

Original: Second, α j ’s are well-deﬁned by constructing the values f c ( p k ) ( x r ) , r = 1 , . . .
Revised: Second, α j ’s are well-deﬁned by constructing the values f c ( p k ) ( x r ) , r = 1 , . . .

Original: , N by a forward pass through the network (note that the network is a directed, acyclic graph; in particular, in the formula of α j , one has l ( c ( p k )) <
Revised: , N by a forward pass through the network (note that the network is a directed, acyclic graph; in particular, in the formula of α j , one has l ( c ( p k )) <

Original: l ( p k ) =
Revised: l ( p k ) =

Original: l ( j ) and thus the computation of α j is feasible given the values of hidden units lying below the layer of unit j , namely f c ( p k ) ).
Revised: l ( j ) and thus the computation of α j is feasible given the values of hidden units lying below the layer of unit j , namely f c ( p k ) ).

Original: Third, if j and j (cid:48) are two neurons from the same layer, i.e. l ( j )
Revised: Third, if j and j (cid:48) are two neurons from the same layer, i.e. l ( j )

Original: =
Revised: =

Original: l ( j (cid:48) ) , then it follows from (9) that α j = α j (cid:48) , meaning that their corresponding weight vectors are scaled by the same factor, thus any potential weight sharing conditions imposed on these neurons can still be satisﬁed.
Revised: l ( j (cid:48) ) , then it follows from (9) that α j = α j (cid:48) , meaning that their corresponding weight vectors are scaled by the same factor, thus any potential weight sharing conditions imposed on these neurons can still be satisﬁed.

Original: The main idea of choosing the above values of α is to obtain
Revised: The main idea of choosing the above values of α is to obtain

Original: [Equation]
Revised: [Equation]

Original: To see this, one ﬁrst observes that the inequality (8) holds for the constructed values of α since they are all positive.
Revised: To see this, one ﬁrst observes that the inequality (8) holds for the constructed values of α since they are all positive.

Original: From (9) it holds for every skip-connection unit p j that
Revised: From (9) it holds for every skip-connection unit p j that

Original: [Equation]
Revised: [Equation]

Original: which combined with (8) leads to
Revised: which combined with (8) leads to

Original: [Equation]
Revised: [Equation]

Original: and thus using (6) we obtain (10).
Revised: and thus using (6) we obtain (10).

Original: Coming back to the main proof of the lemma, since σ p j ( j ∈ [ N ]) are bounded there exists a ﬁnite positive constant C such that it holds that
Revised: Coming back to the main proof of the lemma, since σ p j ( j ∈ [ N ]) are bounded there exists a ﬁnite positive constant C such that it holds that

Original: [Equation]
Revised: [Equation]

Original: By the Leibniz-formula one has
Revised: By the Leibniz-formula one has

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  where S N is the set of all N ! permutations of the set { 1 , . .
Revised: 12) where S N is the set of all N ! permutations of the set { 1 , . .

Original: . , N } and γ is the identity permutation.
Revised: . , N } and γ is the identity permutation.

Original: Now, one observes that for every permutation π (cid:54) =
Revised: Now, one observes that for every permutation π (cid:54) =

Original: γ , there always exists at least one component j where π ( j ) >
Revised: γ , there always exists at least one component j where π ( j ) >

Original: j in which case it follows from (10) and (11)
Revised: j in which case it follows from (10) and (11)

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: By choosing (cid:15) =
Revised: By choosing (cid:15) =

Original: [Equation]
Revised: [Equation]

Original: , we get that det
Revised: , we get that det

Original: [Equation]
Revised: [Equation]

Original: and thus Ψ has full rank.
Revised: and thus Ψ has full rank.

Original: • In the second case we consider the softplus activation function which satisﬁes our Assumption 3.1 that there exists a backward path from every skip-connection neuron p j to the ﬁrst hidden layer s.t.
Revised: • In the second case we consider the softplus activation function which satisﬁes our Assumption 3.1 that there exists a backward path from every skip-connection neuron p j to the ﬁrst hidden layer s.t.

Original: on this path there is no neuron which has skip-connections to the output or shared weights with other skip-connection neurons.
Revised: on this path there is no neuron which has skip-connections to the output or shared weights with other skip-connection neurons.

Original: We choose all the weights and biases similarly to the ﬁrst case.
Revised: We choose all the weights and biases similarly to the ﬁrst case.

Original: The only difference is that for every skip-connection neuron p j (1 ≤ j ≤ N ) , the position of 1 in its weight vector u p j is chosen s.t.
Revised: The only difference is that for every skip-connection neuron p j (1 ≤ j ≤ N ) , the position of 1 in its weight vector u p j is chosen s

Revised: [Equation]

Original:  the value of neuron p j is determined by the ﬁrst neuron on the corresponding backward path as stated in Assumption 3.1, that is,
Revised: t. the value of neuron p j is determined by the ﬁrst neuron on the corresponding backward path as stated in Assumption 3.1, that is,

Original: [Equation]
Revised: [Equation]

Original: For skip-connection neurons we set all { α p 1 , . . .
Revised: For skip-connection neurons we set all { α p 1 , . . .

Original: , α p N } to some scalar variable α , and for non-skip connection neurons j we set α j = 1 .
Revised: , α p N } to some scalar variable α , and for non-skip connection neurons j we set α j = 1 .

Original: From (6) and equations of (5) we have
Revised: From (6) and equations of (5) we have

Original: [Equation]
Revised: [Equation]

Original: Note that with above construction of u and α , the only case where our weight sharing conditions can be potentially violated is between a skip-connection neuron ( α j = α ) with a neuron on a backward path ( α j = 1 ).
Revised: Note that with above construction of u and α , the only case where our weight sharing conditions can be potentially violated is between a skip-connection neuron ( α j = α ) with a neuron on a backward path ( α j = 1 ).

Original: However, this is not possible because our assumption in this case states that there is no weight sharing between a skip-connection neuron and a neuron on one of the backward paths.
Revised: However, this is not possible because our assumption in this case states that there is no weight sharing between a skip-connection neuron and a neuron on one of the backward paths.

Original: Next, by our assumption the recursive backward path c ( k ) ( p j ) does not contain any skipconnection unit and thus will eventually end up at some neuron q j ∈ [ d + 1 , d + n 1 ] in the ﬁrst hidden layer after some ﬁnite number of steps.
Revised: Next, by our assumption the recursive backward path c ( k ) ( p j ) does not contain any skipconnection unit and thus will eventually end up at some neuron q j ∈ [ d + 1 , d + n 1 ] in the ﬁrst hidden layer after some ﬁnite number of steps.

Original: Thus we can write for every j ∈ [ N ]
Revised: Thus we can write for every j ∈ [ N ]

Original: [Equation]
Revised: [Equation]

Original: , where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Moreover, we have from (8) that f c ( p j ) ( x i ) <
Revised: Moreover, we have from (8) that f c ( p j ) ( x i ) <

Original: f c ( p j ) ( x j ) for every i > j. Note that softplus fulﬁlls for t < 0 , σ γ ( t ) ≤ 1 γ e γt , whereas for t > 0 one has σ γ ( t ) ≤ 1 γ + t .
Revised: f c ( p j ) ( x j ) for every i > j. Note that softplus fulﬁlls for t < 0 , σ γ ( t ) ≤ 1 γ e γt , whereas for t > 0 one has σ γ ( t ) ≤ 1 γ + t .

Original: The latter property implies σ ( K ) ( t )
Revised: The latter property implies σ ( K ) ( t )

Original: ≤ Kγ + t .
Revised: ≤ Kγ + t .

Original: Finally, this together implies that there exist positive constants
Revised: Finally, this together implies that there exist positive constants

Original: c 1 , c 2 , c 3 , c 4 such that it hods
Revised: c 1 , c 2 , c 3 , c 4 such that it hods

Original: [Equation]
Revised: [Equation]

Original: This can be made arbitrarily small by increasing α .
Revised: This can be made arbitrarily small by increasing α .

Original: Thus we get 
Revised: Thus we get lim α →∞ det(

Original: [Equation]
Revised: [Equation]

Revised: : N

Revised: [Equation]

Revised: [Equation]

Revised: =

Revised: N

Revised: [Equation]

Original: So far, we have shown that there always exist U such that Ψ has full rank.
Revised: So far, we have shown that there always exist U such that Ψ has full rank.

Original: Since every activation function is real analytic by Assumption 3.1, every entry of Ψ is also a real analytic function of the network parameters where Ψ depends on.
Revised: Since every activation function is real analytic by Assumption 3.1, every entry of Ψ is also a real analytic function of the network parameters where Ψ depends on.

Original: The set of low rank matrices Ψ can be characterized by a system of equations such that all the (cid:0) MN (cid:1) determinants of all N × N sub-matrices of Ψ are zero.
Revised: The set of low rank matrices Ψ can be characterized by a system of equations such that all the (cid:0) MN (cid:1) determinants of all N × N sub-matrices of Ψ are zero.

Original: As the determinant is a polynomial in the entries of the matrix and thus an analytic function of the entries and composition of analytic functions are again analytic, we conclude that each determinant is an analytic function of U .
Revised: As the determinant is a polynomial in the entries of the matrix and thus an analytic function of the entries and composition of analytic functions are again analytic, we conclude that each determinant is an analytic function of U .

Original: As shown above, there exists at least one U such that one of these determinant functions is not identically zero and thus by Lemma A.1, the set of U where this determinant is zero has measure zero.
Revised: As shown above, there exists at least one U such that one of these determinant functions is not identically zero and thus by Lemma A.1, the set of U where this determinant is zero has measure zero.

Original: But as all submatrices need to have low rank in order that Ψ has low rank, it follows that the set of U where Ψ has low rank has just measure zero.
Revised: But as all submatrices need to have low rank in order that Ψ has low rank, it follows that the set of U where Ψ has low rank has just measure zero.

Original: (cid:3)
Revised: (cid:3)

Original: C E XTENSION OF T HEOREM 3.4 TO GENERAL CONVEX LOSSES
Revised: C E XTENSION OF T HEOREM 3.4 TO GENERAL CONVEX LOSSES

Original: In this section, we consider a more general training objective, deﬁned as
Revised: In this section, we consider a more general training objective, deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where G ( U, V ) =
Revised: where G ( U, V ) =

Original: Ψ( U ) V ∈ R N × m is the output of the network for all training samples at some given parameters ( U, V ) , and ϕ : R N × m → R the loss function applied on the network output.
Revised: Ψ( U ) V ∈ R N × m is the output of the network for all training samples at some given parameters ( U, V ) , and ϕ : R N × m → R the loss function applied on the network output.

Original: Assumption C.

Revised: Assumption C.

Original: The loss function ϕ : R N × m → R is convex and bounded from below.
Revised: The loss function ϕ : R N × m → R is convex and bounded from below.

Original: One can easily check that the following loss functions satisfy Assumption C.1 as they are all convex and bounded from below by zero:
Revised: One can easily check that the following loss functions satisfy Assumption C.1 as they are all convex and bounded from below by zero:

Original: The cross-entropy loss from Equation 2, in particular:
Revised: The cross-entropy loss from Equation 2, in particular:

Original: [Equation]
Revised: [Equation]

Original: , where ( x i , y i)
Revised: where ( x i , y i

Original: Ni =1 is the training data with y i being the ground-truth class of x i.
Revised: Ni =1 is the training data with y i being the ground-truth class of x i

Original: The standard square loss (for classiﬁcation/regression tasks)
Revised: The standard square loss (for classiﬁcation/regression tasks)

Original: [Equation]
Revised: [Equation]

Original: where Y ∈ R N × m is the ground-truth matrix.
Revised: where Y ∈ R N × m is the ground-truth matrix.

Original: The multi-class Hinge-loss (for classiﬁcation tasks)
Revised: The multi-class Hinge-loss (for classiﬁcation tasks)

Original: [Equation]
Revised: [Equation]

Original:  where ( x i , y i )
Revised: )) , where ( x i , y i )

Original: Ni =1 is the training data with y i being the ground-truth class of x i .
Revised: Ni =1 is the training data with y i being the ground-truth class of x i .

Original: By Assumption C.1, ϕ is bounded from below, thus it attains a ﬁnite inﬁmum:
Revised: By Assumption C.1, ϕ is bounded from below, thus it attains a ﬁnite inﬁmum:

Original: [Equation]
Revised: [Equation]

Original: Basically, p ∗ serves as a lower bound on our training objective Φ .
Revised: Basically, p ∗ serves as a lower bound on our training objective Φ .

Original: For the above examples, it holds p ∗ = 0 .
Revised: For the above examples, it holds p ∗ = 0 .

Original: Next, we adapt the deﬁnition of “bad local valleys” from Deﬁnition 3.3 to the current setting.
Revised: Next, we adapt the deﬁnition of “bad local valleys” from Deﬁnition 3.3 to the current setting.

Original: Deﬁnition C.

Revised: Deﬁnition C.

Original: The α -sublevel set of Φ is deﬁned as L α =
Revised: The α -sublevel set of Φ is deﬁned as L α =

Original: { ( U, V )
Revised: { ( U, V )

Original: | Φ( U, V ) < α } .
Revised: | Φ( U, V ) < α } .

Original: A local valley is deﬁned as a connected component of some sublevel set L α .
Revised: A local valley is deﬁned as a connected component of some sublevel set L α .

Original: A bad local valley is a local valley on which the training objective Φ cannot be made arbitrarily close to p ∗ .
Revised: A bad local valley is a local valley on which the training objective Φ cannot be made arbitrarily close to p ∗ .

Original: The following result extends Theorem 3.4 to general convex losses.
Revised: The following result extends Theorem 3.4 to general convex losses.

Original: The proofs are mostly similar as before, but we present them below for the completeness and convenience of the reader.
Revised: The proofs are mostly similar as before, but we present them below for the completeness and convenience of the reader.

Original: Theorem C.

Revised: Theorem C.

Original: The following holds under Assumption 3.1 and Assumption C.1:
Revised: The following holds under Assumption 3.1 and Assumption C.1:

Original: There exist uncountably many solutions with zero training error.
Revised: There exist uncountably many solutions with zero training error.

Original: The loss landscape of Φ does not have any bad local valley.
Revised: The loss landscape of Φ does not have any bad local valley.

Original: There exists no suboptimal strict local minimum. 4.
Revised: There exists no suboptimal strict local minimum. 4.

Original: For cross-entropy loss (2) and square loss (14) there exists no local maximum.
Revised: For cross-entropy loss (2) and square loss (14) there exists no local maximum.

Original: Proof:1.
Revised: Proof:

Original: By Lemma 3.2 the set of U such that Ψ( U ) has not fullrank N has Lebesgue measure zero.
Revised: By Lemma 3.2 the set of U such that Ψ( U ) has not full

Revised: rank N

Revised: has

Revised: Lebesgue measure zero.

Original: Given U such that Ψ has full rank, the linear system Ψ( U ) V
Revised: Given U such that Ψ has full rank, the linear system Ψ( U ) V

Original: = Y has for every possible target output matrix Y ∈ R N × m at least one solution V .
Revised: = Y has for every possible target output matrix Y ∈ R N × m at least one solution V .

Original: As this is possible for almost all
Revised: As this is possible for almost all

Original: U , there exist uncountably many solutions achieving zero training error.
Revised: U , there exist uncountably many solutions achieving zero training error.

Original: Let C be a non-empty, connected component of some sub-level set L α where α > p ∗ .
Revised: Let C be a non-empty, connected component of some sub-level set L α where α > p ∗ .

Original: Note that L α = ∅ if α ≤ p ∗ by Deﬁnition C.2.
Revised: Note that L α = ∅ if α ≤ p ∗ by Deﬁnition C.2.

Original: Given any (cid:15) ∈ ( p ∗ , α ) , we will show that
Revised: Given any (cid:15) ∈ ( p ∗ , α ) , we will show that

Original: C always contains a point ( U, V ) s.t.
Revised: C always contains a point ( U, V ) s.t.

Original: Φ( U, V ) ≤ (cid:15) as this would imply that the loss Φ restricted to C can always attain arbitrarily small value close to p ∗ .
Revised: Φ( U, V ) ≤ (cid:15) as this would imply that the loss Φ restricted to C can always attain arbitrarily small value close to p ∗ .

Original: We note that L α =
Revised: We note that L α =

Original: Φ − 1 (( −∞ , α )) is an open set according to Proposition A.2.
Revised: Φ − 1 (( −∞ , α )) is an open set according to Proposition A.2.

Original: Since C is a non-empty connected component of L α , C must also be an open set with non-zero Lebesgue measure.
Revised: Since C is a non-empty connected component of L α , C must also be an open set with non-zero Lebesgue measure.

Original: By Lemma 3.2 the set of U where Ψ( U ) has not full rank has measure zero and thus C must contain a point ( U, V ) such that Ψ( U ) has full rank.
Revised: By Lemma 3.2 the set of U where Ψ( U ) has not full rank has measure zero and thus C must contain a point ( U, V ) such that Ψ( U ) has full rank.

Original: By Assumption C.1, ϕ attains its inﬁmum at p ∗ < (cid:15) , and thus by continuity of ϕ , there exists G ∗ ∈ R N × m such that p ∗ ≤ ϕ
Revised: By Assumption C.1, ϕ attains its inﬁmum at p ∗ < (cid:15) , and thus by continuity of ϕ , there exists G ∗ ∈ R N × m such that p ∗ ≤ ϕ

Original: ( G ∗ ) ≤ (cid:15).
Revised: ( G ∗ ) ≤ (cid:15).

Original: As Ψ( U ) has full rank, there always exist V ∗ such that Ψ( U ) V ∗ =
Revised: As Ψ( U ) has full rank, there always exist V ∗ such that Ψ( U ) V ∗ =

Original: G ∗ .
Revised: G ∗ .

Original: Now, one notes that the loss Φ( U, V ) =
Revised: Now, one notes that the loss Φ( U, V ) =

Original: ϕ (Ψ( U ) V ) is convex in V , and that Φ( U, V ) <
Revised: ϕ (Ψ( U ) V ) is convex in V , and that Φ( U, V ) <

Original: α , thus we have for the line segment V ( λ )
Revised: α , thus we have for the line segment V ( λ )

Original: = λV + (1 − λ ) V ∗ for λ ∈ [0 , 1] ,
Revised: = λV + (1 − [Equation] for [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Thus the whole line segment from ( U, V ) to ( U, V ∗ ) is contained in L α .
Revised: Thus the whole line segment from ( U, V ) to ( U, V ∗ ) is contained in L α .

Original: Since C is a connected component of L α which contains ( U, V ) , it follows that ( U, V ∗ )
Revised: Since C is a connected component of L α which contains ( U, V ) , it follows that ( U, V ∗ )

Original: ∈
Revised: ∈

Original: C. Moreover, one has Φ( U, V ∗ ) =
Revised: C. Moreover, one has Φ( U, V ∗ ) =

Original: ϕ ( G ∗ ) ≤ (cid:15) , which thus implies that the loss can always be made (cid:15) -small inside the set C for every (cid:15) ∈ ( p ∗ , α ) .
Revised: ϕ ( G ∗ ) ≤ (cid:15) , which thus implies that the loss can always be made (cid:15) -small inside the set C for every (cid:15) ∈ ( p ∗ , α ) .

Original: Let ( U 0 , V 0 ) be a strict suboptimal local minimum, then there exists r > 0 such that Φ( U, V ) >
Revised: Let ( U 0 , V 0 ) be a strict suboptimal local minimum, then there exists r > 0 such that Φ( U, V ) >

Original: Φ( U 0 , V 0 ) > p ∗ for all ( U, V ) ∈ B (( U 0 , V 0 ) , r ) \ { ( U 0 , V 0 ) } where B ( · , r ) denotes a closed ball of radius r .
Revised: Φ( U 0 , V 0 ) > p ∗ for all ( U, V ) ∈ B (( U 0 , V 0 ) , r ) \ { ( U 0 , V 0 ) } where B ( · , r ) denotes a closed ball of radius r .

Original: Let α = min
Revised: Let α = min

Original: ( U,V ) ∈ ∂B (cid:0) ( U 0 ,V 0 ) ,r (cid:1) Φ( U, V ) which exists as Φ is continuous and the boundary ∂B (cid:0) ( U 0 , V 0 ) , r (cid:1) of B (cid:0) ( U 0 , V 0 ) , r (cid:1) is compact.
Revised: ( U,V ) ∈ ∂B (cid:0) ( U 0 ,V 0 ) ,r (cid:1) Φ( U, V ) which exists as Φ is continuous and the boundary ∂B (cid:0) ( U 0 , V 0 ) , r (cid:1) of B (cid:0) ( U 0 , V 0 ) , r (cid:1) is compact.

Original: Note that Φ( U 0 , V 0 ) <
Revised: Note that Φ( U 0 , V 0 ) <

Original: α as ( U 0 , V 0 ) is a strict local minimum, and
Revised: α as ( U 0 , V 0 ) is a strict local minimum, and

Original: thus ( U 0 , V 0 ) ∈
Revised: thus ( U 0 , V 0 ) ∈

Original: L α .
Revised: L α .

Original: Let E be the connected component of L α which contains ( U 0 , V 0 ) , that is, ( U 0 , V 0 )
Revised: Let E be the connected component of L α which contains ( U 0 , V 0 ) , that is, ( U 0 , V 0 )

Original: ∈
Revised: ∈

Original: E ⊆
Revised: E ⊆

Original: L α .
Revised: L α .

Original: Since the loss of every point inside E is strictly smaller than α , whereas the loss of every point on the boundary ∂B (cid:0) ( U 0 , V 0 ) , r (cid:1) is greater than or equal to α , E must be contained in the interior of the ball, that is E ⊂ B (cid:0) ( U 0 , V 0 ) , r (cid:1) .
Revised: Since the loss of every point inside E is strictly smaller than α , whereas the loss of every point on the boundary ∂B (cid:0) ( U 0 , V 0 ) , r (cid:1) is greater than or equal to α , E must be contained in the interior of the ball, that is E ⊂ B (cid:0) ( U 0 , V 0 ) , r (cid:1) .

Original: Moreover, Φ( U, V ) ≥ Φ( U 0 , V 0 ) > p ∗ for every ( U, V ) ∈ E and thus the values of Φ restricted to E can not be arbitrarily close to p ∗ , which means that E is a bad local valley, which contradicts G.3.2.
Revised: Moreover, Φ( U, V ) ≥ Φ( U 0 , V 0 ) > p ∗ for every ( U, V ) ∈ E and thus the values of Φ restricted to E can not be arbitrarily close to p ∗ , which means that E is a bad local valley, which contradicts G.3.2.

Original: The proof for cross-entropy loss is similar to Theorem 3.4.
Revised: The proof for cross-entropy loss is similar to Theorem 3.4.

Original: For square loss, one has
Revised: For square loss, one has

Original: [Equation]
Revised: [Equation]

Original: where ⊗ denotes Kronecker product, and I m an m × m identity matrix.
Revised: where ⊗ denotes Kronecker product, and I m an m × m identity matrix.

Original: The hessian of Φ w.r.t.
Revised: The hessian of Φ w.r.t.

Original: V is ∇ 2 vec ( V ) Φ =
Revised: V is ∇ 2 vec ( V ) Φ =

Original: ( I m ⊗ Ψ( U ))
Revised: ( I m ⊗ Ψ( U ))

Original: T ( I m ⊗ Ψ( U )) .
Revised: T ( I m ⊗ Ψ( U )) .

Original: Suppose by contradiction that ( U, V ) is a local maximum.
Revised: Suppose by contradiction that ( U, V ) is a local maximum.

Original: Then the Hessian of Φ is negative semi-deﬁnite.
Revised: Then the Hessian of Φ is negative semi-deﬁnite.

Original: As principal submatrices of negative semi-deﬁnite matrices are again negative semi-deﬁnite, the Hessian of Φ w.r.t V must be also negative semi-deﬁnite.
Revised: As principal submatrices of negative semi-deﬁnite matrices are again negative semi-deﬁnite, the Hessian of Φ w.r.t V must be also negative semi-deﬁnite.

Original: However, Φ is convex in V thus its Hessian restricted to V must be positive semi-deﬁnite.
Revised: However, Φ is convex in V thus its Hessian restricted to V must be positive semi-deﬁnite.

Original: The only matrix which is both p.s.d.
Revised: The only matrix which is both p.s.d.

Original: and n.s.d.
Revised: and n.s.d.

Original: is the zero matrix.
Revised: is the zero matrix.

Original: It follows that ∇ 2 vec ( V ) Φ( U, V ) = 0 and thus Ψ( U ) = 0 .
Revised: It follows that ∇ 2 vec ( V ) Φ( U, V ) = 0 and thus Ψ( U ) = 0 .

Original: By Assumption 3.1, there exists j ∈ [ M ] s.t.
Revised: By Assumption 3.1, there exists j ∈ [ M ] s.t.

Original: σ p j is strictly positive, thus some entries of Ψ( U ) must be strictly positive, and so Ψ( U ) cannot be identically zero, leading to a contradiction.
Revised: σ p j is strictly positive, thus some entries of Ψ( U ) must be strictly positive, and so Ψ( U ) cannot be identically zero, leading to a contradiction.

Original: Therefore Φ has no local maximum.
Revised: Therefore Φ has no local maximum.

Original: (cid:3)
Revised: (cid:3)

Original: D T HE ARCHITECTURE OF CNN13 FROM T ABLE 1: S EE T ABLE
Revised: D T HE ARCHITECTURE OF CNN13 FROM T ABLE 1: S EE T ABLE

Original: [Table]
Revised: [Table]

Original: E V ISUALIZATION OF THE LOSS LANDSCAPE BEFORE AND AFTER ADDING SKIP - CONNECTIONS TO THE OUTPUT LAYER
Revised: E V ISUALIZATION OF THE LOSS LANDSCAPE BEFORE AND AFTER ADDING SKIP - CONNECTIONS TO THE OUTPUT LAYER

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Original: Similar to Li et al.
Revised: Similar to Li et al.

Original: ; Goodfellow et al.
Revised: ; Goodfellow et al.

Original: , we visualize the loss surface restricted to a two dimensional subspace of the parameter space.
Revised: , we visualize the loss surface restricted to a two dimensional subspace of the parameter space.

Original: The subspace is chosen to go through some point ( U 0 , V 0 ) learned by SGD and spanned by two random directions ( U 1 , V 1 ) and ( U 2 , V 2 ) .
Revised: The subspace is chosen to go through some point ( U 0 , V 0 ) learned by SGD and spanned by two random directions ( U 1 , V 1 ) and ( U 2 , V 2 ) .

Original: For the purpose of illustration, we train with SGD a two-hidden-layer fully connected network withand 300 hidden units respectively, followed by a 10 -way softmax.
Revised: For the purpose of illustration, we train with SGD a two-hidden-layer fully connected network with 784 and 300 hidden units respectively, followed by a 10 -way softmax.

Original: The training set consists ofimages, which are randomly selected from MNIST dataset.
Revised: The training set consists of 1024 images, which are randomly selected from MNIST dataset.

Original: After adding skip-connections to the output, the network fulﬁlls M = N = 1024 .
Revised: After adding skip-connections to the output, the network fulﬁlls M = N = 1024 .

Original: Figure 3 shows the heat map of the loss surface before and after adding skip-connections.
Revised: Figure 3 shows the heat map of the loss surface before and after adding skip-connections.

Original: One can see a visible effect that skip-connections have helped to smooth the loss landscape near a small sub-optimal region and allows gradient descent to ﬂow directly from there to the bottom of the landscape with smaller objective value.
Revised: One can see a visible effect that skip-connections have helped to smooth the loss landscape near a small sub-optimal region and allows gradient descent to ﬂow directly from there to the bottom of the landscape with smaller objective value.

Original: F D ISCUSSION OF TRAINING ERROR 
Revised: F D ISCUSSION OF TRAINING ERROR IN T ABLE

Original: Training error for the experiment in Table 2.
Revised: Training error for the experiment in Table 2.

Original: We report in Table 4 the trainingperformance of all models from Table 2. The training error is zero in all cases, except when the original VGG models are used with sigmoid activation function.
Revised: As shown in Table 2, the training error is zero in all cases, except when the original VGG models are used with sigmoid activation function.

Original: The reason, as noticed in our experiments, is because the learning of these sigmoidal networks converges quickly to a constant zero classiﬁer (i.e. the output of the last hidden layer converges to zero), which makes both training and test accuracy converge to 10% and the loss in Equation (2) converges to − log(1 / 10) .While we are not aware of a theoretical [Figure] [Figure]
Revised: The reason, as noticed in our experiments, is because the learning of these sigmoidal networks converges quickly to a constant zero classiﬁer (i.e. the output of the last hidden layer converges to zero), which makes both training and test accuracy converge to 10% and the loss in Equation (2) converges to − log(1 / 10) .

Original:  explanation for this behavior, it is not restricted to the speciﬁc architecture of VGGs but hold in general for plain sigmoidal networks with depth>5 as pointed out earlier by Glorot & Bengio (2010) [Figure] As shown in Table [Figure]
Revised: While we are not aware of a theoretical explanation for this behavior, it is not restricted to the speciﬁc architecture of VGGs but hold in general for plain sigmoidal networks with depth>5 as pointed out earlier by Glorot & Bengio (2010).

Original: Densenets however do not suffer from this phenomenon

Original: [Figure]

Original:  probably because they already have skip-connections between all the hidden layers of a dense block [Figure] thus gradients can easily ﬂow from the output to every layer of a dense block [Figure]
Revised: As shown in Table 2, Densenets however do not suffer from this phenomenon, probably because they already have skip-connections between all the hidden layers of a dense block, thus gradients can easily ﬂow from the output to every layer of a dense block, which makes the training of this network with sigmoid activation function become feasible.

Original: which makes the training of this network with sigmoid activation function become feasible

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: Discussion of convergence speed.
Revised: Discussion of convergence speed.

Original: For sigmoid activation [Figure] we noticed that skip-models when trained with the random sampling approach (skip-rand) often converge much slower than when trained with full SGD (skip-SGD).
Revised: For sigmoid activation, we noticed that skip-models when trained with the random sampling approach (skip-rand) often converge much slower than when trained with full SGD (skip-SGD).

Original: In our experiments, to be sure that one gets absolute zero training error, we set the number of training epochs to 5000 for the former case and 1000 for the later.
Revised: In our experiments, to be sure that one gets absolute zero training error, we set the number of training epochs to 5000 for the former case and 1000 for the later.

Original: Perhaps a better learning rate schedule might help to reduce this number, or maybe not, but this is beyond the scope of this paper.
Revised: Perhaps a better learning rate schedule might help to reduce this number, or maybe not, but this is beyond the scope of this paper.

Original: For softplus activation, we noticed a much faster convergence – all models often converge within 300 epochs to absolute zero training error.
Revised: For softplus activation, we noticed a much faster convergence – all models often converge within 300 epochs to absolute zero training error.

Original: [Figure]
Revised: [Figure]

Revised: Skip-connections are also helpful for training very deep networks with softplus activation.

Original: Previously we have shown that skip-connections are helpful for training deep sigmoidal networks.
Revised: Previously we have shown that skip-connections are helpful for training deep sigmoidal networks.

Original: In this part, we show a similar result for softplus activation function.
Revised: In this part, we show a similar result for softplus activation function.

Original: For the purpose of illustration, we create a small dataset with N = 1000 training images randomly chosen from CIFAR10 dataset.
Revised: For the purpose of illustration, we create a small dataset with N = 1000 training images randomly chosen from CIFAR10 dataset.

Original: We use a very deep network with 150 fully connected layers, each of width 10 , and softplus activation.
Revised: We use a very deep network with 150 fully connected layers, each of width 10 , and softplus activation.

Original: A skip-model is created by adding skip-connections from N randomly chosen neurons to the output units.
Revised: A skip-model is created by adding skip-connections from N randomly chosen neurons to the output units.

Original: We train both networks with SGD.
Revised: We train both networks with SGD.

Original: The best learning rate for each model is empirically chosen from (cid:8) 10 − 2 , 10 − 3 , 10 − 4 , 10 − 5 (cid:9) .
Revised: The best learning rate for each model is empirically chosen from (cid:8) 10 − 2 , 10 − 3 , 10 − 4 , 10 − 5 (cid:9) .

Original: We report the training loss and training error of both models in
Revised: We report the training loss and training error of both models in

Original: Figure 5.
Revised: Figure 5.

Original: One can see that the skip-network easily converge to zero training error within 200 epochs, whereas the original network has stronger ﬂuctuations and fails to converge after 1000 epochs.
Revised: One can see that the skip-network easily converge to zero training error within 200 epochs, whereas the original network has stronger ﬂuctuations and fails to converge after 1000 epochs.

Original: This is directly related to our result of Theorem 3.4 in the sense that skip-connections can help to smooth the loss landscape and enable effective training of very deep networks.
Revised: This is directly related to our result of Theorem 3.4 in the sense that skip-connections can help to smooth the loss landscape and enable effective training of very deep networks.

Original: G A DDITIONAL EXPERIMENTS : M AX - POOLING OF VGG S ARE REPLACED BY 2 X 2 CONVOLUTIONAL LAYERS OF STRIDE
Revised: G A DDITIONAL EXPERIMENTS : M AX - POOLING OF VGG S ARE REPLACED BY 2 X 2 CONVOLUTIONAL LAYERS OF STRIDE

Original: The original VGG Simonyan & Zisserman (2015) and Densenet Huang et al.
Revised: The original VGG Simonyan & Zisserman (2015) and Densenet Huang et al.

Original: (2017) contain pooling layers in their architecture.
Revised: (2017) contain pooling layers in their architecture.

Original: In particular, original VGGs have max-pooling layers, and original
Revised: In particular, original VGGs have max-pooling layers, and original

Original: Densenets have averaging pooling layers.
Revised: Densenets have averaging pooling layers.

Original: In the following, we will clarify how/if these pooling layers have been used in our experiments in Table 2, and whether and how our theretical results are appicable to this case, as well as presenting additional experimental results in this regard.
Revised: In the following, we will clarify how/if these pooling layers have been used in our experiments in Table 2, and whether and how our theretical results are appicable to this case, as well as presenting additional experimental results in this regard.

Original: First of all, we note that Densenets Huang et al.
Revised: First of all, we note that Densenets Huang et al.

Original: (2017) contain pooling layers only after the ﬁrst dense block.
Revised: (2017) contain pooling layers only after the ﬁrst dense block.

Original: Meanwhile, as noted in Table 2, our experiments with Densenets only use skip-connections from hidden units of the ﬁrst dense block, and thus Lemma 3.2 and Theorem 3.4 are applicable.
Revised: Meanwhile, as noted in Table 2, our experiments with Densenets only use skip-connections from hidden units of the ﬁrst dense block, and thus Lemma 3.2 and Theorem 3.4 are applicable.

Original: The reason is that one can restrict the full-rank analysis of matrix Ψ in Lemma 3.2 to the hidden units of the ﬁrst dense block, so that it follows that the set of parameters of the ﬁrst dense block where Ψ has not full rank has Lebesgue measure zero, from which our results of Theorem 3.4 follow immediately.
Revised: The reason is that one can restrict the full-rank analysis of matrix Ψ in Lemma 3.2 to the hidden units of the ﬁrst dense block, so that it follows that the set of parameters of the ﬁrst dense block where Ψ has not full rank has Lebesgue measure zero, from which our results of Theorem 3.4 follow immediately.

Original: However for VGGs in Table 2, we kept their max-pooling layers similar to the original architecture as we wanted to have a fair comparison between our skip-models and the original models.
Revised: However for VGGs in Table [Table] we kept their max-pooling layers similar to the original architecture as we wanted to have a fair comparison between our skip-models and the original models.

Original: In this setting, our results are not directly applicable because we lose the analytic property of the entries of Ψ w.r.t.
Revised: In this setting, our results are not directly applicable because we lose the analytic property of the entries of Ψ w.r.t.

Original: its dependent parameters, which is crucial to prove Lemma 3.2.
Revised: its dependent parameters, which is crucial to prove Lemma 3.2.

Original: Therefore in this section, we would like to present additional results to Table 2 in which we replace all max-pooling layers of all
Revised: Therefore in this section, we would like to present additional results to Table 2 in which we replace all max-pooling layers of all

Original: VGG models from Table 2 with 2x2 convolutional layers of stride 2.
Revised: VGG models from Table 2 with 2x2 convolutional layers of stride 2.

Original: In this case, the whole network consists of only convolutional and fully connected layers, hence our theoretical results are applicable.
Revised: In this case, the whole network consists of only convolutional and fully connected layers, hence our theoretical results are applicable

Original:  The experimental results are presented in Table 5.
Revised: [Table] The experimental results are presented in Table 4.

Original: Overall, our main observations are similar as before.
Revised: Overall, our main observations are similar as before.

Original: The performance gap between original models and their corresponding skip-variants are approximately the same as in Table 2 or slightly more pronounced in some cases.
Revised: The performance gap between original models and their corresponding skip-variants are approximately the same as in Table 2 or slightly more pronounced in some cases.

Original: A one-to-one comparison with Table 2 also shows that the performance of skip-models themselves have decreased by 4 − 7% after the replacement of max-pooling layers with 2x2 convolutional layers.
Revised: A one-to-one comparison with Table 2 also shows that the performance of skip-models themselves have decreased by 4 − 7% after the replacement of max-pooling layers with 2x2 convolutional layers.

Original: This is perhaps not so surprising because the problem gets potentially harder when the network has more layers to be learned, especially in case of sigmoid activation where the decrease is sharper.
Revised: This is perhaps not so surprising because the problem gets potentially harder when the network has more layers to be learned, especially in case of sigmoid activation where the decrease is sharper.

Original: Similar to Table 2, adding skip-connections to the output units still prove to be very helpful – it improves the result for softplus while making the training of deep networks with sigmoid activation become possible at all.
Revised: Similar to Table 2, adding skip-connections to the output units still prove to be very helpful – it improves the result for softplus while making the training of deep networks with sigmoid activation become possible at all.

Original: Finally, the training of full network with SGD still yields signiﬁcantly better solutions in terms of generalization error than the random feature approach.
Revised: Finally, the training of full network with SGD still yields signiﬁcantly better solutions in terms of generalization error than the random feature approach.

Original: This conﬁrms once again the implicit bias of
Revised: This conﬁrms once again the implicit bias of

Original: SGD towards high quality solutions among inﬁnitely many solutions with zero training error.
Revised: SGD towards high quality solutions among inﬁnitely many solutions with zero training error.

Revised: H D ATA - AUGMENTATION RESULTS FOR T ABLE

Revised: The following Table 5 shows additional results to Table 2 where data-augmentation is used now.

Revised: For data-augmentation, we follow the procedure as described in (Zagoruyko & Komodakis, 2016) by considering random crops of size 32 × 32 after 4 pixel padding on each side of the training images and random horizontal ﬂips with probability 0 .

Revised: For the convenience of the reader, we also repeat the results of Table 2 in the new table 5.

Original: [Table]
Revised: [Table]