S KIP C ONNECTIONS E LIMINATE S INGULARITIES
Anonymous authors
Paper under double-blind review
A BSTRACT
Skip connections made the training of very deep networks possible and have become an indispensable component in a variety of neural architectures.
A completely satisfactory explanation for their success remains elusive.
Here, we present a novel explanation for the beneﬁts of skip connections in training very deep networks.
The difﬁculty of training deep networks is partly due to the singularities caused by the non-identiﬁability of the model.
Several such singularities have been identiﬁed in previous works: (i) overlap singularities caused by the permutation symmetry of nodes in a given layer, (ii) elimination singularities corresponding to the elimination, i.e. consistent deactivation, of nodes, (iii) singularities generated by the linear dependence of the nodes.
These singularities cause degenerate manifolds in the loss landscape that slow down learning.
We argue that skip connections eliminate these singularities by breaking the permutation symmetry of nodes, by reducing the possibility of node elimination and by making the nodes less linearly dependent.
Moreover, for typical initializations, skip connections move the network away from the “ghosts” of these singularities and sculpt the landscape around them to alleviate the learning slow-down.
These hypotheses are supported by evidence from simpliﬁed models, as well as from experiments with deep networks trained on real-world datasets.
I NTRODUCTION
Skip connections are extra connections between nodes in different layers of a neural network that skip one or more layers of nonlinear processing.
The introduction of skip (or residual) connections has substantially improved the training of very deep neural networks (He et al., 2015; 2016;
Huang et al., 2016; Srivastava et al., 2015).
Despite informal intuitions put forward to motivate skip connections, a clear understanding of how these connections improve training has been lacking.
Such understanding is invaluable both in its own right and for the possibilities it might offer for further improvements in training very deep neural networks.
In this paper, we attempt to shed light on this question.
We argue that skip connections improve the training of deep networks partly by eliminating the singularities inherent in the loss landscapes of deep networks.
These singularities are caused by the non-identiﬁability of subsets of parameters when nodes in the network either get eliminated ( elimination singularities ), collapse into each other ( overlap singularities ) (Wei et al., 2008), or become linearly dependent ( linear dependence singularities ).
Saad & Solla (1995); Amari et al. ; Wei et al. identiﬁed the elimination and overlap singularities and showed that they signiﬁcantly slow down learning in shallow networks; Saxe et al.
(2013) showed that linear dependence between nodes arises generically in randomly initialized deep linear networks and becomes more severe with depth.
We show that skip connections eliminate these singularities and provide evidence suggesting that they improve training partly by ameliorating the learning slow-down caused by the singularities.
R ESULTS 2. S INGULARITIES IN FULLY - CONNECTED LAYERS AND HOW SKIP CONNECTIONS BREAK THEM
In this work, we focus on three types of singularity that arise in fully-connected layers: elimination and overlap singularities (Amari et al., 2006; Wei et al., 2008), and linear dependence singularities (Saxe et al., 2013).
The linear dependence singularities can arise exactly only in linear networks,
J a = J b
J a = J b =α J + βJ a b J =
J = identifiable? yesno ...
w w = w =
J
J ...
b w w a w b ... w w a ...
Eliminationsingularities a b c
Overlapsingularity ... skip
Linear dependence singularity
J a
J c w c w a w b
J a J c J b J b w c w a w b In linear dependence singularities a hidden become linearly dependent, making their outgoing whereas the elimination and overlap singularities can arise in non-linear networks as well. These singularities are all related to the non-identiﬁability of the model.
The Hessian of the loss function becomes singular at these singularities (Supplementary Note 1), hence they are sometimes also called degenerate or higher-order saddles (Anandkumar & Ge, 2016).
Elimination singularities arise when a hidden unit is effectively killed, e.g. when its incoming (or outgoing) weights become zero (Figure 1a).
This makes the outgoing (or incoming) connections of the unit non-identiﬁable.
Overlap singularities are caused by the permutation symmetry of the hidden units at a given layer and they arise when two units become identical, e.g. when their incoming weights become identical (Figure 1b).
In this case, the outgoing connections of the units are no longer identiﬁable individually (only their sum is identiﬁable).
Linear dependence singularities arise when a subset of the hidden units in a layer become linearly dependent (Figure 1c).
Again, the outgoing connections of these units are no longer identiﬁable individually (only a linear combination of them is identiﬁable).
How do skip connections eliminate these singularities?
Skip connections between adjacent layers break the elimination singularities by ensuring that the units are active at least for some inputs, even when their adjustable incoming or outgoing connections become zero (Figure 1a; right).
They eliminate the overlap singularities by breaking the permutation symmetry of the hidden units at a given layer (Figure 1b; right).
Thus, even when the adjustable incoming weights of two units become identical, the units do not collapse into each other, since their distinct skip connections still disambiguate them.
They also eliminate the linear dependence singularities by adding linearly independent (in fact, orthogonal in most cases) inputs to the units (Figure 1c; right).
2. W HY ARE SINGULARITIES HARMFUL FOR LEARNING ?
The effect of elimination and overlap singularities on gradient-based learning has been analyzed previously for shallow networks (Amari et al., 2006; Wei et al., 2008).
Figure 2a shows the simpliﬁed two hidden unit model analyzed in Wei et al.
(2008) and its reduction to a two-dimensional system in terms of the overlap and elimination variables, h and z .
Both types of singularity cause degenerate manifolds in the loss landscape, represented by the lines h
= 0 and z = ± 1 in Figure 2b, corresponding to the overlap and elimination singularities respectively.
The elimination manifolds divide the overlap manifolds into stable and unstable segments.
According to the analysis presented in Wei et al.
(2008), these manifolds give rise to two types of plateaus in the learning dynamics: on-singularity plateaus which are caused by the random walk behavior of stochastic gradient descent (SGD) along a stable segment of the overlap manifolds (thick segment on the h = 0 line in Figure 2b) until it escapes the stable segment, and (more relevant in practical cases) near-singularity plateaus which manifest themselves as a general slowing of the dynamics near the overlap manifolds, even when the initial location is not within the basin of attraction of the stable segment.
Although this analysis only holds for two hidden units, for higher dimensional cases, it suggests that overlaps between hidden units signiﬁcantly slow down learning along the overlap directions.
These overlap directions become more numerous as the number of hidden units increases, thus reducing the effective dimensionality of the model.
We provide empirical evidence for this claim below. J a
J b w a w b   – – z 1 h –   l og s p ee d  100 Time z,h
Near-singularity plateau Gradient flow zh z = a –w b h= ||
J a –J b ||c b a
J a
J c J b w c w a w b d e w a + w b w overlap eliminationparameter m m = || J a + J b –J c || linear dependency ||
J c ||
+ =
{+l og s p ee d 0 – –2 – –4 –
+}
As mentioned earlier linear dependence singularities arise exactly only in linear networks.
However, we expect them to hold approximately and thus have consequences for learning, in the non-linear case as well.
Figure 2d-e shows an example in a toy single-layer nonlinear network: learning along a linear dependence manifold, represented by m here, is much slower than learning along other directions, e.g. the norm of the incoming weight vector J c in the example shown here.
Saxe et al.
(2013) demonstrated that this linear dependence problem arises generically, and becomes worse with depth, in randomly initialized deep linear networks.
Because learning is signiﬁcantly slowed down along linear dependence directions compared to other directions, these singularities effectively reduce the dimensionality of the model, similarly to the overlap manifolds.
2. P LAIN NETWORKS ARE MORE DEGENERATE THAN NETWORKS WITH SKIP CONNECTIONS
To investigate the relationship between degeneracy, training difﬁculty and skip connections in deep networks, we conducted several experiments with deep fully-connected networks.
We compared three different architectures. (i) The plain architecture is a fully-connected feedforward network with no skip connections, described by the equation:
[Equation]
where f is the ReLU nonlinearity and x 0 denotes the input layer.
(ii)
The residual architecture introduces identity skip connections between adjacent layers (note that we do not allow skip connections from the input layer):
[Equation]
(iii)
The hyper-residual architecture adds skip connections between each layer and all layers above it:
[Equation]
The skip connectivity from the immediately preceding layer is always the identity matrix, whereas the remaining skip connections Q k are ﬁxed, but allowed to be different from the identity (see
Supplementary Note 2 for further details).
This architecture is inspired by the DenseNet architecture (Huang et al., 2016).
In both architectures, each layer projects skip connections to layers above it.
However, in the DenseNet architecture, the skip connectivity matrices are learned, whereas in the hyper-residual architecture considered here, they are ﬁxed.
In the experiments of this subsection, the networks all had L = 20 hidden layers (followed by a softmax layer at the top) and n = 128 hidden units (ReLU) in each hidden layer.
Hence, the networks had the same total number of parameters.
The biases were initialized to 0 and the weights were initialized with the Glorot normal initialization scheme (Glorot & Bengio, 2010).
The networks were trained on the CIFAR-100 dataset (with coarse labels) using the Adam optimizer (Kingma & [Figure] Ba, 2014) with learning rate 0 . 0005 and a batch size of 500 .
Because we are mainly interested in understanding how singularities, and their removal, change the shape of the loss landscape and consequently affect the optimization difﬁculty, we primarily monitor the training accuracy rather than test accuracy in the results reported below.
To measure degeneracy, we estimated the eigenvalue density of the Hessian during training for the three different network architectures.
The probability of small eigenvalues in the eigenvalue density reﬂects the dimensionality of the degenerate parameter space.
To estimate this eigenvalue density in our ∼ 1M -dimensional parameter spaces, we ﬁrst estimated the ﬁrst four moments of the spectral density using the method of Skilling (Skilling, 1989) and ﬁt the estimated moments with a ﬂexible mixture density model (see Supplementary Note 3 for details) consisting of a narrow
Gaussian component to capture the bulk of the spectral density, and a skew Gaussian density to capture the tails (see Figure 3d for example ﬁts).
From the ﬁtted mixture density, we estimated the fraction of degenerate eigenvalues and the fraction of negative eigenvalues during training.
We validated our main results, as well as our mixture model for the spectral density, with smaller networks with ∼ 14K parameters where we could calculate all eigenvalues of the Hessian numerically (Supplementary Note 4).
For these smaller networks, the mixture model slightly underestimated the fraction of degenerate eigenvalues and overestimated the fraction of negative eigenvalues; however, there was a highly signiﬁcant linear relationship between the actual and estimated fractions.
Figure 3b shows the evolution of the fraction of degenerate eigenvalues during training.
A large value at a particular point during optimization indicates a more degenerate model.
By this measure, the hyper-residual architecture is the least degenerate and the plain architecture is the most degenerate.
We observe the opposite pattern for the fraction of negative eigenvalues (Figure 3c).
The differences between the architectures are more prominent early on in the training and there is an indication of a crossover later during training, with less degenerate models early on becoming slightly more degenerate later on as the training performance starts to saturate (Figure 3b).
Importantly, the hyper-residual architecture has the highest training speed and the plain architecture has the lowest training speed (Figure 3a), consistent with our hypothesis that the degeneracy of a model increases the training difﬁculty and skip connections reduce the degeneracy.
2. T RAINING ACCURACY IS RELATED TO DISTANCE FROM DEGENERATE MANIFOLDS
To establish a more direct relationship between the elimination, overlap and linear dependence singularities discussed earlier on the one hand, and model degeneracy and training difﬁculty on the other, we exploited the natural variability in training the same model caused by the stochasticity of stochastic gradient descent (SGD) and random initialization.
Speciﬁcally, we trained 100 plain networks (30 hidden layers, 128 neurons per layer) on CIFAR-100 using different random initializations and random mini-batch selection.
Training performance varied widely across runs.
We compared the best 10 and the worst 10 runs (measured by mean accuracy over 100 training epochs, Figure 4a).
The worst networks were more degenerate (Figure 4b); they were signiﬁcantly closer to elimination singularities, as measured by the average l 2 -norm of the incoming weights of their hidden units (Figure 4c); they were signiﬁcantly closer to overlap singularities (Figure 4d), as measured by the mean correlation between the incoming weights of their hidden units; and their hidden units
[Figure]
were signiﬁcantly more linearly dependent (Figure 4e), as measured by the mean variance explained by the top three eigenmodes of the covariance matrices of the hidden units in the same layer.
2. B ENEFITS OF SKIP CONNECTIONS AREN ’ T EXPLAINED BY GOOD INITIALIZATION ALONE
To investigate if the beneﬁts of skip connections can be explained in terms of favorable initialization of the parameters, we introduced a malicious initialization scheme for the residual network by subtracting the identity matrix from the initial weight matrices, W l .
If the beneﬁts of skip connections can be explained primarily by favorable initialization, this malicious initialization would be expected to cancel the effects of skip connections at initialization and hence signiﬁcantly deteriorate the performance.
However, the malicious initialization only had a small adverse effect on the performance of the residual network (Figure 5; ResMalInit), suggesting that the beneﬁts of skip connections cannot be explained by favorable initialization alone.
This result reveals a fundamental weakness in previous explanations of the beneﬁts of skip connections based purely on linear models (Hardt &
Ma, 2016; Li et al., 2016).
In Supplementary Note 5 we show that skip connections do not eliminate the singularities in deep linear networks, but only shift the landscape so that typical initializations are farther from the singularities.
Thus, in linear networks, any beneﬁts of skip connections are due entirely to better initialization.
In contrast, skip connections genuinely eliminate the singularities in nonlinear networks (Supplementary Note 1).
The fact that malicious initialization of the residual network does reduce its performance suggests that “ghosts” of these singularities still exist in the loss landscape of nonlinear networks, but the performance reduction is only slight, suggesting that skip connections alter the landscape around these ghosts to alleviate the learning slow-down that would otherwise take place near them.
2. A LTERNATIVE WAYS OF ELIMINATING THE SINGULARITIES
If the success of skip connections can be attributed, at least partly, to eliminating singularities, then alternative ways of eliminating them should also improve training.
We tested this hypothesis by introducing a particularly simple way of eliminating singularities: for each layer we drew random target biases from a Gaussian distribution, N ( µ, σ ) , and put an l 2 -norm penalty on learned biases deviating from those targets.
This breaks the permutation symmetry between units and eliminates the overlap singularities.
In addition, positive µ values decrease the average threshold of the units and make the elimination of units less likely (but not impossible), hence reducing the elimination singularities.
Decreased thresholds can also increase the dimensionality of the responses in a given layer by reducing the fraction of times different units are identically zero, thereby making them less linearly dependent.
Note that setting µ = 0 and σ = 0 corresponds to the standard l 2 -norm regularization of the biases, which does not eliminate any of the overlap or elimination singularities.
Hence, we expect the performance to be worse in this case than in cases with properly eliminated singularities.
On the other hand, although in general, larger values of µ and σ correspond to greater elimination of singularities, the network also has to perform well in the classiﬁcation task and very large µ , σ values might be inconsistent with the latter requirement.
Therefore, we expect the performance to be optimal for intermediate values of µ and σ .
In the experiments reported below, we optimized the hyperparameters µ , σ , and λ , i.e. the mean and the standard deviation of the target [Figure] bias distribution and the strength of the bias regularization term, through random search (Bergstra & Bengio, 2012).
We trained 30-layer fully-connected feedforward networks on CIFAR-10 and CIFAR-100 datasets.
Figure 5a-b shows the training accuracy of different models on the two datasets.
For both datasets, among the models shown in Figure 5, the residual network performs the best and the plain network the worst.
Our simple singularity elimination through bias regularization scheme (BiasReg, cyan) signiﬁcantly improves performance over the plain network.
Importantly, the standard l 2 -norm regularization on the biases (BiasL2Reg ( µ = 0 , σ = 0 ), magenta) does not improve performance over the plain network.
These results are consistent with the singularity elimination hypothesis.
There is still a signiﬁcant performance gap between our BiasReg network and the residual network despite the fact that both break degeneracies.
This can be partly attributed to the fact that the residual network breaks the degeneracies more effectively than the BiasReg network (Figure 5c).
Secondly, even in models that completely eliminate the singularities, the learning speed would still depend on the behavior of the gradient norms, and the residual network fares better than the BiasReg network in this respect as well.
At the beginning of training, the gradient norms with respect to the layer activities do not diminish in earlier layers of the residual network (Figure 6a, Epoch 0), demonstrating that it effectively solves the vanishing gradients problem (Hochreiter, 1991; Bengio et al., 1994).
On the other hand, both in the plain network and in the BiasReg network, the gradient norms decay quickly as one descends from the top of the network.
Moreover, as training progresses (Figure 6a, Epochs 1 and 2), the gradient norms are larger for the residual network than for the plain or the BiasReg network.
Even for the maliciously initialized residual network, gradients do not decay quickly at the beginning of training and the gradient norms behave similarly to those of the residual network during training (Figure 6a; ResMalInit), suggesting that skip connections boost the gradient norms near the ghosts of the singularities and reduce the learning slow-down that would otherwise take place near them.
Adding a single batch normalization layer (Ioffe & Szegedy, 2015) in the middle of the BiasReg network alleviates the vanishing gradients problem for this network and brings its performance closer to that of the residual network (Figure 6a-b; BiasReg+BN).
2. N ON - IDENTITY SKIP CONNECTIONS
If the singularity elimination hypothesis is correct, there should be nothing special about identity skip connections.
Skip connections other than identity should lead to training improvements if they eliminate singularities.
For the permutation symmetry breaking of the hidden units, ideally the skip connection vector for each unit should disambiguate that unit maximally from all other units in that layer.
This is because as shown by the analysis in Wei et al.
(2008) (Figure 2), even partial overlaps between hidden units signiﬁcantly slow down learning (near-singularity plateaus).
Mathematically, the maximal disambiguation requirement corresponds to an orthogonality condition on the skip connectivity matrix (any full-rank matrix breaks the permutation symmetry, but only orthogonal matrices maximally disambiguate the units).
Adding orthogonal vectors to different hidden units is also useful for breaking potential (exact or approximate) linear dependencies between them.
We therefore tested random dense orthogonal matrices as skip connectivity matrices.
Random dense
[Figure]
[Figure]
-10 and
CIFAR-100 datasets (Figure 7a, black vs. blue).
This is because, even with skip connections, units can be deactivated for some inputs because of the ReLU nonlinearity (recall that we do not allow skip connections from the input layer).
When this happens to a single unit at layer l , that unit is effectively eliminated for that subset of inputs, hence eliminating the skip connection to the corresponding unit at layer l
+1 , if the skip connectivity is the identity.
This causes a potential elimination singularity for that particular unit.
With dense skip connections, however, this possibility is reduced, since all units in the previous layer are used.
Moreover, when two distinct units at layer l are deactivated together, the identity skips cannot disambiguate the corresponding units at the next layer, causing a potential overlap singularity.
On the other hand, with dense orthogonal skips, because all units at layer l are used, even if some of them are deactivated, the units at layer l + 1 can still be disambiguated with the remaining active units.
Figure 7b conﬁrms for the CIFAR-100 dataset that throughout most of the training, the hidden units of the network with dense orthogonal skip connections have a lower probability of zero responses than those of the network with identity skip connections.
Next, we gradually decreased the degree of “orthogonality” of the skip connectivity matrix to see how the orthogonality of the matrix affects performance.
Starting from a random dense orthogonal matrix, we ﬁrst divided the matrix into two halves and copied the ﬁrst half to the second half.
Starting from n orthonormal vectors, this reduces the number of orthonormal vectors to n/ 2 .
We continued on like this until the columns of the matrix were repeats of a single unit vector.
We predict that as the [Figure] number of orthonormal vectors in the skip connectivity matrix is decreased, the performance should deteriorate, because both the permutation symmetry-breaking capacity and the linear-dependencebreaking capacity of the skip connectivity matrix are reduced.
Figure 7 shows the results for n =hidden units.
Darker colors correspond to “more orthogonal” matrices (e.g. “128” means allskip vectors are orthonormal to each other, “1” means all 128 vectors are identical).
The blue line is the identity skip connectivity.
More orthogonal skip connectivity matrices yield better performance, consistent with our hypothesis.
The less orthogonal skip matrices also suffer from the vanishing gradients problem.
So, their failure could be partly attributed to the vanishing gradients problem.
To control for this effect, we also designed skip connectivity matrices with eigenvalues on the unit circle (hence with eigenvalue spectra equivalent to an orthogonal matrix), but with varying degrees of orthogonality (see Supplementary
Note 6 for details).
More speciﬁcally, the columns (or rows) of an orthogonal matrix are orthonormal to each other, hence the covariance matrix of these vectors is the identity matrix.
We designed matrices where this covariance matrix was allowed to have non-zero off-diagonal values, reﬂecting the fact that the vectors are not orthogonal any more.
By controlling the magnitude of the correlations between the vectors, we manipulated the degree of orthogonality of the vectors.
We achieved this by setting the eigenvalue spectrum of the covariance matrix to be given by λ i = exp( − τ ( i − 1)) where λ i denotes the i -th eigenvalue of the covariance matrix and τ is the parameter that controls the degree of orthogonality: τ = 0 corresponds to the identity covariance matrix, hence to an orthonormal set of vectors, whereas larger values of τ correspond to gradually more correlated vectors.
This orthogonality manipulation was done while ﬁxing the eigenvalue spectrum of the skip connectivity matrix to be on the unit circle.
Hence, the effects of this manipulation cannot be attributed to any change in the eigenvalue spectrum, but only to the degree of orthogonality of the skip vectors.
The results of this experiment are shown in Figure 8.
More orthogonal skip connectivity matrices still perform better than less orthogonal ones (Figure 8c-d), even when their eigenvalue spectrum is ﬁxed and the vanishing gradients problem does not arise (Figure 8b), suggesting that the results of the earlier experiment (Figure 7) cannot be explained solely by the vanishing gradients problem.
D ISCUSSION
In this paper, we proposed a novel explanation for the beneﬁts of skip connections in terms of the elimination of singularities.
Our results suggest that elimination of singularities contributes at least partly to the success of skip connections.
However, we emphasize that singularity elimination is not the only factor explaining the beneﬁts of skip connections.
Even in completely non-degenerate models, other independent factors such as the behavior of gradient norms would affect training performance.
Indeed, we presented evidence suggesting that skip connections are also quite effective at dealing with the problem of vanishing gradients and not every form of singularity elimination can be expected to be equally good at dealing with such additional problems that beset the training of deep networks.
Alternative explanations: Several of our experiments rule out vanishing gradients as the sole explanation for training difﬁculties in deep networks and strongly suggest an independent role for the singularities arising from the non-identiﬁability of the model.
(i)
In Figure 4, all nets have the exact same plain architecture and similarly vanishing gradients at the beginning of training, yet they have diverging performances correlated with measures of distance from singular manifolds.
(ii) Vanishing gradients cannot explain the difference between identity skips and dense orthogonal skips in
Figure 7, because both eliminate vanishing gradients, yet dense orthogonal skips perform better.
(iii)
In Figure 8, spectrum-equalized non-orthogonal skips often have larger gradient norms, yet worse performance than orthogonal skips.
(iv) Vanishing gradients cannot even explain the BiasReg results in Figure 5.
The BiasReg and the plain net have almost identical (and vanishing) gradients early on in training (Figure 6a), yet the former has better performance as predicted by the symmetry-breaking hypothesis.
(v) Similar results hold for two-layer shallow networks where the problem of vanishing gradients does not arise (Supplementary Note 7).
In particular, shallow residual nets are less degenerate and have better accuracy than shallow plain nets; moreover, gradient norms and accuracy are strongly correlated with distance from the overlap manifolds in these shallow nets.
Our malicious initialization experiment with residual nets (Figure 5) suggests that the beneﬁts of skip connections cannot be explained solely in terms of well-conditioning or improved initialization either.
This result reveals a fundamental weakness in purely linear explanations of the beneﬁts of skip connections (Hardt & Ma, 2016; Li et al., 2016).
Unlike in nonlinear nets, improved initialization entirely explains the beneﬁts of skip connections in linear nets (Supplementary Note 5).
A recent paper (Balduzzi et al., 2017) suggested that the loss of spatial structure in the covariance of the gradients, a phenomenon called “shattered gradients”, could be partly responsible for training difﬁculties in deep nonlinear networks.
They argued that skip connections alleviate this problem by essentially making the model “more linear”.
It is easy to see that the shattered gradients problem is distinct from both the vanishing/exploding gradients problem and the degeneracy problems considered in this paper, since shattered gradients arise only in sufﬁciently non-linear deep networks (linear networks do not shatter gradients), whereas vanishing/exploding gradients, as well as the degeneracies considered here, arise in linear networks too.
The relative contribution of each of these distinct problems to training difﬁculties in deep networks remains to be determined.
Symmetry-breaking in other architectures: We only reported results from experiments with fullyconnected networks, but we note that limited receptive ﬁeld sizes and weight sharing between units in a single feature channel in convolutional neural networks also reduce the permutation symmetry in a given layer.
The symmetry is not entirely eliminated since although individual units do not have permutation symmetry in this case, feature channels do, but they are far fewer in number than the number of individual units.
Similarly, a recent extension of the residual architecture called ResNeXt (Xie et al., 2016) uses parallel, segregated processing streams inside the “bottleneck” blocks, which can again be seen as a way of reducing the permutation symmetry inside the block.
Our method of singularity reduction through bias regularization (BiasReg; Figure 5) can be thought of as indirectly putting a prior over the unit activities.
More complicated joint priors over hidden unit responses that favor decorrelated (Cogswell et al., 2015) or clustered (Liao et al., 2016) responses have been proposed before.
Although the primary motivation for these regularization schemes was to improve the generalizability or interpretability of the learned representations, they can potentially be understood from a singularity elimination perspective as well.
For example, a prior that favors decorrelated responses can facilitate the breaking of permutation symmetries and linear dependencies between hidden units.
Our results lead to an apparent paradox: over-parametrization and redundancy in large neural network models have been argued to make optimization easier.
Yet, our results seem to suggest the opposite.
We think that there is an important distinction to be made between data-independent and data-dependent degeneracies.
The degeneracies we have dealt with in this paper are dataindependent in the sense that they arise from intrinsic properties of the model architecture itself (e.g. permutation symmetry of the nodes in a given layer) and hold for any data.
Data-dependent degeneracies, on the other hand, arise from simply having more degrees of freedom than the number of data points.
An example would be ﬁtting two data points with a quadratic model (three parameters).
Although this problem is degenerate, the particular degeneracy that arises depends on the data points.
We conjecture that these two types of degeneracies have different consequences both for the model’s expressive capacity and for optimization under SGD, with data-independent degeneracies reducing the model’s effective expressive capacity and being more harmful for optimization with
SGD.
The results reported here suggest that it could be useful for neural network researchers to pay closer attention to data-independent degeneracies inherent in their models.
As a general design principle, we recommend reducing such degeneracies in a model as much as possible
S UPPLEMENTARY M ATERIALS S UPPLEMENTARY N OTE 1: S INGULARITY OF THE H ESSIAN IN NON - LINEAR MULTILAYER NETWORKS
Because the cost function can be expressed as a sum over training examples it is enough to consider the cost for a single example: E =
12 ||
y − x L
|| 2 ≡ 12 e (cid:62) e where x l are deﬁned recursively as x l = f ( W l − 1 x l − 1 ) for l = 1 , . . .
, L .
We denote the inputs to units at layer l by the vector h l : h l = W l − 1 x l − 1 .
We ignore the biases for simplicity
The derivative of the cost function with respect to a single weight W l ij between layers l and l + 1 is given by:
[Equation]
Now, consider a different connection between the same output unit i at layer l + 1 and a different input unit j (cid:48) at layer l .
The crucial thing to note is that if the units j and j (cid:48) have the same set of incoming weights, then the derivative of the cost function with respect to W l,ij becomes identical to its derivative with respect to W l,ij (cid:48) :
∂E/∂ W l,ij =
∂E/∂
W l,ij (cid:48) .
This is because in this condition x l,j (cid:48) =
x l,j for all possible inputs and all the remaining terms in Equation 1 are independent of the input index j .
Thus, the columns (or rows) corresponding to the connections W l,ij and W l,ij (cid:48) in the Hessian become identical, making the Hessian degenerate.
This is a re-statement of the simple observation that when the units j and j (cid:48) have the same set of incoming weights, the parameters W l,ij and W l,ij (cid:48) become non-identiﬁable (only their sum is identiﬁable).
Thus, this corresponds to an overlap singularity .
A similar argument shows that when a set of units at layer l , say units indexed j , j (cid:48) , j (cid:48)(cid:48) become linearly dependent, the columns of the Hessian corresponding to the weights W l,ij , W l,ij (cid:48) and W l,ij (cid:48)(cid:48) become linearly dependent as well, thereby making the Hessian singular.
Again, this is just a re-statement of the fact that these weights are no longer individually identiﬁable in this case (only a linear combination of them is identiﬁable).
This corresponds to a linear dependence singularity .
In non-linear networks, except in certain degenerate cases where the units saturate together, they may never be exactly linearly dependent, but they can be approximately linearly dependent, which makes the Hessian close to singular.
Moreover, it is easy to see from Equation 1 that, when the presynaptic unit x l,j is always zero, i.e. when that unit is effectively killed, the column (or row) of the Hessian corresponding to the parameter W l,ij becomes the zero vector for any i , and thus the Hessian becomes singular.
This is a re-statement of the simple observation that when the unit x l,j is always zero, its outgoing connections, W l,ij , are no longer identiﬁable.
This corresponds to an elimination singularity .
In the residual case, the only thing that changes in Equation 1 is that the factors W (cid:62) k diag( f (cid:48) k +1 ) on the right-hand side become W (cid:62) k diag( f (cid:48)
k +1 )
+ I where I is an identity matrix of the appropriate size.
The overlap singularities are eliminated, because x l,j (cid:48) and x l,j cannot be the same for all possible inputs in the residual case (even when the adjustable incoming weights of these units are identical).
Similarly, elimination singularities are also eliminated, because x l,j cannot be identically zero for all possible inputs (even when the adjustable incoming weights of this unit are all zero), assuming that the corresponding unit at the previous layer x l − 1 ,j is not always zero, which, in turn, is guaranteed with an identity skip connection if x l − 2 ,j is not always zero etc., all the way down to the ﬁrst hidden layer. Any linear dependence between x l,j , x l,j (cid:48) and x l,j (cid:48)(cid:48) is also eliminated by adding linearly independent inputs to them, assuming again that the corresponding units in the previous layer are linearly independent.
S UPPLEMENTARY N OTE 2: S IMULATION DETAILS
In Figure 3, for the skip connections between non-adjacent layers in the hyper-residual networks, i.e. Q k , we used matrices of the type labeled “32” in Figure 7, i.e. matrices consisting of four copies of a set of 32 orthonormal vectors.
We found that these matrices performed slightly better than orthogonal matrices.
We augmented the training data in both CIFAR-10 and CIFAR-100 by adding reﬂected versions of each training image, i.e. their mirror images.
This yields a total of 100000 training images for both datasets.
The test data were not augmented, consisting of 10000 images in both cases.
We used the standard splits of the data into training and test sets.
For the BiasReg network of Figures 5-6, random hyperparameter search returned the following values for the target bias distributions: µ = 0 .
51 , σ = 0 .
96 for CIFAR-10 and µ = 0 .
91 , σ = 0 .for CIFAR-100.
The toy model shown in Figure 2b-c consists of the simulation of Equations 3 . 7 and 3 . 9 in Wei et al. The toy model shown in Figure 2e is the simulation of learning dynamics in a network with 3 input, 3 hidden and 3 output units, parametrized in terms of the norms and unit-vector directions of J a − J b − J c , J a + J b − J c , J c , and the output weights.
A teacher model with random parameters is ﬁrst chosen and a large set of “training data” is generated from the teacher model.
Then the gradient ﬂow ﬁelds with respect to the two parameters m =
||
J a + J b − J c
||
and || J c || are plotted with the assumption that the remaining parameters are already at their optima (a similar assumption was made in the analysis of Wei et al.
We empirically conﬁrmed that the ﬂow ﬁeld is generic.
S UPPLEMENTARY N OTE 3: E STIMATING THE EIGENVALUE SPECTRAL DENSITY OF THE H ESSIAN IN DEEP NETWORKS
We use Skilling’s moment matching method (Skilling, 1989) to estimate the eigenvalue spectra of the Hessian.
We ﬁrst estimate the ﬁrst few non-central moments of the density by computing m k =
N r (cid:62) H k r where r is a random vector drawn from the standard multivariate Gaussian with zero mean and identity covariance, H is the Hessian and N is the dimensionality of the parameter space.
Because the standard multivariate Gaussian is rotationally symmetric and the Hessian is a symmetric matrix, it is easy to show that m k gives an unbiased estimate of the k -th moment of the spectral density:
[Equation]
where λ i are the eigenvalues of the Hessian, and p ( λ ) is the spectral density of the Hessian as
N
→ ∞ .
In Equation 2, we make use of the fact that ˜ r 2 i are random variables with expected value 1 .
Despite appearances, the products in m k do not require the computation of the Hessian explicitly and can instead be computed efﬁciently as follows:
[Equation]
where the Hessian times vector computation can be performed without computing the Hessian explicitly through Pearlmutter’s R -operator
(Pearlmutter, 1994).
In terms of the vectors v k , the estimates of the moments are given by the following:
[Equation]
For the results shown in Figure 3, we use 20-layer fully-connected feedforward networks and the number of parameters is N = 709652 .
For the remaining simulations, we use 30-layer fullyconnected networks and the number of parameters is N = 874772 .
We estimate the ﬁrst four moments of the Hessian and ﬁt the estimated moments with a parametric density model.
The parametric density model we use is a mixture of a narrow Gaussian distribution (to capture the bulk of the density) and a skew-normal distribution (to capture the tails):
[Equation]
with 4 parameters in total: the mixture weight w , and the location ξ , scale ω and shape α parameters of the skew-normal distribution.
We ﬁx the parameters of the Gaussian component to µ = 0 and σ = 0 .
Since the densities are heavy-tailed, the moments are dominated by the tail behavior of the model, hence the ﬁts are not very sensitive to the precise choice of the parameters of the
Gaussian component.
The moments of our model can be computed in closed-form.
We had difﬁculty ﬁtting the parameters of the model with gradient-based methods, hence we used a simple grid search method instead.
The ranges searched over for each parameter was as follows.
w : logarithmically spaced between 10 − 9 and 10 − 3 ; α : linearly spaced between − 50 and 50 ; ξ : linearly spaced between − 10 and 10 ; ω : logarithmically spaced between 10 − 1 and 10 3 .
57 parameters were evaluated along each parameter dimension for a total of 57
≈ 10M parameter conﬁgurations evaluated.
The estimated moments ranged over several orders of magnitude.
To make sure that the optimization gave roughly equal weight to ﬁtting each moment, we minimized a normalized objective function:
[Equation]
where ˆ m k ( w, α, ξ, ω ) is the model-derived estimate of the k -th moment.
S UPPLEMENTARY N OTE 4: V ALIDATION OF THE RESULTS WITH SMALLER NETWORKS
Here, we validate our main results for smaller, numerically tractable networks.
The networks in this section are 10-layer fully-connected feedforward networks.
The networks are trained on CIFAR100.
The input dimensionality is reduced from 3072 to 128 through PCA.
In what follows, we calculate the fraction of degenerate eigenvalues by counting the number of eigenvalues inside a small window of size 0 . 2 around 0 , and the fraction of negative eigenvalues by the number of eigenvalues to the left of this window.
We ﬁrst compare residual networks with plain networks (Figure S1).
The networks here havehidden units in each layer yielding a total of 4852 parameters.
This is small enough that we can calculate all eigenvalues of the Hessian numerically.
We observe that residual networks have better training and test performance (Figure S1a-b); they are less degenerate (Figure S1d) and have more negative eigenvalues than plain networks (Figure S1c).
These results are consistent with the results reported in Figure 3 for deeper and larger networks.
Next, we validate the results reported in Figure 4 by running 400 independent plain networks and comparing the best-performing 40 with the worst-performing 40 among them (Figure S2).
Again, the networks here have 16 hidden units in each layer with a total of 4852 parameters.
We observe that the best networks are less degenerate (Figure S2d) and have more negative eigenvalues than the worst networks (Figure S2c).
Moreover, the hidden units of the best networks have less overlap (Figure S2f), and, at least initially during training, have slightly larger weight norms than the worstperforming networks (Figure S2e).
Again, these results are all consistent with those reported in Figure 4 for deeper and larger networks.
Finally, using numerically tractable plain networks, we also tested whether we could reliably estimate the fractions of degenerate and negative eigenvalues with our mixture model.
Just as we do for the larger networks, we ﬁrst ﬁt the mixture model to the ﬁrst four moments of the spectral density estimated with the method of Skilling (1989).
We then estimate the fraction of degenerate and negative eigenvalues from the ﬁtted mixture model and compare these estimates with those obtained from the numerically calculated actual eigenvalues.
Because for the larger networks, the networks were found to be highly degenerate, we restrict the analysis here to conditions where the fraction of degenerate eigenvalues was at least 99 . 8% .
We used 10-layer plain networks with 32 hidden units in each layer (with a total of 14292 parameters) for this analysis.
We observe that, at least for these small networks, the mixture model usually underestimates the fraction of degenerate eigenvalues and overestimates the fraction of negative eigenvalues.
However, there is a highly signiﬁcant positive correlation between the actual and estimated fractions (Figure S3). [Figure] [Figure] [Figure]
S UPPLEMENTARY N OTE 5: D YNAMICS OF LEARNING IN LINEAR NETWORKS WITH SKIP CONNECTIONS
To get a better analytic understanding of the effects of skip connections on the learning dynamics, we turn to linear networks.
In an L -layer linear plain network, the input-output mapping is given by (again ignoring the biases for simplicity):
[Equation]
where x 1 and x L are the input and output vectors, respectively.
In linear residual networks with identity skip connections between adjacent layers, the input-output mapping becomes:
[Equation]
Finally, in hyper-residual linear networks where all skip connection matrices are assumed to be the identity, the input-output mapping is given by:
[Equation]
In the derivations to follow, we do not have to assume that the connectivity matrices are square matrices.
If they are rectangular matrices, the identity matrix I should be interpreted as a rectangular identity matrix of the appropriate size.
This corresponds to zero-padding the layers when they are not the same size, as is usually done in practice.
Three-layer networks: Dynamics of learning in plain linear networks with no skip connections was analyzed in Saxe et al.
For a three-layer network ( L = 3 ), the learning dynamics can be expressed by the following differential equations (Saxe et al., 2013):
[Equation]
Here a α and b α are n -dimensional column vectors (where n is the number of hidden units) connecting the hidden layer to the α -th input and output modes, respectively, of the input-output correlation matrix and s α is the corresponding singular value (see Saxe et al.
for further details).
The ﬁrst term on the right-hand side of Equations 10-11 facilitates cooperation between a α and b α corresponding to the same input-output mode α , while the second term encourages competition between vectors corresponding to different modes.
In the simplest scenario where there are only two input and output modes, the learning dynamics of
Equations 10, 11 reduces to:
[Equation]
How does adding skip connections between adjacent layers change the learning dynamics?
Considering again a three-layer network ( L = 3 ) with only two input and output modes, a straightforward extension of Equations 12-15 shows that the learning dynamics changes as follows: [Equation] d
[Equation]
where u 1 and u 2 are orthonormal vectors (similarly for v 1 and v 2 ).
The derivation proceeds essentially identically to the corresponding derivation for plain networks in Saxe et al.
The only differences are: (i) we substitute the plain weight matrices W l with their residual counterparts
W l + I and (ii) when changing the basis from the canonical basis for the weight matrices W 1 , Wto the input and output modes of the input-output correlation matrix, U and V , we note that:
[Equation]
where U and V are orthogonal matrices and the vectors a α , b α , u α and v α in Equations 16correspond to the α -th columns of the matrices W 1 , W (cid:62) 2 , U and V , respectively.
Figure S4 shows, for two different initializations, the evolution of the variables a 1 and a 2 in plain and residual networks with two input-output modes and two hidden units.
When the variables are initialized to small random values, the dynamics in the plain network initially evolves slowly (Figure S4a, blue); whereas it is much faster in the residual network (Figure S4a, red).
This effect is attributable to two factors.
First, the added orthonormal vectors u α and v α increase the initial velocity of the variables in the residual network.
Second, even when we equalize the initial norms of the vectors, a α and a α + v α (and those of the vectors b α and b α + u α ) in the plain and the residual networks, respectively, we still observe an advantage for the residual network (Figure S4b), because the cooperative and competitive terms are orthogonal to each other in the residual network (or close to orthogonal, depending on the initialization of a α and b α ; see right-hand side of Equations 16-19), whereas in the plain network they are not necessarily orthogonal and hence can cancel each other (Equations 12-15), thus slowing down convergence.
Singularity of the Hessian in linear three-layer networks: The dynamics in Equations 10, 11 can be interpreted as gradient descent on the following energy function:
[Equation]
This energy function is invariant to a (simultaneous) permutation of the elements of the vectors a α and b α for all α .
This causes degenerate manifolds in the landscape.
Speciﬁcally, for the permutation symmetry of hidden units, these manifolds are the hyperplanes a αi = a αj ∀ α , for each pair of hidden units i , j (similarly, the hyperplanes b αi = b αj ∀ α ) that make the model non-identiﬁable.
Formally, these correspond to the singularities of the Hessian or the Fisher information matrix.
Indeed, we shall quickly check below that when a αi = a αj ∀ α for any pair of hidden units i , j , the Hessian becomes singular ( overlap singularities ).
The Hessian also has additional singularities at the hyperplanes a αi = 0 ∀ α for any i and at b αi = 0 ∀ α for any i ( elimination singularities ).
Starting from the energy function in Equation 22 and taking the derivative with respect to a single input-to-hidden layer weight, a αi :
[Equation]
and the second derivatives are as follows:
[Equation]
Note that the second derivatives are independent of mode index α , reﬂecting the fact that the energy function is invariant to a permutation of the mode indices.
Furthermore, when b βi = b βj for all β , the columns in the Hessian corresponding to a αi and a αj become identical, causing an additional degeneracy reﬂecting the non-identiﬁability of a αi and a αj .
A similar derivation establishes that a βi = a βj for all β also leads to a degeneracy in the Hessian, this time reﬂecting the non-identiﬁability of b α i and b α j .
These correspond to the overlap singularities.
[Figure] In addition, it is easy to see from Equations 24, 25 that when b αi = 0 ∀ α , the right-hand sides of both equations become identically zero, reﬂecting the non-identiﬁability of a αi for all α .
A similar derivation shows that when a αi = 0 ∀ α , the columns of the Hessian corresponding to b αi become identically zero for all α , this time reﬂecting the non-identiﬁability of b αi for all α .
These correspond to the elimination singularities.
When we add skip connections between adjacent layers, i.e. in the residual architecture, the energy function changes as follows:
[Equation]
and straightforward algebra yields the following second derivatives:
[Equation]
Unlike in the plain network, setting b βi = b βj for all β , or setting b αi = 0 ∀ α , does not lead to a degeneracy here, thanks to the orthogonal skip vectors u β .
However, this just shifts the locations of the singularities.
In particular, the residual network suffers from the same overlap and elimination singularities as the plain network when we make the following change of variables: b β → b β − u β and a β → a β − v β .
Networks with more than three-layers: As shown in Saxe et al.
(2013), in linear networks with more than a single hidden layer, assuming that there are orthogonal matrices R l and R l +1 for each layer l that diagonalize the initial weight matrix of the corresponding layer (i.e. R (cid:62) l +1 W l (0)
R l =
D l is a diagonal matrix), dynamics of different singular modes decouple from each other and each mode α evolves according to gradient descent dynamics in an energy landscape described by (Saxe et al., 2013):
[Equation]
where a αl can be interpreted as the strength of mode α at layer l and N l is the total number of layers.
In residual networks, assuming further that the orthogonal matrices R l satisfy R (cid:62) l +1 R l = I , the energy function changes to:
[Equation]
and in hyper-residual networks, it is:
[Equation]
Figure S5a illustrates the effect of skip connections on the phase portrait of a three layer network.
The two axes, a and b , represent the mode strength variables for l = 1 and l = 2 , respectively: i.e. a ≡ a α 1 and b ≡ a α 2 .
The plain network has a saddle point at (0 , 0)
(Figure S5a; left).
The dynamics around this point is slow, hence starting from small random values causes initially very slow learning.
The network funnels the dynamics through the unstable manifold a = b to the stable hyperbolic solution corresponding to ab = s .
Identity skip connections between adjacent layers in the residual architecture move the saddle point to ( − 1 , − 1) (Figure S5a; middle).
This speeds up the dynamics around the origin, but not as much as in the hyper-residual architecture where the saddle point is moved further away from the origin and the main diagonal to ( − 1 , − 2) (Figure S5a; right).
We found these effects to be more pronounced in deeper networks.
Figure S5b shows the dynamics of learning in 10-layer linear networks, demonstrating a clear advantage for the residual architecture over the plain architecture and for the hyper-residual architecture over the residual architecture. [Figure] a - [Figure]
b a
Plain2 - Residual2 -  2 Hyper − residual Time   u b hyper-residual residual plain Singularity of the Hessian in reduced linear multilayer networks with skip connections: The derivative of the cost function of a linear multilayer residual network (Equation 30) with respect to the mode strength variable at layer i , a i , is given by (suppressing the mode index α and taking τ = 1 ):
[Equation]
32) and the second derivatives are:
[Equation]
[Equation] It is easy to check that the columns (or rows) corresponding to a i and a j in the Hessian become identical when a i = a j , making the Hessian degenerate.
The hyper-residual architecture does not eliminate these degeneracies but shifts them to different locations in the parameter space by adding distinct constants to a i and a j (and to all other variables).
S UPPLEMENTARY N OTE 6: D ESIGNING SKIP CONNECTIVITY MATRICES WITH VARYING DEGREES OF ORTHOGONALITY
We generated the covariance matrix of the eigenvectors by S = QΛQ (cid:62) , where Q is a random orthogonal matrix and Λ is the diagonal matrix of eigenvalues, Λ ii = exp( − τ ( i − 1)) , as explained in the main text.
We ﬁnd the correlation matrix through R =
D − 1 / 2 SD − 1 / 2 where D is the diagonal matrix of the variances: i.e. D ii = S ii .
We take the Cholesky decomposition of the correlation matrix, R = TT (cid:62) .
Then the designed skip connectivity matrix is given by Σ =
TULU − 1 T − 1 , where L and U are the matrices of eigenvalues and eigenvectors of another randomly generated orthogonal matrix, O : i.e. O = ULU (cid:62) .
With this construction, Σ has the same eigenvalue spectrum as O , however the eigenvectors of Σ are linear combinations of the eigenvectors of O such that their correlation matrix is given by R .
Thus, the eigenvectors of Σ are not orthogonal to each other unless τ = 0 .
Larger values of τ yield more correlated, hence less orthogonal, eigenvectors.
S UPPLEMENTARY N OTE 7: V ALIDATION OF THE RESULTS WITH SHALLOW NETWORKS
To further demonstrate the generality of our results and the independence of the problem of singularities from the vanishing gradients problem in optimization, we performed an experiment with shallow plain and residual networks with only two hidden layers and 16 units in each hidden layer.
Because we do not allow skip connections from the input layer, a network with two hidden layers is the shallowest network we can use to compare the plain and residual architectures.
Figure S6 shows the results of this experiment.
The residual network performs slightly better both on the training and test data (Figure S6a-b); it is less degenerate (Figure S6d) and has more negative eigenvalues [Figure] [Figure] (Figure S6c); it has larger gradients (Figure S6e) —note that the gradients in the plain network do not vanish even at the beginning of training— and its hidden units have less overlap than the plain network (Figure S6f).
Moreover, the gradient norms closely track the mean overlap between the hidden units and the degeneracy of the network (Figure S6d-f) throughout training.
These results suggest that the degeneracies caused by the overlaps of hidden units slow down learning, consistent with our symmetry-breaking hypothesis and with the results from larger networks.