Paper:
We look at the eigenvalues of the Hessian of a loss function before and after training. The eigenvalue distribution is seen to be composed of two parts, the bulk which is concentrated around zero, and the edges which are scattered away from zero. We present empirical evidence for the bulk indicating how overparametrized the system is, and for the edges that depend on the input data.
We look at the eigenvalues of the Hessian of a loss function before and after training. The eigenvalue distribution is seen to be composed of two parts, the bulk which is concentrated around zero, and the edges which are scattered away from zero. We present empirical evidence for the bulk indicating how overparametrized the system is, and for the edges that depend on the input data.
1 INTRODUCTION
Given a (piece-wise) differentiable loss function, and a gradient based algorithm to minimize it, the knowledge of the second order information about it can tell us quite a bit about how the landscape looks like, and how we could modify our algorithm to make it go faster and find better solutions. But, one of the biggest challenges in second order optimization methods is in accessing that second order information itself. In particular, in deep learning there have been many proposals to accelerate training using second order information. Ngiam et al. (2011) has an in depth review of some of the proposals for approximating the Hessian of the loss function. Nevertheless, given the computational complexity of the problems at hand, it is hard to acquire information on what the actual Hessian looks like. This work is part of a series of papers that explore the data-model-algorithm connection along with Sagun et al. (2014; 2015) and it builds on top of the intuition developed in LeCun et al. (2012). We also note that the singularity of the Fisher information matrix have been explored (see for instance Watanabe (2007)). In another recent work, Dauphin et al. (2014) investigate saddle points of the landscape, in particular, they locate saddle points that are near the training path. In this work, however, we strictly focus on the Hessian of the loss function at the exact point of the training. We train the main examples using gradient descent. We perform our calculations of the Hessian using the implementation for the exact Hessian vector product that has been introduced in Pearlmutter (1994). And we find two new observations: one where the eigenvalues are zero, and one where we have a large, positive, and discrete set of eigenvalues.
In this short note, we show how the data and the architecture depends on the eigenvalues of the Hessian of the loss function. In particular, we observe that the top discrete eigenvalues depend on the data, and the bulk of the eigenvalues depend on the architecture. Furthermore, as we keep growing the size of the network, we observe that the discrete part that depends on data remains the same, but the concentration around zero sharpens.
There are various conclusions and implications of this singularity. Recent research suggest new insights into convergence properties of gradient based algorithms in non-convex systems (Lee et al., 2016; Hardt et al., 2015). The results come together with their implications on neural networks. However, the proofs require the system at hand to be non-degenerate. An immediate conclusion of our observation is that the Hessian of the loss function is very singular. Therefore, a lot of the theory and methodology that assumes non-singular Hessian cannot be applied without an appropriate modification.
2 THE CASE OF THE FULLY-CONNECTED NETWORK
2.1 MNIST WITH INCREASING SIZES OF HIDDEN LAYERS
We calculate the exact Hessian of the loss function of a network with two hidden layers. The inputs are 1000 randomly selected examples of 28 ∗ 28 MNIST data, the network has one hidden layer with ReLU nonlinearity, the top layer has a softmax and a negative log likelihood loss function at the end. We train the system with gradient descent (i.e. with minibatch size equal to the number of examples). We plot the histogram of the eigenvalues of the Hessian for a varying number of hidden units after convergence. The Hessian at the end of the training turns out to be extremely singular, and increasing the number of units in hidden layers only add to the singularity of the Hessian (see figure 1). The effect of the training on the eigenvalue spectrum of the model with 10 hidden-units is visible when comparing figure 1 and figure 2 (left).
2.2 VARYING THE DATA
To demonstrate how the eigenvalue distribution may depend on data itself, we keep the same architecture and change the inputs to random patterns. Initially, a random point in the weight space is selected, and we calculated the Hessian without any training (first two histograms of figure 2). After training the system until the norm of the gradient is close to zero. We again calculate the exact Hessian and plot the histogram of their eigenvalues which can be seen in the last one in figure 2.
3 A SIMPLER CASE
In this section, we will repeat the same experiment in two-dimensional data, in an attempt to understand better the connection between the data and the spectrum of the Hessian. A simple figure can be seen in figure 3. We create two Gaussian blobs, centered at (1, 1) and (−1,−1), and first we keep the standard deviation the same, and increase the network size.
The network architecture is similar, this time with two hidden layers and a fully connected network with ReLU nonlinearities including a softmax at the top layer combined with a negative loglikelihood loss function. We train the system with gradient descent with constant step size. At the end of the training the norm of the gradient is at the order of 10−4.
There are two eigenvalues that are isolated, and away from the bulk of the spectrum. Increasing the network only adds to the concentration of eigenvalues at and around zero (see figure 4). To give an insight into how the Hessian’s themselves look like, in figure 6 we plot the full Hessian matrices for three of the systems above after training.
Moreover, this property of the singular and discrete parts is not specific to the log loss. In figure 5, we plot the histogram for a system that is trained on the same data as in figure 3, and the same
architecture. But the training is carried out with the mean square loss rather than the negative loglikelihood. Consistent with our previous observations, we still see the same discrete, data-dependent part, and the part that is at zero.
The training procedure itself acts like a process by which the eigenvalues concentrate at zero. To demonstrate this in further detail we calculated the full Hessian peridoically throughout the training. In figure 7 we plot the eigenvalue profile as the training progresses.
All of the training has been done with random initializations on the weight space with the same standard deviation. In other words, initial points are randomly chosen on the surface of a sphere with a fixed radius given the total number of parameters. This begs the question of the effect of the choice of the initial point. Therefore, now we fix the network size, and repeat the experiment with different random initializations over 5K times. In figure 8 we plot the fluctuations of the top eigenvalue.
The next question is how the spectrum responds to the increased complexity of data. The notion of complexity for a given dataset can be tricky to describe, here we use a loose notion of complexity to point out the fact that the more complex data is the less separable one. To this end, we keep the architecture the same, and increase the standard deviation of the two Gaussian blobs. They are still centered at the same two points, but it becomes harder to separate them as they merge together. Gradient descent still converges to a low-cost value, but the error is higher, and it can’t learn how to separate them perfectly as the blobs merge together. In figure 9, we observe that the top two
eigenvalues grow significantly, and beyond its natural fluctuations due to the initialization. We also note that the norm of the weights are similar for all the cases, therefore the growth in the sizes of eigenvalues can not solely be accounted for the growth in weights.
4 CONCLUSION
We show that the Hessian of the loss functions in deep learning is degenerate. This has implications on the theoretical work which requires improvements in its premises. One such step has been taken in Panageas & Piliouras (2016) in relaxing the isolated singularity condition that was assumed in Lee et al. (2016). From a practical point of view this has multiple implications:
• The landscape may be flat beyond the notion of wide basins. • Training stops at a point that has a small gradient. The norm of the gradient is not zero,
therefore it does not, technically speaking, converge to a critical point. • There are still negative eigenvalues even when they are small in magnitude.
This suggests that we may be able to look beyond the classical notions of basins when exploring the energy landscapes of loss functions. Next obvious question is to find low energy paths between
solutions to show the kind of flatness in such landscapes. This will be explored in a subsequent work in the same series.
We also demonstrate the two phases of the spectrum, one that is concentrated around zero that depends on the size of the model, and the second part that is away from the bulk of the spectrum, that is isolated and depends on the data.
This kind of two-phased non-degeneracy can, in fact, be a desirable property. A degenerate Hessian implies locally flat regions. A degenerate Hessian at the scale that we observe in deep learning may imply flat regions across space, at the global scale.
• We can devise separate methods for the directions that correspond to the top eigenvalues. • We can take advantage of the directions that correspond to the zero or small eigenvalues by
attempting to find paths of low energies in the weight space.
As a first step to the last item, initial experiments are promising: Let’s take a random point on the weight space and train two systems from that point: (1) with gradient descent, and (2) with stochastic gradient descent. At each step, take a straight line between the two points and interpolate the cost value. The resulting profile is completely flat even when the points keep diverging from one another. Next, take two random initial points on the weight space, so now they are orthogonal to each other. And train two systems with different shuffling of data for SGD. This time one would expect the line interpolation to give arbitrary values since the initial points are completely orthogonal, surprisingly, the line interpolation also decreases albeit not as flat as the previous one. Further considerations on
connecting paths between solutions in the weight space of loss functions can be found in Freeman & Bruna (2016).
ACKNOWLEDGMENTS
We would like to thank Afonso Bandeira, Yann Dauphin, Ruoyu Sun, Arthur Szlam and Soumith Chintala for valuable discussions. We also thank the reviewers for valuable feedback. Part of the research has been conducted when the first author was an intern at FAIR.

Review:
Studying the Hessian in deep learning, the experiments in this paper suggest that the eigenvalue distribution is concentrated around zero and the non zero eigenvalues are related to the complexity of the input data. I find most of the discussions and experiments to be interesting and insightful. However, the current paper could be significantly improved.
Quality:
It seems that the arguments in the paper could be enhanced by more effort and more comprehensive experiments. Performing some of the experiments discussed in the conclusion could certainly help a lot. Some other suggestions:
1- It would be very helpful to add other plots showing the distribution of eigenvalues for some other machine learning method for the purpose of comparison to deep learning.
2- There are some issues about the scaling of the weights and it make sense to normalize the weights each time before calculating the Hessian otherwise the result might be misleading.
3- It might worth trying to find a quantity that measures the singularity of Hessian because it is difficult to visually conclude something from the plots.
4- Adding some plots for the Hessian during the optimization is definitely needed because we mostly care about the Hessian during the optimization not after the convergence.
Clarity:
1- There is no reference to figures in the main text which makes it confusing for the reading to know the context for each figure. For example, when looking at Figure 1, it is not clear that the Hessian is calculated at the beginning of optimization or after convergence.
2- The texts in the figures are very small and hard to read.