We present PYHESSIAN, a new scalable framework that enables fast computation of Hessian (i.e., secondorder derivative) information for deep neural networks. This framework is developed in Pytorch, and it enables distributedmemory execution on both cloud and supercomputer systems. PYHESSIAN enables fast computations of the top Hessian eigenvalues, the Hessian trace, and the full Hessian eigenvalue/spectral density. This general framework can be used to analyze neural network models, including the topology of the loss landscape (i.e., curvature information) to gain insight into the behavior of different models/optimizers. To illustrate this, we apply PYHES-SIAN to analyze the effect of residual connections and Batch Normalization layers on the smoothness of the loss landscape during training. One recent claim, based on simpler first-order analysis, is that residual connections and Batch Normalization make the loss landscape "smoother," thus making it easier for Stochastic Gradient Descent to converge to a good solution. We perform an extensive analysis of this hypothesis, on four residual networks (ResNet20/32/38/56) on the Cifar-10/100 dataset, by measuring directly the Hessian spectrum using PYHESSIAN. This analysis leads to finer-scale insight, demonstrating that while conventional wisdom is sometimes validated, in other cases it is simply incorrect. In particular, we find that Batch Normalization layers do not necessarily make the loss landscape smoother, especially for shallow networks. Instead, the claimed smoother loss landscape only becomes evident for deeper neural networks, at least within this ResNet series. We have open-sourced our PYHESSIAN framework for Hessian spectrum computation in [1].
This was later confirmed for deep NN models by measuring the Hessian spectrum of the network with/without BN REF .
PyHessian: Neural Networks Through the Lens of the Hessian
ArXiv
ArXiv
Mathematics Computer Science