The Fisher information matrix (FIM) is a fundamental quantity to represent the characteristics of a stochastic model, including deep neural networks (DNNs). The present study reveals novel statistics of FIM that are universal among a wide class of DNNs. To this end, we use random weights and large width limits, which enables us to utilize mean field theories. We investigate the asymptotic statistics of the FIM's eigenvalues and reveal that most of them are close to zero while the maximum takes a huge value. This implies that the eigenvalue distribution has a long tail. Because the landscape of the parameter space is defined by the FIM, it is locally flat in most dimensions, but strongly distorted in others. We also demonstrate the potential usage of the derived statistics through two exercises. First, small eigenvalues that induce flatness can be connected to a norm-based capacity measure of generalization ability. Second, the maximum eigenvalue that induces the distortion enables us to quantitatively estimate an appropriately sized learning rate for gradient methods to converge.
In specific, REF applied mean field theory to study the statistics of the spectrum and the appropriate size of the learning rate.
Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
ArXiv
ArXiv
Computer Science Mathematics Physics