Why do deep neural networks generalize with a very high dimensional parameter space? We took an information theoretic approach. We find that the dimensionality of the parameter space can be studied by singular semi-Riemannian geometry and is upper-bounded by the sample size. We adapt Fisher information to this singular neuromanifold. We use random matrix theory to derive a minimum description length of a deep learning model, where the spectrum of the Fisher information matrix plays a key role to improve generalisation.
Also an information-theoretic approach, REF derived a novel formulation of the minimum description length in the context of deep learning by utilizing tools from singular semi-Riemannian geometry.
Lightlike Neuromanifolds , Occam ’ s Razor and Deep Learning