In a series of recent theoretical works, it has been shown that strongly over-parameterized neural networks trained with gradient-based methods could converge linearly to zero training loss, with their parameters hardly varying. In this note, our goal is to exhibit the simple structure that is behind these results. In a simplified setting, we prove that "lazy training" essentially solves a kernel regression. We also show that this behavior is not so much due to over-parameterization than to a choice of scaling, often implicit, that allows to linearize the model around its initialization. These theoretical results complemented with simple numerical experiments make it seem unlikely that "lazy training" is behind the many successes of neural networks in high dimensional tasks.
REF showed that training over-parameterized neural networks actually belongs to a so-called "lazy training" regime, in which the model behaves like its linearization around the initialization.
On Lazy Training in Differentiable Programming
Advances in Neural Information Processing Systems (NeurIPS), Dec 2019, Vancouver, Canada
Computer Science Mathematics