  In spite of advances in understanding lazy training, recent work attributes
the practical success of deep learning to the rich regime with complex
inductive bias. In this paper, we study rich regime training empirically with
benchmark datasets, and find that while most parameters are lazy, there is
always a small number of active parameters which change quite a bit during
training. We show that re-initializing (resetting to their initial random
values) the active parameters leads to worse generalization. Further, we show
that most of the active parameters are in the bottom layers, close to the
input, especially as the networks become wider. Based on such observations, we
study static Layer-Wise Sparse (LWS) SGD, which only updates some subsets of
layers. We find that only updating the top and bottom layers have good
generalization and, as expected, only updating the top layers yields a fast
algorithm. Inspired by this, we investigate probabilistic LWS-SGD, which mostly
updates the top layers and occasionally updates the full network. We show that
probabilistic LWS-SGD matches the generalization performance of vanilla SGD and
the back-propagation time can be 2-5 times more efficient.
