  Deep networks are typically trained with many more parameters than the size
of the training dataset. Recent empirical evidence indicates that the practice
of overparameterization not only benefits training large models, but also
assists - perhaps counterintuitively - building lightweight models.
Specifically, it suggests that overparameterization benefits model pruning /
sparsification. This paper sheds light on these empirical findings by
theoretically characterizing the high-dimensional asymptotics of model pruning
in the overparameterized regime. The theory presented addresses the following
core question: "should one train a small model from the beginning, or first
train a large model and then prune?". We analytically identify regimes in
which, even if the location of the most informative features is known, we are
better off fitting a large model and then pruning rather than simply training
with the known informative features. This leads to a new double descent in the
training of sparse models: growing the original model, while preserving the
target sparsity, improves the test accuracy as one moves beyond the
overparameterization threshold. Our analysis further reveals the benefit of
retraining by relating it to feature correlations. We find that the above
phenomena are already present in linear and random-features models. Our
technical approach advances the toolset of high-dimensional analysis and
precisely characterizes the asymptotic distribution of over-parameterized
least-squares. The intuition gained by analytically studying simpler models is
numerically verified on neural networks.
