  Modern deep neural networks are highly over-parameterized compared to the
data on which they are trained, yet they often generalize remarkably well. A
flurry of recent work has asked: why do deep networks not overfit to their
training data? We investigate the hypothesis that deeper nets are implicitly
biased to find lower rank solutions and that these are the solutions that
generalize well. We prove for the asymptotic case that the percent volume of
low effective-rank solutions increases monotonically as linear neural networks
are made deeper. We then show empirically that our claim holds true on finite
width models. We further empirically find that a similar result holds for
non-linear networks: deeper non-linear networks learn a feature space whose
kernel has a lower rank. We further demonstrate how linear
over-parameterization of deep non-linear models can be used to induce low-rank
bias, improving generalization performance without changing the effective model
capacity. We evaluate on various model architectures and demonstrate that
linearly over-parameterized models outperform existing baselines on image
classification tasks, including ImageNet.
