  Recent work has highlighted the role of initialization scale in determining
the structure of the solutions that gradient methods converge to. In
particular, it was shown that large initialization leads to the neural tangent
kernel regime solution, whereas small initialization leads to so called "rich
regimes". However, the initialization structure is richer than the overall
scale alone and involves relative magnitudes of different weights and layers in
the network. Here we show that these relative scales, which we refer to as
initialization shape, play an important role in determining the learned model.
We develop a novel technique for deriving the inductive bias of gradient-flow
and use it to obtain closed-form implicit regularizers for multiple cases of
interest.
