Inspired by BatchNorm, there has been an explosion of normalization layers in deep learning. Recent works have identified a multitude of beneficial properties in BatchNorm to explain its success. However, given the pursuit of alternative normalization techniques, these properties need to be generalized so that any given layer's success/failure can be accurately predicted. In this work ,  we take a first step towards this goal by extending known properties of BatchNorm in randomly initialized deep neural networks (DNNs) to nine recently proposed normalization layers. Our primary findings follow : (i) Similar to BatchNorm, activations-based normalization layers can avoid exploding activations in ResNets  ; (ii) Use of GroupNorm ensures rank of activationsis at least \Omega(\sqrt{\frac{\text{width ; (iii) Small group sizes result in large gradient norm in earlier layers, hence justifying training instability issues in Instance Normalization and illustrating a speed-stability tradeoff in GroupNorm. Overall, our analysis reveals several general mechanisms that explain the success of normalization techniques in deep learning, providing us with a compass to systematically explore the vast design space of DNN normalization layers.
Inspired by BatchNorm, there has been an explosion of normalization layers for deep neural networks (DNNs). However , these alternative normalization layers have seen minimal use, partially due to a lack of guiding principles that can help identify when these layers can serve as a replacement for BatchNorm. To address this problem, we take a theoretical approach, generalizing the known beneficial mechanisms of BatchNorm to several recently proposed normalization techniques. Our generalized theory leads to the following set of principles : (i) similar to BatchNorm, activations-based normalization layers can prevent exponential growth of activations in ResNets , but parametric layers require explicit remedies ; (ii) use of GroupNorm can ensure informative forward propagation, with different samples being assigned dissimilar activations, but increasing group size results in increasingly indistinguishable activations for different samples, explaining slow convergence speed in models with LayerNorm ; (iii) small group sizes result in large gradient norm in earlier layers, hence explaining training instability issues in Instance Normalization and illustrating a speed-stability tradeoff in GroupNorm. Overall, our analysis reveals a unified set of mechanisms that underpin the success of normalization methods in deep learning, providing us with a compass to systematically explore the vast design space of DNN normalization layers.
R
in deep learning. Recent works have identified a multitude of beneficial properties in BatchNorm to explain its success. However, given the pursuit of alternative normalization techniques, these properties need to be generalized so that any given layer's success/failure can be accurately predicted. In this work
for deep neural networks (DNNs). However
A
these alternative normalization layers have seen minimal use, partially due to a lack of guiding principles that can help identify when these layers can serve as a replacement for BatchNorm. To address this problem,
R
first step towards this goal by extending known properties of BatchNorm in randomly initialized deep neural networks (DNNs) to nine
theoretical approach, generalizing the known beneficial mechanisms of BatchNorm to several
R
layers. Our primary findings follow
techniques. Our generalized theory leads to the following set of principles
R
Similar
similar
R
avoid exploding
prevent exponential growth of
A
, but parametric layers require explicit remedies
R
Use of GroupNorm ensures rank of activationsis at least \Omega(\sqrt{\frac{\text{width
use of GroupNorm can ensure informative forward propagation, with different samples being assigned dissimilar activations, but increasing group size results in increasingly indistinguishable activations for different samples, explaining slow convergence speed in models with LayerNorm
R
Small
small
R
justifying
explaining
R
several general mechanisms that explain
a unified set of mechanisms that underpin
R
techniques
methods