Batch Normalization (BN) improves both convergence and generalization in training neural networks. This work understands these phenomena theoretically. We analyze BN by using a basic block of neural networks, consisting of a kernel layer, a BN layer, and a nonlinear activation function. This basic network helps us understand the impacts of BN in three aspects. First, by viewing BN as an implicit regularizer, BN can be decomposed into population normalization (PN) and gamma decay as an explicit regularization. Second, learning dynamics of BN and the regularization show that training converged with large maximum and effective learning rate. Third, generalization of BN is explored by using statistical mechanics. Experiments demonstrate that BN in convolutional neural networks share the same traits of regularization as the above analyses. * The first three authors contribute equally. Corresponding to pluo.empirically in a recent work (Morcos et al., 2018), but has not been established analytically. (2) σ B reduces kurtosis of the input distribution as well as correlations between neurons. (3) The regularization strengths of these statistics are inversely proportional to the batch size M , indicating that BN with large batch would decrease generalization. (4) Removing either one of µ B and σ B could imped convergence and generalization. • Second, by using ordinary differential equations (ODEs), Sec.3 shows that gamma decay enables the network trained with BN to converge with large maximum and effective learning rate, leading to faster training speed compared to the network trained without BN or trained with weight normalization (WN) (Salimans & Kingma, 2016) that is a counterpart of BN. • Third, Sec.4 compares generalization errors of BN, WN, and vanilla SGD by using statistical mechanics. The "large-scale" regime is of interest, where number of samples P and number of neurons N are both large but their ratio P/N is finite. In this regime, the generalization errors are quantified both analytically and empirically. Numerical results in Sec.5 show that BN in CNNs has the same traits of regularization as disclosed by the above analyses.
al. REF decomposed BN into population normalization (PN) and gamma decay as an explicit regularization, and claimed that BN allowed bigger learning rate in the training process.
Towards Understanding Regularization in Batch Normalization
ArXiv
ArXiv
Computer Science Mathematics