Paper:
Batch Normalization (BN) is one of the most widely used techniques in Deep Learning field. But its performance can awfully degrade with insufficient batch size. This weakness limits the usage of BN on many computer vision tasks like detection or segmentation, where batch size is usually small due to the constraint of memory consumption. Therefore many modified normalization techniques have been proposed, which either fail to restore the performance of BN completely, or have to introduce additional nonlinear operations in inference procedure and increase huge consumption. In this paper, we reveal that there are two extra batch statistics involved in backward propagation of BN, on which has never been well discussed before. The extra batch statistics associated with gradients also can severely affect the training of deep neural network. Based on our analysis, we propose a novel normalization method, named Moving Average Batch Normalization (MABN). MABN can completely restore the performance of vanilla BN in small batch cases, without introducing any additional nonlinear operations in inference procedure. We prove the benefits of MABN by both theoretical analysis and experiments. Our experiments demonstrate the effectiveness of MABN in multiple computer vision tasks including ImageNet and COCO.

Review:
The paper extends recently proposed BatchRenormalization (BRN) technique which uses exponential moving average (EMA) statistics in forward and backward passes of BatchNorm (BN) instead of vanilla batch statistics. Motivation of the work is to stabilize training neural networks on small batch size setup. Authors propose to replace EMA in backward pass by simple moving average (SMA) and show that under some assumptions such replacement reduces variance. Also they consider slightly different way of normalization without centralizing features X, but centralizing convolutional kernels according to Qiao et al. (2019).
Concerns:
Changing batch statistics with moving averages in Eq. (4) and Eq. (16) may introduce bias in stochastic gradients. Authors do not reflect this problem in the work.
Assumption (3) from Theorem 3 does not hold in practice since authors centralize weights not features. Authors do not study the influence of this on the performance of the method.
Authors’ main motivation is to study small batch size experimental setup which is important in segmentation and detection, but they don’t include main competitor (BRN) in these experiments.
Overall, the proposed method has a lack novelty and thorough comparison against BRN. Therefore, I would suggest rejecting the current version.
--------------------------------------------------------
Update after author rebuttal
Thank you for your clarification. Below I provide an updated review for the paper.
The paper proposes the improvement of batch normalization techniques for the case of small batch size. Authors reveal new statistics used in gradient calculation of original BatchNorm and show their instability in case of small batch size. Thereafter they improve the method by reducing the number of statistics in backward pass resulting in more stability and performance increase. The authors provide a good experimental study on the influence of individual parts of the method. However, I still have concerns about the strictnesses of theorems assumptions and interpretability of their results in practice (i.e., o(1) biases in Theorem 1, o(1) assumptions in Theorem 2).
Overall, almost all of my concerns were justified. Therefore I increase my score to 6.