  We present a plug-in replacement for batch normalization (BN) called
exponential moving average normalization (EMAN), which improves the performance
of existing student-teacher based self- and semi-supervised learning
techniques. Unlike the standard BN, where the statistics are computed within
each batch, EMAN, used in the teacher, updates its statistics by exponential
moving average from the BN statistics of the student. This design reduces the
intrinsic cross-sample dependency of BN and enhance the generalization of the
teacher. EMAN improves strong baselines for self-supervised learning by 4-6/1-2
points and semi-supervised learning by about 7/2 points, when 1%/10% supervised
labels are available on ImageNet. These improvements are consistent across
methods, network architectures, training duration, and datasets, demonstrating
the general effectiveness of this technique.
