  We study batch normalisation in the context of variational inference methods
in Bayesian neural networks, such as mean-field or MC Dropout. We show that
batch-normalisation does not affect the optimum of the evidence lower bound
(ELBO). Furthermore, we study the Monte Carlo Batch Normalisation (MCBN)
algorithm, proposed as an approximate inference technique parallel to MC
Dropout, and show that for larger batch sizes, MCBN fails to capture epistemic
uncertainty. Finally, we provide insights into what is required to fix this
failure, namely having to view the mini-batch size as a variational parameter
in MCBN. We comment on the asymptotics of the ELBO with respect to this
variational parameter, showing that as dataset size increases towards infinity,
the batch-size must increase towards infinity as well for MCBN to be a valid
approximate inference technique.
