Paper:
Likelihood from a generative model is a natural statistic for detecting out-ofdistribution (OoD) samples. However, generative models have been shown to assign higher likelihood to OoD samples compared to ones from the training distribution, preventing simple threshold-based detection rules. We demonstrate that OoD detection fails even when using more sophisticated statistics based on the likelihoods of individual samples. To address these issues, we propose a new method that leverages batch normalization. We argue that batch normalization for generative models challenges the traditional i.i.d. data assumption and changes the corresponding maximum likelihood objective. Based on this insight, we propose to exploit in-batch dependencies for OoD detection. Empirical results suggest that this leads to more robust detection for high-dimensional images.

Review:
The paper makes the observation that likelihood models trained with batch norm assign much lower likelihoods to "training batches" of OoD data (batch norm statistics computed over over minibatch) than evaluation batches of OoD data (batch norm statistics over entire training set).
One issue with comparing this method to most other OoD detection works is that it considers OoD detection on *batches* of (all OoD data) or (all in-distribution data). As soon as the problem is changed to "classify between OoD batches" and not single samples, there are a large number of possible statistical tests one can perform to perform OoD (T-test between likelihoods of each batch) and the problem becomes *much* easier. In some ways, this makes things more well-defined (hard to compare distributions when one of them is just a single sample from an arbitrary distribution). 
However, that brings me to a big concern I have with the evaluation protocol. The batch size used for train/evaluation is rather large (64, this detail is hidden in the Appendix and I would have appreciated the number put in the main experiments section). If you take a likelihood model and evaluate on 64 samples from SVHN, you are all but guaranteed to sample a sample with *exceedingly* low likelihood, which dominates the mean statistic, making it possible to separate SVHN batch from CIFAR10 batches. I suspect that OoD datasets have plenty of these "extremely low likelihood" examples that will drag the mean likelihood down a lot.
This is consistent with your batch normalization experiments: in training mode, the likelihood is computed from mean activations over a batch of OoD samples, several of which probably contribute to the low likelihoods. In evaluation mode, likelihoods for each OoD sample are evaluated independently, which results in a similar observation to prior work showing that CIFAR10 likelihoods are inaccurate for SVHN. In other words, I think there is a mistake made here: it is the phenomenon that *batch likelihoods*, not *batch norm*, that is responsible for this method working well. One experiment that is missing from your paper (and would prove my hypothesis wrong) would be if you adapted the OoD criteria to compare the *mean* likelihoods in the evaluation mode, and show that for OoD datasets, the difference between batches still remains small.  
Nits:
- "...such as learning a mixture of Gaussians", I believe this toy example was on univariate gaussians, not mixtures.
- Choi et al. 2018 and should also be included in the citation that "CIFAR10 gives higher likelihood estimates to SVHN than CIFAR10 ones" (this was a concurrent discovery between the two papers)
- Choi et al. 2018 is not the right citation for "we evaluate the area under the ROC curve (AUC) and average precision (AP)" for each binary classification task, a more appropriate one would be Hendryks and Gimpel 2017.
- No doubt the authors realized already but Page 7 has some "related work still needs some work, but there...." which should be deleted.
- It took me awhile to find the batch size used in training and evaluation mode (64), which was on page 16.