Paper:
Glow: Generative Flow with Invertible 1×1 Convolutions
Flow-based generative models (Dinh et al., 2014) are conceptually attractive due to tractability of the exact log-likelihood, tractability of exact latent-variable inference, and parallelizability of both training and synthesis. In this paper we propose Glow, a simple type of generative flow using an invertible 1 × 1 convolution. Using our method we demonstrate a significant improvement in log-likelihood on standard benchmarks. Perhaps most strikingly, we demonstrate that a flow-based generative model optimized towards the plain log-likelihood objective is capable of efficient realistic-looking synthesis and manipulation of large images. The code for our model is available at https://github.com/openai/glow.

Review:
This paper proposes some modifications to Real NVP (Dinh et al.) and claim improved generative modeling performance. Quantitative evaluations (likelihood scores) are used to compare with Dinh et al. on CelebA and Imagenet 32x32 and Imagenet 64x64.  The proposed improvements are: 1. Using a learnt invertible channel transformation instead of the fixed shuffling used by Dinh et al. 2. Don't use BatchNorm. Replace it with an affine layer whose parameters are initialized to maintain zero-mean unit-variance. This is called ActNorm in the paper. 3. Generalize Dinh's additive coupling layer into an affine coupling layer. 4. The images are compressed into a 5-bit color space instead of the 8-bit color space typically used.  The paper attempts to solve a significant problem (non-autoregressive likelihood-based deep generative models). However, the paper has a number of problems which makes it hard to accept as a NIPS paper.  In short, the quantitative and qualitative evaluation fall short of convincing an informed reader about the usefulness of the idea in the paper. The proposed changes might be useful, but as it stands now, this paper doesn't give any evidence. Let me explain in more detail:  ## Samples   It's not clear how to judge the qualitative samples. The paper only present one set of samples for the same model (the proposed GLOW model) and the same dataset (CelebA 5-bit). To my knowledge, Dinh et al. don't use 5-bit color space and this paper does not provide any samples of any baseline models on the 5-bit color space. So it's simply impossible to properly judge the samples. The paper also doesn't present any nearest-neighbour analysis (in pixel-space or feature-space) to judge overfitting.   ## Compute  It's not clear whether the contributions are due to the problem-level choices made (such as 5-bit color), solution-level choices (1x1 invertible conv, etc.) or simply more compute used for more iterations. The paper doesn't mention any details about how much compute, gpus or iterations were used in comparison to the Dinh et al. baseline. The authors mention multi-GPU experiments which hints that a relatively large amount of compute was used, and thus doesn't give me confidence that the improvement is due to the solution-level changes.  ## Missing Results  The paper presents quantitative results for Imagenet datasets, and 5-bit CelebA, but does not provide samples for the same. This indicates the model did not fare well in terms of sample quality in these other harder datasets, which in itself is okay, but skipping out in providing the results is an attempt at misleading readers. Since sample quality is an important claim of the paper, and failing to provide the same for the other datasets (that were trained and evaluated quantitatively) is conspicuous.  ## Missing Ablations  This is a minor point. The paper makes no attempt at ablating the individual changes proposed. This could have been done on MNIST if compute was an issue.  ## Missing comparison to autoregressive models  This is a minor point. The paper fails to include likelihood scores for autoregressive models in the table of results. While this doesn't matter for informed readers, it misleads readers who are not tracking this literature into thinking that Glow is a state-of-the-art generative model.  # Suggestions for Improvement  1. Compare against the main baseline models (Real NVP) using the same amount of compute. 2. If 1 is not possible, then add a note in the table of results comparing the amount of compute resources used by the two models. 3. Include and discuss samples of the model which were trained on datasets other than CelebA 5-bit. 4. Include nearest-neighbour analysis to judge overfitting.  # Summary  The paper presents some intuitive modifications to Dinh et al's real NVP model. However, the evaluations fall short of supporting any of the claims made in the paper.  ---- # After Author Reponse  I had missed figure 3 in my previous read. Figure 3 does provide evidence that the 1x1 convolution improves NLL. I assume this is a controlled setting, i.e. all other variables are the same. Perhaps it would have been useful to mention this in the paper. Since the baseline result is same as the one in the Real NVP paper, it's easy to conclude that the baseline results are picked from the Real NVP paper, and not reproduced in a controlled setting. I will update my review, since this part of the review was based on an oversight.  The other parts of my review however still stand. It will be more informative both to reviewers and to readers if the authors also include samples from the controlled baselines (i.e. the reverse and shuffle baselines). I'm sure the authors are well-aware that NLL is not the best metric to compare across different models. Fairly trivial changes in modeling assumptions can hurt or improve NLL scores with pretty much no change in subjective quality of samples. Since we are in the business of evaluating samples subjectively, let's do it somewhat more objectively, by including samples from the baseline. One of the central points in my review was that the authors to change the problem itself by using a 5-bit color space. All the samples shown in the paper are from the 5-bit model. So, while I can judge that the samples "subjectively realistic-looking", it's not clear whether this is due to the choice to shift from 8-bit to 5-bit or the improved modeling. This point remains unaddressed and un-acknowledged by the authors.  In summary, I acknowledge that I had missed figure 3's significance in my previous read, and that does improve my judgement of the paper, but the paper still lacks sufficient rigor in evaluation to be accepted as a NIPS paper.