Paper:
We argue that flow-based density models based on continuous bijections are limited in their ability to learn target distributions with complicated topologies, and propose localised generative flows (LGFs) to address this problem.
LGFs are composed of stacked continuous mixtures of bijections, which enables each bijection to learn a local region of the target rather than its entirety. Our method is a generalisation of existing flow-based methods, which can be used without modification as the basis for an LGF model. Unlike normalising flows, LGFs do not permit exact computation of log likelihoods, but we propose a simple variational scheme that performs well in practice. We show empirically that LGFs yield improved performance across a variety of density estimation tasks.

Review:
Summary:
This paper conjectures that normalizing flows are fundamentally limited due to the architecture assumption that the generative function g is continuous in x. It is argued that this constraint makes maximum likelihood estimation difficult in general. Localised generative flows are proposed as a solution and consist in modeling the generative model as a continuous mixture of bijections. Experiments suggest an improvement over MAF.
Decision:
The observation that continuity imposes a hard constraint on the network is sound, and the proposed solution appears to show some improvement. However, in its current state, this work appears to be quite fragile both from a theoretical and experimental point of view. First, it is only conjectured that this constraint poses actual problems. Second, the experimental evaluation is weak and insufficient. It omits comparisons with more recent generative flows that have shown to be able to model discontinuous densities. For this reason, I do not recommend the paper for acceptance.
Further arguments:
- The whole paper rests on intuition without strong theoretical backup. 
- The experiments are quite poor and results frankly oversold. It is said the method "improves performance across a variety of common density benchmarks". While we see improvements in Table 1 over MAF, the comparison omits all recent architectures based on Normalizing Flows, such as TAN (Olivia et al, 2018), NAF (Huang et al, 2018), B-NAF (De Cao et al, 2019) or SOS (Jaini et al, 2019). All of those methods have reported better results than those provided in Table 1. They have also been shown empirically to work for discontinuous densities. While I understand that LGF can be combined with any flow architecture, the question remains whether using a continuous mixture translates into significant improvements for those baselines as well. The experimental benchmarks also omit datasets such as BSDS300, for which the higher dimensionality is usually challenging. The same goes for Table 2 which omits recent and better results, such as Glow or FFJORD.
- Closer to LGF, a proper experimental comparison to RAD (Dinh et al, 2019) would be appreciated.
- The proposed architecture supposedly enables better generative models. However, this comes at the price that the density can no longer be evaluated exactly and analytically. Since normalizing flows are also typically slow for sampling, this makes the benefits of the proposed architecture quite limited. In particular, it is not clear why generative models that are good at sampling only (e.g., GANs) should not then be preferred?
- As a result of the point above, the experimental results are reported only in terms of approximated negative log-likelihood. I do not think this is fair, since models like MAF do provide exact values. It also makes the comparison with previous methods more difficult.
Further feedback:
- As per ICLR policy, higher standards should be applied to papers with 9 or more pages. I am confident the paper could be written within 8 pages only.