Paper:
Deep generative modeling using flows has gained popularity owing to the tractable exact log-likelihood estimation with efficient training and synthesis process. However, flow models suffer from the challenge of having high dimensional latent space, same in dimension as the input space. An effective solution to the above challenge as proposed by Dinh et al. (2016) is a multi-scale architecture, which is based on iterative early factorization of a part of the total dimensions at regular intervals. Prior works on generative flows involving a multi-scale architecture perform the dimension factorization based on a static masking. We propose a novel multi-scale architecture that performs data dependent factorization to decide which dimensions should pass through more flow layers. To facilitate the same, we introduce a heuristic based on the contribution of each dimension to the total log-likelihood which encodes the importance of the dimensions. Our proposed heuristic is readily obtained as part of the flow training process, enabling versatile implementation of our likelihood contribution based multi-scale architecture for generic flow models. We present such implementations for several state-ofthe-art flow models and demonstrate improvements in log-likelihood score and sampling quality on standard image benchmarks. We also conduct ablation studies to compare proposed method with other options for dimension factorization.

Review:
Summary: 
The paper investigates a strategy for factoring out dimensions in multi-scale generative flow models. The strategy relies on a heuristic that employs the availability of per-dimension log-determinant terms (e.g. as affine couplings) to decide which half of the dimensions to factor out. Quantitative experimental results support the proposed strategy’s efficacy.
Recommendation: Weak Reject
The paper misses several important comparisons and baselines - both in terms of placing the proposed method in a broader context (e.g. how could the method be used in multiscale i-ResNet), as well as providing experimental comparisons that would definitively demonstrate the method’s contribution (e.g. is it the heuristic dimension splitting or the unfactored pre-training that provides the benefit.
The applicability of the proposed method appears to be overstated. The paper suggests that the method is applicable to any multi-scale flow architecture. However, it depends on the availability of per-dimension log-determinants. As the paper states, these values are readily available for methods that rely on affine couplings (e.g. RealNVP, Glow), but not for methods such as i-ResNets or FFJORD, that also employ multi-scale architectures to factor out variables.
Major comments:
1. Section 2.1 introduces flows as a mapping from data x to the latent code z, and gives the log-likelihood formula (2) that can be optimized during model training. This presentation is at odds with Section 3, where the authors talk about sampling a latent variable that fixes the “log-latent density” and only leaves the log-determinant term to be optimized (e.g. “[...] the log-latent-density term depends on the choice of the latent distribution and if fixed given the sampled latent variable. [...] once the variable is sampled, maximizing the log-det term results in maximized likelihood.” The two presentations are at odds with each other and should be reconciled, ideally by altering Section 3 since it does not reflect the way flow models are trained.
2. The authors made an implicit assumption that the log-det term is decomposable into per-dimension component. As mentioned earlier in this review, generally speaking this is not true and methods such as i-ResNets or FFJORD do not (directly?) permit such decomposition, while they do employ multi-scale architectures akin to that of RealNVP. This assumption manifests itself in several places in the paper, which are listed below. Either the authors could make this assumption and the limitations imposed by it very clear, or they could provide additional information on how their method extends to methods that do not provide per-dimension log-det contributions.
a) In Section 3 “The total log-det term is nothing by the sum of log-det terms for contributed by each dimension”. This is obvious (and possibly holds) only for affine couplings.
b) In Section 3 “Nevertheless, such an analogy can be extended for other flow models which involve a multiscale architecture [...]”. As mentioned, I am not convinced this is true.
c) In Section 4 “Since our method focuses individually on the dimensions using a heuristic which is always available in flow model training, it can prove to be have more versatility in being compatible with generic multi-scale architecture”. Same as above - in fact I believe that the opposite may be true.
d) Section 4 does not mention methods that employ multi-scale architectures whil not allowing for discerning per-dimension contributions of the log-det. Mentioning these methods is important for accurately positioning the authors contribution relative to the existing literature.
e) The method is only applied to the RealNVP architecture. The claim of it being generally applicable to multi-scale architectures would be much stronger if the comparison would include other multi-scale architectures. Here, I strongly recommend i-ResNets or a similar architecture based on log-det approximation to the entire coupling layer.
3. Unless I missed it, the authors do not mention how they aggregate the log-det values. They are available per datapoint. Are they aggregated over the entire dataset?
4. In Section 4 the authors imply that the contribution of a dimension towards the total likelihood is dominated by the log determinant (“[...] as the contribution by the variable (dimension) towards the total log-det (~ total log-likelihood)”). This is not obvious, and I am not sure it is always true - it probably depends on the choice of the base distribution p(z).
5. The authors show experimentally that the use of their variable factorisation heuristic improves log-likelihoods over the baseline RealNVP model; they further perform ablation studies to confirm that preferential factorisation of low log-det dimensions provides best log-likelihoods and qualitative results on the CelebA dataset. However, several points remain unclear.
a) What are the relative contributions of pre-training without dimension factorisation and the proposed factorisation heuristic? Do the experiments shown in Table 2 all employ the pre-training scheme? If so, this should be clearly stated. If not, this set of experiments should be performed.
b) The networks used in the experiments use between m=1 and 4 factorisation layers. So in total, there are 2^m possible decisions of which half of the variables at those layers to factorise. These numbers are not so large as to prevent us from training all possible combinations of the networks. How does the proposed heuristic compare to just training all (or a few sampled) factorisations decided a priori? This comparison should also investigate the effect of using pre-training.
6. In the qualitative comparison (Section 5.2) the authors argue two points (i) RealNVP with the proposed method generates finer details on CelebA; and (ii) provides a reasonable latent space that can be interpolated in.
a) With regards to (i), I personally find it difficult to argue about fine details in what are effectively low-resolution images. The differences (which are already hard to see and easy to imagine) could be present due to minute changes in the training code/procedure/random seed/etc. I would be more confident about these differences if the authors could confirm that the methods were trained using the same code base, random seeds, dataset shuffle and framework versions.
b) How do the qualitative interpolation results compare to RealNVP without the proposed factorisation heuristic?
Minor comments:
- In Section 2.1 “Let x be a high-dimensional random vector with unknown true distribution x ~ p(x) [...]”. Here x appears to refer to the data or a sample from the data distribution, and not just a random vector. This could be made clearer.
- In Section 2.1 the “jacobian” should be written with a capital “J”
- In Section 2.1 and elsewhere, when denoting outputs of intermediate flows f_i as y_i, it is mentioned that y_0 = x, but not that y_k = z. If the latter is true, mentioning this would improve clarity.
- Section 3 “In a multi-scale architecture, it is apparent that the variables getting exposed to more layers of flow will be more expressive in nature [...]”. I do not find this to be an obvious fact. Perhaps the authors could provide an intuition for why this is true or demonstrate this on a toy example? My confusion may stem from the fact that I am unsure what it means for a variable (i.e. a dimension we factor out) to be more expressive.
- Some phrasing is a little unusual (e.g. “less (more) log-det tems” instead of “smaller (larger) log-det term”; “to be have”).
- In Section 5.1 “code length” is introduced without context. The connection between code length, log-likelihood and compression can be made more clear.
------------------------------------------------
Post rebuttal update
------------------------------------------------
I’d like to thank the authors for additional elaboration on their baseline experiments that consider their heuristic-based scheme for factoring out dimensions (the low-low-low scheme) and the “anti-heuristic” (which considers the high-high-high scheme). I agree with the authors that both schemes would have been within the 2^m possibilities I encourage them to consider, however I do not fully agree with some of the conclusions the authors draw for these experiments.
First, given that the method proposed by the authors is a heuristic, it is not obvious that the high-high-high and the low-low-low schemes indeed correspond the worst and best possible of the 2^m schemes. Running the 2^m experiments could demonstrate this empirically and strengthen the manuscript.
Second, if we speak about the high-high-high and low-low-low factorizations, then we’ve already gone through the computationally expensive effort of pre-training a model that does not factor out any variables. It is precisely because of this that I’d like to see a baseline that doesn’t do any pre-training, but simply enumerates the 2^m schemes and selects the one with the best validation score. It is not obvious that this approach would computationally more expensive than the pre-training step -- another point I believe is important to cover.
To summarize, I do not believe that the baselines currently present in the paper are a substitute for the ones requested.
Including new results requires a review of those new results, so I would not be in favour of adding new results at the camera ready stage. I therefore retain my score.