Paper:
Analyzing Inverse Problems with Invertible Neural Networks
For many applications, in particular in natural science, the task is to determine hidden system parameters from a set of measurements. Often, the forward process from parameter-to measurement-space is well-defined, whereas the inverse problem is ambiguous: multiple parameter sets can result in the same measurement. To fully characterize this ambiguity, the full posterior parameter distribution, conditioned on an observed measurement, has to be determined. We argue that a particular class of neural networks is well suited for this task -so-called Invertible Neural Networks (INNs). Unlike classical neural networks, which attempt to solve the ambiguous inverse problem directly, INNs focus on learning the forward process, using additional latent output variables to capture the information otherwise lost. Due to invertibility, a model of the corresponding inverse process is learned implicitly. Given a specific measurement and the distribution of the latent variables, the inverse pass of the INN provides the full posterior over parameter space. We prove theoretically and verify experimentally, on artificial data and real-world problems from medicine and astrophysics, that INNs are a powerful analysis tool to find multi-modalities in parameter space, uncover parameter correlations, and identify unrecoverable parameters.

Review:
1) Summary
The authors propose to use invertible networks to solve ambiguous inverse problems. This is done by training one group of Real-NVP output variables supervised while training the other group via maximum likelihood under a Gaussian prior as done in the standard Real-NVP. Further, the authors suggest to not only train the forward model, but also the inverse model with an MMD critic, similar to previous works that used a more flexible GAN critic [1].
2) Clarity
The paper is easy to understand and the main idea is well-motivated. 
3) Significance
The main contribution of this work is of conceptual nature and illustrates how invertible networks are a promising framework for many inverse problems. I really like the main idea and think it is inspiring. However, the experiments and technical contributions are rather limited. 
Theoretical / ML contribution: 
Using an MMD to factorize groups of latent variables is well-known and combining flow-based maximum likelihood training in the forward model with GAN-like objectives in the inverse model has been done before as well.
Experimental contribution: 
I am not fully convinced by the experiments. 
The inverse kinematics experiment shows that the posterior collapses from large uncertainty to almost a point for the right-most joint. This seems like a negative result to me. 
The medical experiment also seems rather limited, because if I understand correctly the tissue data is artificial and the proposed INN only outperforms competitors (despite ABC) on two out of three measurements. Further, the authors should have explained the experimental setup of the tissue experiment better, as it is not a standard task in the field. 
In the astronomy experiment figure 4 shows strong correlations between some of the z variables, the authors claim that this is a feature of their method, but I argue that they should not be present if training with the factorial prior was successful. It would be good to show the correlation between y and z variables as well if they show high dependencies, learning was not very successful. Simply eyeballing the shape of the posterior is not enough to conclude independence. 
In summary, even though interesting, the significance of the experimental results is hard to judge and I am a bit worried that if the proposed model is making some strange mistakes on artificial toy-data, how well it will perform on challenging realistic problems. 
4) Main Concerns
The authors claim that specifying a prior/posterior distribution in density modeling is complicated and typically the chosen distributions are too simplistic. This argument is, of course, valid, but they also have the same problem and specify z to be factorial Gaussian. So the same "hen-and-egg" problem applies here.
The authors also seem to suggest that they are the first to train flow-based models in forward and inverse direction, but this has already been done in the flow-GAN paper [1].
MMD does not easily scale to high-dimensional problems, this is not a problem here as all artificial problems considered are very low-dimensional. But when applying the proposed algorithm in realistic settings, one will likely need extensions of MMD, like used in MMD GANs, which would introduce min/max games on both sides of the network. This will likely be hard to train and constitutes a fundamental limitation of the approach that needs to be discussed.
5) Minor Concerns
- Some basic citations on normalizing flows seem to be missing, e.g. [2,3].
- How does one guarantee that padded regions are actually zero on output when padding input with zeros? Small variance in those dimensions could potentially code important information. Is this considered as part of y or z?
- The authors require the existence of inverse and set this equal to bijectivity, but injectivity would be sufficient.
- The authors mention that z is conditioned on y, but in their notation, the conditional density p(z|y) never shows up explicitly. It should be made clear, that p(z)=p(z|y) is a consequence of their additional MMD penalty and only holds at convergence.
[1] Grover et al., "Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models"
[2] Tabak and Turner, "Density estimation by dual ascent of the log-likelihood"
[3] Deco and Brauer, "Nonlinear higher-order statistical decorrelation by volume-conserving neural architectures"