Paper:
CAUSALGAN: LEARNING CAUSAL IMPLICIT GENER- ATIVE MODELS WITH ADVERSARIAL TRAINING
We introduce causal implicit generative models (CiGMs): models that allow sampling from not only the true observational but also the true interventional distributions. We show that adversarial training can be used to learn a CiGM, if the generator architecture is structured based on a given causal graph. We consider the application of conditional and interventional sampling of face images with binary feature labels, such as mustache, young. We preserve the dependency structure between the labels with a given causal graph. We devise a two-stage procedure for learning a CiGM over the labels and the image. First we train a CiGM over the binary labels using a Wasserstein GAN where the generator neural network is consistent with the causal graph between the labels. Later, we combine this with a conditional GAN to generate images conditioned on the binary labels. We propose two new conditional GAN architectures: CausalGAN and CausalBEGAN. We show that the optimal generator of the CausalGAN, given the labels, samples from the image distributions conditioned on these labels. The conditional GAN combined with a trained CiGM for the labels is then a CiGM over the labels and the generated image. We show that the proposed architectures can be used to sample from observational and interventional image distributions, even for interventions which do not naturally occur in the dataset.

Review:
In their paper "CausalGAN: Learning Causal implicit Generative Models with adv. training" the authors address the following issue: Given a causal structure between "labels" of an image (e.g. gender, mustache, smiling, etc.), one tries to learn a causal model between these variables and the image itself from observational data. Here, the image is considered to be an effect of all the labels. Such a causal model allows us to not only sample from conditional observational distributions, but also from intervention distributions. These tasks are clearly different, as nicely shown by the authors' example of "do(mustache = 1)" versus "given mustache = 1" (a sample from the latter distribution contains only men). The paper does not aim at learning causal structure from data (as clearly stated by the authors). The example images look convincing to me.
I like the idea of this paper. IMO, it is a very nice, clean, and useful approach of combining causality and the expressive power of neural networks. The paper has the potential of conveying the message of causality into the ICLR community and thereby trigger other ideas in that area. For me, it is not easy to judge the novelty of the approach, but the authors list related works, none of which seems to solve the same task. The presentation of the paper, however, should be improved significantly before publication. (In fact, because of the presentation of the paper, I was hesitating whether I should suggest acceptance.) Below, I give some examples (and suggest improvements), but there are many others. There is a risk that in its current state the paper will not generate much impact, and that would be a pity. I would therefore like to ask the authors to put a lot of effort into improving the presentation of the paper. 
- I believe that I understand the authors' intention of the caption of Fig. 1, but "samples outside the dataset" is a misleading formulation. Any reasonable model does more than just reproducing the data points. I find the argumentation the authors give in Figure 6 much sharper. Even better: add the expression "P(male = 1 | mustache = 1) = 1". Then, the difference is crystal clear.
- The difference between Figures 1, 4, and 6 could be clarified.    
- The list of "prior work on learning causal graphs" seems a bit random. I would add Spirtes et al 2000, Heckermann et al 1999, Peters et al 2016, and Chickering et al 2002. 
- Male -> Bald does not make much sense causally (it should be Gender -> Baldness)... Aha, now I understand: The authors seem to switch between "Gender" and "Male" being random variables. Make this consistent, please. 
- There are many typos and comma mistakes. 
- I would introduce the do-notation much earlier. The paragraph on p. 2 is now written without do-notation ("intervening Mustache = 1 would not change the distribution"). But this way, the statements are at least very confusing (which one is "the distribution"?).
- I would get rid of the concept of CiGM. To me, it seems that this is a causal model with a neural network (NN) modeling the functions that appear in the SCM. This means, it's "just" using NNs as a model class. Instead, one could just say that one wants to learn a causal model and the proposed procedure is called CausalGAN? (This would also clarify the paper's contribution.)
- many realizations = one sample (not samples), I think. 
- Fig 1: which model is used to generate the conditional sample?  
- The notation changes between E and N and Z for the noises. I believe that N is supposed to be the noise in the SCM, but then maybe it should not be called E at the beginning. 
- I believe Prop 1 (as it is stated) is wrong. For a reference, see Peters, Janzing, Scholkopf: Elements of Causal Inference: Foundations and Learning Algorithms (available as pdf), Definition 6.32. One requires the strict positivity of the densities (to properly define conditionals). Also, I believe the Z should be a vector, not a set. 
- Below eq. (1), I am not sure what the V in P_V refers to.
- The concept of data probability density function seems weird to me. Either it is referring to the fitted model, then it's a bad name, or it's an empirical distribution, then there is no pdf, but a pmf.
- Many subscripts are used without explanation. r -> real? g -> generating? G -> generating? Sometimes, no subscripts are used (e.g., Fig 4 or figures in Sec. 8.13)
- I would get rid of Theorem 1 and explain it in words for the following reasons. (1) What is an "informal" theorem? (2) It refers to equations appearing much later. (3) It is stated again later as Theorem 2. 
- Also: the name P_g does not appear anywhere else in the theorem, I think. 
- Furthermore, I would reformulate the theorem. The main point is that the intervention distributions are correct (this fact seems to be there, but is "hidden" in the CIGN notation in the corollary).
- Re. the formulation in Thm 2: is it clear that there is a unique global optimum (my intuition would say there could be several), thus: better write "_a_ global minimum"?
- Fig. 3 was not very clear to me. I suggest to put more information into its caption. 
- In particular, why is the dataset not used for the causal controller? I thought, that it should model the joint (empirical) distribution over the labels, and this is part of the dataset. Am I missing sth?
- IMO, the structure of the paper can be improved. Currently, Section 3 is called "Background" which does not say much. Section 4 contains CIGMs, Section 5 Causal GANs, 5.1. Causal Controller, 5.2. CausalGAN, 5.2.1. Architecture (which the causal controller is part of) etc. An alternative could be: 
Sec 1: Introduction 
Sec 1.1: Related Work
Sec 2: Causal Models
Sec 2.1: Causal Models using Generative Models (old: CIGM)
Sec 3: Causal GANs
Sec 3.1: Architecture (including controller)
Sec 3.2: loss functions 
...
Sec 4: Empricial Results (old: Sec. 6: Results)
- "Causal Graph 1" is not a proper reference (it's Fig 23 I guess). Also, it is quite important for the paper, I think it should be in the main part. 
- There are different references to the "Appendix", "Suppl. Material", or "Sec. 8" -- please be consistent (and try to avoid ambiguity by being more specific -- the appendix contains ~20 pages). Have I missed the reference to the proof of Thm 2?
- 8.1. contains copy-paste from the main text.
- "proposition from Goodfellow" -> please be more precise
- What is Fig 8 used for? Is it not sufficient to have and discuss Fig 23? 
- IMO, Section 5.3. should be rewritten (also, maybe include another reference for BEGAN).
- There is a reference to Lemma 15. However, I have not found that lemma.
- I think it's quite interesting that the framework seems to also allow answering counterfactual questions for realizations that have been sampled from the model, see Fig 16. This is the case since for the generated realizations, the noise values are known. The authors may think about including a comment on that issue.
- Since this paper's main proposal is a methodological one, I would make the publication conditional on the fact that code is released.