Paper:
We capitalize on the natural compositional structure of images in order to learn object segmentation with weakly labeled images. The intuition behind our approach is that removing objects from images will yield natural images, however removing random patches will yield unnatural images. We leverage this signal to develop a generative model that decomposes an image into layers, and when all layers are combined, it reconstructs the input image. However, when a layer is removed, the model learns to produce a different image that still looks natural to an adversary, which is possible by removing objects. Experiments and visualizations suggest that this model automatically learns object segmentation on images labeled only by scene better than baselines.

Review:
This paper creates a layered representation in order to better learn segmentation from unlabeled images. It is well motivated, as Fig. 1 clearly shows the idea that if the segmentation was removed properly, the result would still be a natural image. However, the method itself as described in the paper leaves many questions about whether they can achieve the proposed goal.
I cannot see from the formulation why would this model work as it is advertised. The formulation (3-4) looks like a standard GAN, with some twist about measuring the GAN loss in the z space (this has been used in e.g. PPGN and CVAE-GAN). I don't see any term that would guarantee:
1) Each layer is a natural image. This was advertised in the paper, but the loss function is only on the final product G_K. The way it is written in the paper, the result of each layer does not need to go through a discriminator. Nothing seems to have been done to ensure that each layer outputs a natural image.
2) None of the layers is degenerate. There does not seem to be any constraint either regularizing the content in each layer, or preventing any layer to be non-degenerate.
3) The mask being contiguous. I don't see any term ensuring the mask being contiguous, I imagine normally without such terms doing such kinds of optimization would lead to a lot of fragmented small areas being considered as the mask.
The claim that this paper is for unsupervised semantic segmentation is overblown. A major problem is that when conducting experiments, all the images seem to be taken from a single category, this implicitly uses the label information of the category. In that regard, this cannot be viewed as an unsupervised algorithm.
Even with that, the results definitely looked too good to be true. I have a really difficult time believing why such a standard GAN optimization would not generate any of the aforementioned artifacts and would perform exactly as the authors advertised. Even if it does work as advertised, the utilization of implicit labels would make it subject to comparisons with a lot of weakly-supervised learning papers with far better results than shown in this paper. Hence I am pretty sure that this is not up to the standards of ICLR.
I have read the rebuttal and still not convinced. I don't think the authors managed to convince me that this method would work the way it's advertised. I also agree with Reviewer 2 that there is a lack of comparison against baselines.