Paper:
Recent deep generative models are able to provide photo-realistic images as well as visual or textual content embeddings useful to address various tasks of computer vision and natural language processing. Their usefulness is nevertheless often limited by the lack of control over the generative process or the poor understanding of the learned representation. To overcome these major issues, very recent work has shown the interest of studying the semantics of the latent space of generative models. In this paper, we propose to advance on the interpretability of the latent space of generative models by introducing a new method to find meaningful directions in the latent space of any generative model along which we can move to control precisely specific properties of the generated image like the position or scale of the object in the image. Our method does not require human annotations and is particularly well suited for the search of directions encoding simple transformations of the generated image, such as translation, zoom or color variations. We demonstrate the effectiveness of our method qualitatively and quantitatively, both for GANs and variational auto-encoders.  (Brock et al., 2018), showing that the position of the object can be controlled within the image.

Review:
This paper proposes a method to learn and control continuous factors of variations within generative models by finding meaningful directions in the latent space which correspond to specified properties. A new method is proposed for inverting generative models and embedding images in the latent space when an encoder is not available. Specifically, reconstruction error is defined in the Fourier domain such that the weighting on high frequency image components can be reduced. Results are evaluated with qualitative comparison to previous embedding methods. Using this image embedding technique, a dataset of latent space trajectories is created by manipulating a desired property in images (such as position or scale) via affine transformations and recording the latent space vectors of the original and new images. The dataset is then used to learn a simple model of the latent space transformation corresponding to changes in the desired image property, which in turn can be used to manipulate images accordingly. To evaluate the effectiveness of this image manipulation approach, a saliency detector is used to measure the change in position or scale of objects in generated images as the latent codes are changed.
Overall, I would tend towards accepting this work. The goal of being able to manipulate continuous factors of variation within generative models is useful for controllable image synthesis, and the proposed method clearly achieves the desired result.
Things to improve the paper:
1) The paper proposes a new reconstruction error metric which is optimized to embed images into the latent space of the generative models. While this new metric is compared qualitatively to existing methods, quantitative evaluation is lacking. It would be useful to also include quantitative comparison of methods measuring the perceptual distance between the original image and the embedded image, perhaps by using Learned Perceptual Image Patch Similarity (LPIPS) [1].
Minor things to improve the paper that did not impact the score:
2) In the abstract: "Our method is weakly supervised...". I am not sure if this method would be considered weakly supervised. I might tend more towards calling it self-supervised, since we have exact labels that are derived from transformations applied to the images themselves.
3) In the first paragraph of the introduction: "an increasing number of applications are emerging such as image in-painting, dataset-synthesis, deep-fakes... ". I find the use of the ellipses here to be a bit strange, since it seems like the sentence is trailing off mid-thought. I would recommend the use of "etc." over "...".
4) In Section 2.2, second paragraph, the dSprite dataset is mentioned but not cited. The reference is not given until Section 3. Should the citation be paired with the first mention of the dataset? Or even just in both places.
5) In Section 3, Implementation details: "The first part is injected at the bottom layer while next parts are used to modify the style of the generated image thanks to AdaIN layers (Huang & Belongie, 2017)". BigGAN uses conditional BatchNorm instead of AdaIN, although they are both very similar. I think the proper citation here is [2], which first introduced conditional BatchNorm.
Questions:
6) I am not fully convinced of the argument that using a saliency detector makes the method more general purpose than a dedicated object detector. The majority of high quality generative models are class conditional, hence requiring a labelled dataset, and therefore an object detector can easily be trained on the same dataset. Additionally, Section 3.2 mentions that "We performed quantitative analysis on ten chosen categories for which the object can be easily segmented by using saliency detection approach", which seems to indicate that the saliency detector struggles with some objects. How does the saliency detector perform on more complicated objects?
References:
[1] Zhang, Richard, et al. "The unreasonable effectiveness of deep features as a perceptual metric." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
[2] De Vries, Harm, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C. Courville. "Modulating early visual processing by language." In Advances in Neural Information Processing Systems, pp. 6594-6604. 2017.
### Post-Rebuttal Comments ###
Thanks you for addressing my concerns and for adding the quantitative reconstructions measures. Appendix C looks much more complete now. My overall opinion of the paper remains about the same, so I will leave my score unchanged.