Paper:
Image-to-image translation for cross-domain disentanglement
Deep image translation methods have recently shown excellent results, outputting high-quality images covering multiple modes of the data distribution. There has also been increased interest in disentangling the internal representations learned by deep methods to further improve their performance and achieve a finer control. In this paper, we bridge these two objectives and introduce the concept of crossdomain disentanglement. We aim to separate the internal representation into three parts. The shared part contains information for both domains. The exclusive parts, on the other hand, contain only factors of variation that are particular to each domain. We achieve this through bidirectional image translation based on Generative Adversarial Networks and cross-domain autoencoders, a novel network component. Our model offers multiple advantages. We can output diverse samples covering multiple modes of the distributions of both domains, perform domainspecific image transfer and interpolation, and cross-domain retrieval without the need of labeled data, only paired images. We compare our model to the state-ofthe-art in multi-modal image translation and achieve better results for translation on challenging datasets as well as for cross-domain retrieval on realistic datasets.
In this paper, we combine the disentanglement objective with image-to-image translation, and introduce the concept of cross-domain disentanglement. The aim is to disentangle the domain specific factors from the factors that are shared across the domains. To do so, we partition the representation into three parts; the shared part containing information that is common to both domains, and two exclusive parts, which only represent those factors of variation that are particular to each domain (see example in figure 1). Cross-domain disentanglement for image-to-image translation has several advantages that allow for applications that would otherwise not be feasible: (i) Sample diversity: we can generate a distribution of images conditioned on the input image, whereas most image-to-image architectures can only generate deterministic results [23,49]. Our approach is similar to the recent work of Zhu et al. [50], although we explicitly model variations in both domains, whereas they only consider variations in the output domain; (ii) Cross domain retrieval: we can retrieve similar images in both domains based on the part of the representation that is shared between the domains, and, unlike [4], we do not require labeled data to learn the shared representation; (iii) Domain-specific image transfer: domain-specific features can be transferred between images; and (iv) Domain-specific interpolation: we can interpolate between two images with respect to domain-specific features.
Our model is based on bidirectional image translation across domains, using a pair of Generative Adversarial Networks (GANs) [17]. We enforce a disentangled structure in the learned representation through an adequate combination of multiple losses and a new network component called crossdomain autoencoder. We demonstrate the disentanglement properties of our method on variations on the MNIST dataset [26], and apply it to bidirectional multi-modal image translation in more complex datasets [3,38], achieving better results than state-of-the-art methods [23,50] due to the finer control and generality granted by our disentangled representation. Additionally, we outperform [50] in cross-domain retrieval on realistic datasets [23,45]. Our code and models are publicly available at https://github.com/agonzgarc/cross-domain-disen.

Review:
In overall, I think this paper proposes a nice framework to learn the disentangled features. The authors addressed my questions in the rebuttal. And I suggest the author add the original BicycleGAN (with skip connections) results to the final version if the paper is accepted. Besides, I also suggest the author soften the claim about GRL loss or explain when the GRL loss can improve the performance and when not.  My final rating is accept. ======================================== This paper proposes a framework to disentangle the domain-exclusive factor and domain-shared factors for paired image-to-image translation task.   Pros: * The proposed cross-domain autoencoder pipeline is an interesting way to implement the domain-shared constraint. It seems a good alternative to the weight-sharing technique which is used in previous image-to-image translation and domain adaptation works. * Empirical experiments show that the proposed method can learn the disentangled features successfully.  Cons: * In the abstract line 13, it is mentioned that the proposed method can perform cross-domain retrieval without the need of labeled. I am confused about this setting because the proposed method does need paired data for training, right? Besides, in line 269, it is mentioned that “Moreover, we do not need image labels, as opposed to specialized approaches such as [13].” However, [13] does not need paired data. So, what does the image labels mean here? * In line 28, it is mentioned that “In spite of the current popularity of these models, the learned representation (the output of the encoder), has not been studied.” I do not agree with this strong statement, since there are many related works on disentangled representation learning just not for image translation specifically, e.g. 1) Disentangled Representation Learning GAN for Pose-Invariant Face Recognition, in CVPR’17; 2) Disentangled person image generation, in CVPR’18. * In line 298, the authors remove the skip connections in [5,12] in order to do a fair comparison. I do not think it is a fair setting. Because the skip connections are very important for the architecture used in [5,12], while the proposed method cannot utilize the skip connections.   Given that this paper is about the disentangled image-to-image translation, it is suggested that some recent work on disentangled representation learning can be included in the related work, e.g.,   [a] K. Bousmalis, G. Trigeorgis, N. Silberman, D. Krishnan and D. Erhan. “Domain separation networks.” In NIPS, 2016. [b] L. Ma, Q. Sun, S. Georgoulis, L. Van Gool, B. Schiele and M. Fritz. “Disentangled person image generation.” In CVPR, 2018. [c] Y.C. Liu, Y.Y. Yeh, T.C. Fu, S.D. Wang, W.C. Chiu and Y.C. Frank Wang. “Detach and Adapt: Learning Cross-Domain Disentangled Deep Representation.” In CVPR, 2018.