Paper:
Style transfer usually refers to the task of applying color and texture information from a specific style image to a given content image while preserving the structure of the latter. Here we tackle the more generic problem of semantic style transfer: given two unpaired collections of images, we aim to learn a mapping between the corpus-level style of each collection, while preserving semantic content shared across the two domains. We introduce XGAN, a dual adversarial autoencoder, which captures a shared representation of the common domain semantic content in an unsupervised way, while jointly learning the domain-to-domain image translations in both directions. We exploit ideas from the domain adaptation literature and define a semantic consistency loss which encourages the model to preserve semantics in the learned embedding space. We report promising qualitative results for the task of face-to-cartoon translation. The cartoon dataset we collected for this purpose will also be released as a new benchmark for semantic style transfer.

Review:
This paper proposes a new GAN-based model for unpaired image-to-image translation. The model is very similar to DTN [Taigman et al. 2016] except with trained encoders and a domain confusion loss to encourage the encoders to map source and target domains to a shared embedding. Additionally, an optional teacher network is introduced, but this feels rather tangential and problem-specific. The paper is clearly presented and I enjoyed the aesthetics of the figures. The method appears technically sound, albeit a bit complicated. The new cartoon dataset is also a nice contribution.
My main criticism of this paper is the experiments. At the end of reading, I don’t know clearly which aspects of the method are important, why they are important, and how the proposed system compares against past work. First, the baselines are insufficient. Only DTNs are compared against, yet there are many other recent methods for unpaired image-to-image translation, notably, cycle-consistency-based methods and UNIT. These methods should also be compared against, as there is little evidence that DTNs are actually SOTA on cartoons (rather, the cartoon dataset was not public so other papers did not compare on that dataset). Second, although I appreciated the ablation experiments, they are not comprehensive, as discussed more below. Third, there is no quantitative evaluation. The paper states that quantifying performance on style transfer is an unsolved problem, but this is no excuse for not at least trying. Indeed, there are many proposed metrics in the literature for quantifying style transfer / image generation, including the Inception score [Salimans et al. 2016], conditional variants like the FCN-score [Isola et al. 2017], and human judgments. These metrics could all be adapted to the present task (with appropriate modifications, e.g., switching from Inception to a face attribute classifier). Additionally, as the paper mentions at the end, the method could be applied to domain adaptation, where plenty of standard metrics and benchmarks exist.
Ultimately, the qualitative results in the paper are not convincing to me. It’s hard to see the advantages/disadvantages in each comparison. For example in Figure 8, it’s hard to even see any overall change in the outputs due to ablating the semantic consistency loss and the teacher loss (especially since I’m comparing these to Figure 6, which is referred to “Selected results” and therefore might not be a fair comparison). Perhaps the effect of the ablations would be clearer if the figures showed a single input followed by a series of outputs for that same input, each with a different term ablated. A careful reader might be able to examine the images for a long time and find some insights, but it would be much better if the paper distilled these insights into a more concise and convincing form. I feel sort of like I’m looking at raw data, and it still needs to be analyzed.
I also think the ablations are not sufficiently comprehensive. In particular, there is no ablation of the domain adversarial loss. This seems like an important one to test since it’s one of the main differences from DTNs. I was a bit confused by the “finetuned DTN” in Section 7.2. Is this an ablation experiment where the domain adversarial loss and teacher loss are removed? If so, referring to it as so may be clearer than calling it a finetuned DTN. Interestingly, the results of this method look pretty decent, suggesting that the domain adversarial loss might not be having a big effect, in which case XGAN looks very close indeed to DTNs. It would be great here to actually quantify the mentioned sensitivity to hyperparameters.
In terms of presentation, at several points, the paper argues that previous, pixel-domain methods are more limited than the proposed feature-space method, but little evidence is given to support these claims. For example, “we argue that such a pixel-level constraint is not sufficient in our case” in the intro, and “our proposed semantic consistency loss acts at the feature level, allowing for more flexible transformations” in related work. I would like to see more motivation for these assertions, and ultimately, the limitations should be concretely demonstrated in experiments. In models like CycleGAN the pixel-level constraint is between inputs and reconstructed inputs, and I don’t see why this necessarily is overly restrictive on the kinds of transformations in the outputs. The phrasing in the current paper seems to suggest that the pixel-level constraints are between input and output, which, I agree, would be directly restrictive. The reasoning here should be clarified. Better yet would be to provide empirical evidence that pixel-domain methods are not successful (e.g., by comparing against CycleGAN).
The usage of the term “semantic” is also somewhat confusing. In what sense is the latent space semantic? The paper should clarify exactly what this term refers to, perhaps simply defining it to mean a “low-dimensional shared embedding.”
I think the role of the GAN objective is somewhat underplayed. It is quite interesting that the current model achieves decent results even without the GAN. However, there is no experiment keeping the GAN but ablating other parts of the method. Other papers have shown that a GAN objective plus, e.g., cycle-consistency, can do quite well on this kind of problem. It could be that different terms in the current objective are somewhat redundant, so that you can choose any two or three, let’s say, and get good results. To check this, it would be great to see more comprehensive ablation experiments. 
Minor comments:
1. Page 1: I wouldn’t call colorization one-to-one. Even though there is a single ground truth, I would say colorization is one-to-many in the sense that many outputs may be equally probable according to a Bayes optimal observer.
2. Fig 1: It should be clarified that the left example is not a result of the method. At a glance this looks like an exciting new result and I think that could mislead casual readers.
3. Fig 1 caption: “an other” —> “another”
4. Page 2: “Recent work … fail for more general transformations” — DiscoGAN (Kim et al. 2017) showed some success beyond pixel-aligned transformations
5. Page 5: “particular,the” —> “particular, the”; quotes around “short beard” are backwards
6. Page 6: “founnd” —> “found”
7. Page 11: what is \mathcal{L}_r? I don’t see it defined above.