Paper:
Disentangling factors of variation in deep representations using adversarial training
We introduce a conditional generative model for learning to disentangle the hidden factors of variation within a set of labeled observations, and separate them into complementary codes. One code summarizes the specified factors of variation associated with the labels. The other summarizes the remaining unspecified variability. During training, the only available source of supervision comes from our ability to distinguish among different observations belonging to the same class. Examples of such observations include images of a set of labeled objects captured at different viewpoints, or recordings of set of speakers dictating multiple phrases. In both instances, the intra-class diversity is the source of the unspecified factors of variation: each object is observed at multiple viewpoints, and each speaker dictates multiple phrases. Learning to disentangle the specified factors from the unspecified ones becomes easier when strong supervision is possible. Suppose that during training, we have access to pairs of images, where each pair shows two different objects captured from the same viewpoint. This source of alignment allows us to solve our task using existing methods. However, labels for the unspecified factors are usually unavailable in realistic scenarios where data acquisition is not strictly controlled. We address the problem of disentaglement in this more general setting by combining deep convolutional autoencoders with a form of adversarial training. Both factors of variation are implicitly captured in the organization of the learned embedding space, and can be used for solving single-image analogies. Experimental results on synthetic and real datasets show that the proposed method is capable of generalizing to unseen classes and intra-class variabilities.

Review:
This paper proposes a method to separate style and content in a neural representation. This is achieved by defining a latent code with vectors for specified ("content") and unspecified ("style") variation. A GAN loss encourages samples from the unspecified vector conditioned on the specified vector to be indistinguishable from empirical samples from a specified content class. This way the unspecified vector is encouraged to model all variation not related to class. Unlike previous methods, this is achieved without forcing the latent code to explicitly include a categorical class variable. Experiments show that style and content are successfully separated on a few simple datasets. 
I found this to be a quite interesting paper. It is well-written, except for many typos throughout, which are easily fixable. I did, however, find the model somewhat hard to follow. I think it would help to clarify early on what is the difference between using s in the latent code versus using a class label indicator. This distinction strikes me as a primary contribution of the paper.  The model is novel to my knowledge, and uses GANs in an intriguing way. The experiments are convincing that style and content were successfully separated.  I think the main weakness of this paper is that the experiments do not convincingly show that the novelty of the model paid off. A primary novelty is that the proposed model does not require class labels in the code layer. Instead the code layer encodes class-specific information as a continuous vector of features. The paper argues that an advantage might be that the model can be applied to novel classes, not seen at training time. Unfortunately, the experiments show little evidence that this actually works. The NORB experiments are the only ones that test performance on novel classes, and in these examples it does not look like style is well separated from content. As a result, it is not clear what is gained by coding class-specific features s rather than using the older approach of coding a discrete class label in the latent code. The paper does, however, show that using s allows for smooth interpolation between classes, which is an interesting property.  Overall this paper has demonstrated a novel method to disentangle factors of variation, employing a GAN, and this method has potential advantages in terms of how it codes class-specific information (and that it doesn't rely on aligned training data). I think the paper could be improved by including more tests of generalization to novel classes (surely this works on MNIST?), and perhaps finding other ways to exploit the novel encoding of class-specific information.