Paper:
Learning Latent Subspaces in Variational Autoencoders
Variational autoencoders (VAEs) [10,20] are widely used deep generative models capable of learning unsupervised latent representations of data. Such representations are often difficult to interpret or control. We consider the problem of unsupervised learning of features correlated to specific labels in a dataset. We propose a VAE-based generative model which we show is capable of extracting features correlated to binary labels in the data and structuring it in a latent subspace which is easy to interpret. Our model, the Conditional Subspace VAE (CSVAE), uses mutual information minimization to learn a low-dimensional latent subspace associated with each label that can easily be inspected and independently manipulated. We demonstrate the utility of the learned representations for attribute manipulation tasks on both the Toronto Face [23] and CelebA [15]  datasets.

Review:
Updated (due to rebuttal & discussion w/ R2):  The authors reiterate in their rebuttal their core contributions of "extracting information beyond binary labels" and "attribute manipulation from a single image", together with the promise to clarify it in the paper. The contributions are relevant to the community, since this form of hierarchical disentangling seems novel.  That said, there is some degree of similarity of the proposed variational approach to IGN (Deep Convolutional Inverse Graphics Network https://arxiv.org/abs/1503.03167). IGN is cited, but not discussed in detail, and an empirical comparison is not provided, despite being applicable to the current setting as well.  Nevertheless, since the selling point of the paper seems to be the ability to discover sub-categories from only category labels, which is not addressed in IGN and is an interesting empirical find, I increased my score to be marginally above the acceptance threshold.  ---  The paper proposes an approach for learning variational auto-encoders for labeled data in which the latent space consists of two parts: one that captures the labels information and one that captures the remaining information for producing the data. Along with the variational lower-bound corresponding to the introduced latent variables, the proposed training objective additionally penalizes the mutual information between the latent variable corresponding to the remaining data and the labels. The approach is illustrated on a toy example and two image datasets, the Toronto Faces dataset and CelebA.  I think the approach is described clearly, and several experiments, qualitative and quantitative, are presented. That said, since there are many works on the topic, some seemingly quite similar in the use of variational auto-encoders with various regularizers for disentangling latent representations, I worry that the proposed approach only incrementally contributes to the topic. See below for specific questions on this matter.  Comments: * The work on building fair representations (where the learned representations are encouraged to be independent of "sensitive" factors) seems relevant and not discussed. Could you highlight the distinction? For instance, The Variational Fair Autoencoder [1] seems relevant, where the additional "fairness" regularizer uses maximum mean discrepancy rather than mutual information. From this perspective, I believe that the contribution of the current paper is the use of the mutual information regularizer, and a discussion on the benefits of one metric versus the other would be useful. From a quick search, the idea of using mutual information to "obfuscate" sensitive information is also mentioned in [3], though in a somewhat different setting. * In the related work section, I wish the authors provided more detail of the proposed approach with respect to the cited works (e.g. w.r.t. [21-23] and w.r.t. [3]) so that we can understand which specific contributions are being made and how to evaluate their merits. For instance, the fact that previous work used GANs rather than VAEs (lines 61-62) is a pretty weak point in favor of the paper, unless properties of VAEs are shown to be necessary for solving the task, or the method shown to empirically perform much better than the GAN counterparts.  Experiments: * I like the toy example, it's illustrative and a good sanity check for the method. * None of the presented applications is focused on the specific contributions of this paper, e.g. the additional mutual information regularizer. Instead, the experiments are quite generic for attribute manipulation/style transfer which have been presented in many works recently, e.g. the ones mentioned in the related work. From this perspective, the empirical contribution does not stand out.  [1] https://arxiv.org/abs/1511.00830 ICLR 2016 [2] https://www.cs.toronto.edu/~toni/Papers/icml-final.pdf ICML 2013  Summary: The submission is ok, but, unless the contributions are highlighted more clearly, it only incrementally contributes to the literature, hence the slightly negative score.