Paper:
Deep latent variable models, such as variational autoencoders, have been successfully used to disentangle factors of variation in image datasets. The structure of the representations learned by such models is usually observed after training and iteratively refined by tuning the network architecture and loss function. Here we propose a method that can explicitly place information into a specific subset of the latent variables. We demonstrate the use of the method in a task of disentangling global structure from local features in images. One subset of the latent variables is encouraged to represent local features through an auxiliary modelling task. In this auxiliary task, the global structure of an image is destroyed by dividing it into pixel patches which are then randomly shuffled. The full set of latent variables is trained to model the original data, obliging the remainder of the latent representation to model the global structure. We demonstrate that this approach successfully disentangles the latent variables for global structure from local structure by observing the generative samples of SVHN and CIFAR10. We also clustering the disentangled global structure of SVHN and found that the emerging clusters represent meaningful groups of global structures -including digit identities and the number of digits presence. Finally, we discuss the problem of evaluating the clustering accuracy when ground truth categories are not expressive enough.

Review:
This work proposes an approach for explicitly placing information in a subset of the latent variables. The approach is to construct an auxiliary generative model that takes as input the set of latent variables subtracted from the target subset, which is used to model modified data samples that do not contain the desired information. 
Experiments focus on learning global information. The auxiliary model is then given data that have their global information destroyed via random shuffling of image patches.
# Approach seems limited.
 - This approach seems very limited, as there must exist a known transformation that removes the desired information. Apart from global vs. local, can the authors provide more examples of what sort of information this approach can disentangle? (Even for global vs. local, is there a transformation that can remove local information as opposed to global information?)
 - Can this approach learn multiple factors as opposed to just two? 
 - What if the desired factors are not clearly disjoint and collectively exhaustive? (e.g. mustache vs. gender on human faces.)
# More ablations or experiments with comparable settings would be desirable.
 - What is the choice of beta in the beta-VAE training objective? Apart from 1.2, this isn't mentioned. My concern here is that beta might be affecting the result more than the proposed training algorithm. Can the proposed approach perform just as well without a modified objective? Ablation studies that show the proposed algorithm can improve upon the baseline in all settings would make this a stronger paper. (e.g. this approach with normal VAE objective, and normal VAE objective without auxiliary task for the clustering experiment.)
 - Why were 30 discrete categories used in the clustering experiment? Is this still comparable to the approaches that use 10, which would correspond to the number of classes?
# Related work.
There are some well-cited works that the authors may have missed. These are ultimately different approaches, but perhaps the authors can obtain some inspiration from these:
- Tranforming autoencoders [1] also apply a transformation to the image, but the goal is to learn the factor corresponding to the transformation, rather than the complement as in this work.
- An opposing approach for explicit information placement with a modified training procedure (where the target information is directly placed in the target subset and can handle multiple factors) is DC-IGN [2]. I believe the DC-IGN approach is more general and can handle a superset of the tasks of this approach, without requiring an auxiliary decoder. Comparing to this approach, I wonder if it would be better to provide samples that exhibit a particular factor, or samples that conceal the factor?
[1] Hinton, Geoffrey E., Alex Krizhevsky, and Sida D. Wang. "Transforming auto-encoders." International Conference on Artificial Neural Networks. Springer, Berlin, Heidelberg, 2011.
[2] Kulkarni, Tejas D., et al. "Deep convolutional inverse graphics network." Advances in neural information processing systems. 2015.
---- Update since rebuttal ----
I thank the authors for clarifying how this work fits in with related works and clarifying the hyperparameters. I maintain my concerns that the experiments are limited and do not showcase the individual benefit of using explicit information placement. More experiments based on different transformations that the authors have mentioned would make this a stronger contribution. The use of beta>1 is fine if it helps alongside the use of this approach, but it would have been better to see the effects of this approach and beta>1 (and other hyperparameters such as k in Table 1) in isolation.