Paper:
We present a generative adversarial network (GAN) that conducts manifold learning and alignment (MLA): A task to learn the multi-manifold structure underlying data and to align those manifolds without any correspondence information. Our main idea is to exploit the powerful abstraction ability of encoder architecture. Specifically, we define multiple generators to model multiple manifolds, but in a particular way that their inverse maps can be commonly represented by a single smooth encoder. Then, the abstraction ability of the encoder enforces semantic similarities between the generators and gives a plausibly aligned embedding in the latent space. In experiments with MNIST, 3D-Chair, and UT-Zap50k datasets, we demonstrate the superiority of our model in learning the manifolds by FID scores and in aligning the manifolds by disentanglement scores. Furthermore, by virtue of the abstractive modeling, we show that our model can generate data from an untrained manifold, which is unique to our model.

Review:
The paper suggests performing simultaneous manifold learning and alignment by multiple generators that share common weight matrices and a constructed inverse map that is instantiated by a single encoder. The method utilizes a special regularizer to guide the training. It has been empirically well tested on a multi-manifold learning task, manifold alignment, feature disentanglement, and style transfer.
Overall, this is an interesting idea with a motivated approach, however, I would like several points to be addressed before I could increase a score.
1. It seems that the method makes direct use of the number of classes in the datasets used. How would it fare compared to other models when the number of manifolds is not known (e.g. CelebA dataset)?
2. In MADGAN (Ghosh et al., 2017) generators share first layers which possibly makes them not independent as claimed in the paper, thus it is worth checking if MADGAN exhibits any kind of manifold alignment and could be a baseline for disentanglement with multiple generators.
3. There are hyperparameters \lambda and \mu for the regularizers in the model. It would be helpful to study their effect of different values on the training and encoding.
4. Is there a reason for DMWGAN/InfoGAN scores being omitted in Table 1?
Minor remark - there are a number of typos in the text.