Paper:
We introduce a novel autoencoder model that deviates from traditional autoencoders by using the full latent vector to independently modulate each layer in the decoder. We demonstrate how such an 'automodulator' allows for a principled approach to enforce latent space disentanglement, mixing of latent codes, and a straightforward way to utilize prior information that can be construed as a scale-specific invariance. Unlike the GAN models without encoders, autoencoder models can directly operate on new real input samples. This makes our model directly suitable for applications involving real-world inputs. As the architectural backbone, we extend recent generative autoencoder models that retain input identity and image sharpness at high resolutions better than VAEs. We show that our model achieves state-of-the-art latent space disentanglement and achieves high quality and diversity of output samples, as well as faithfulness of reconstructions.

Review:
The submission proposes an autoencoder architecture which combines two recent GAN-based architectural innovations, namely the progressive growing of the decoder architecture (as well as the encoder architecture in this case) and the use of the encoded representation to modulate the decoder via a feature-wise transformation mechanism.
I think the overall idea behind the paper is valuable, but I don’t think the submission meets the acceptance bar from a clarity point of view. I also have concerns with its characterization of the literature.
Clarity-related comments:
- “This allows the layers to work independently of each other.” This is an imprecise use of the term “independently”. How can two layers work independently if one’s input is the output of the other?
- “The reconstructed samples could be re-introduced to the encoder, repeating the process, and requiring consistency between passes.” The rest of the paragraph is built on the premise that this is a desirable property, but I’m not sure I understand why this is a desirable property in the first place.
- How to define “disentanglement” in the context of representation learning is in itself an unsettled question as far as I’m aware, but the submission uses the term without an intuitive or formal definition. What do the authors mean by “disentangled representation”? What is measured by perceptual path length (PPL), and in which ways does PPL relate to the author’s definition of “disentangled representation”?
- “[...] a new autoencoder-like model with powerful properties not found in regular autoencoders, including style transfer.” The term “style transfer” is overloaded; what do the authors mean?
- The submission defines AdaIn as a way to combine “content” and “style”, and defines the style “y” in terms of mean and variance. In the context of the AdaIn paper, this makes sense: the instance normalization shifting and scaling coefficients are heuristically defined as the channel-wise means and standard deviations of a “style” stack of feature maps. However, in the context of this submission I’m not sure this definition makes as much sense: the instance normalization shifting and scaling coefficients are the result of a linear projection of the latent representation and do not involve the channel-wise means and standard deviations of an external stack of feature maps; is this correct?
- “This setup follows the same logic as that of Karras et al. (2019), but we do not require an ad-hoc disentanglement stack.” Can the authors clarify what they mean by an “ad-hoc disentanglement stack”?
- Section 3.2 uses some notation for the encoder without introducing it first. I believe the only way to understand that \phi(x) refers to the encoder network is to look at Figure 2.
- Section 3.2 as a whole is hard to follow, in part due to the use of imprecise language (“mutually independent”, “representation of those levels disentangled in z”). At some point probability distributions are introduced (up until now the reader is operating under the assumption that the model is an autoencoder with no probabilistic interpretation), and mutual information is mentioned to justify an L2 reconstruction loss in z-space (which I would argue is an instance of mathiness that does not serve the reader’s comprehension). Can the authors explain in plain language how layer-specific losses are defined and how the complete loss is obtained?
- The submission presents model samples, but as far as I can tell the procedure for sampling is not provided. Unlike VAEs, autoencoders do not explicitly model the empirical distribution -- although reconstruction in denoising autoencoders is related to the score of the empirical distribution (Alain & Bengio, 2014). How are samples obtained from the trained model?
Literature-related comments:
- The use of normalization layers to implement feature-wise transformation mechanisms is fairly widespread nowadays, but for instance normalization specifically the work of Dumoulin et al. (2017) pre-dates that of Huang & Belongie (2017). Both are cited by Karras et al. (2019) in relation to AdaIn (which is termed “conditional instance normalization” in Dumoulin et al. (2017)).
- I disagree with the characterization of ALI/BiGAN as “hybrid models that combine the properties of VAEs and GANs”: unlike AAE and AVB, which minimize KL-divergence terms in the VAE loss adversarially, the objective for ALI/BiGAN is purely adversarial. I would also include IAE (Makhzani et al., 2018) and BigBiGAN (Donahue et al., 2019) in the list of GAN variants that incorporate an inference mechanism.
- The submission repeatedly asserts that GANs lack an inference mechanism: “Unlike GANs, autoencoder models can directly operate on input samples.”; “To work on new input images, GANs either need to be extended with a separate encoder, or inverted [...].”; “[...] GANs show good image quality, but have no built-in encoding mechanism [...]”; “[...] the problem with GANs is that they lack the encoder [...]”. This is false: see for example ALI, BiGAN, and BigBiGAN. The problem in my opinion is elsewhere: the kinds of reconstructions these models yield are not suited to the downstream applications investigated in this submission, because they oftentimes fail to preserve low-level details.
References:
- Alain, G., & Bengio, Y. (2014). What regularized auto-encoders learn from the data-generating distribution. The Journal of Machine Learning Research, 15(1), 3563-3593.
- Dumoulin, V., Shlens, J., & Kudlur, M. (2017). A learned representation for artistic style. In Proceedings of the International Conference on Learning Representations.
- Makhzani, A. (2018). Implicit autoencoders. arXiv:1805.09804.
- Donahue, J., & Simonyan, K. (2019). Large scale adversarial representation learning. arXiv:1907.02544.