Original: D EEP A UTOMODULATORS
Revised: D EEP A UTOMODULATORS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce a novel autoencoder model that deviates from traditional autoencoders by using the full latent vector to independently modulate each layer in the decoder.
Revised: We introduce a novel autoencoder model that deviates from traditional autoencoders by using the full latent vector to independently modulate each layer in the decoder.

Original: We demonstrate how such an ‘automodulator’ allows for a principled approach to enforce latent space disentanglement, mixing of latent codes, and a straightforward way to utilize prior information that can be construed as a scalespeciﬁc invariance.
Revised: We demonstrate how such an ‘automodulator’ allows for a principled approach to enforce latent space disentanglement, mixing of latent codes, and a straightforward way to utilize prior information that can be construed as a scale-speciﬁc invariance.

Original: Unlike GANs, autoencoder models can directly operate on new real input samples.
Revised: Unlike the GAN models without encoders, autoencoder models can directly operate on new real input samples.

Original: This makes our model directly suitable for applications involving real-world inputs.
Revised: This makes our model directly suitable for applications involving real-world inputs.

Original: As the architectural backbone, we extend recent generative autoencoder models that retain input identity and image sharpness at high resolutions better than VAEs.
Revised: As the architectural backbone, we extend recent generative autoencoder models that retain input identity and image sharpness at high resolutions better than VAEs.

Original: We show that our model achieves state-of-the-art latent space disentanglement and achieves high quality and diversity of output samples, as well as faithfulness of reconstructions.
Revised: We show that our model achieves state-of-the-art latent space disentanglement and achieves high quality and diversity of output samples, as well as faithfulness of reconstructions.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: This paper introduces a new generative autoencoder for learning representations of image data sets with an architecture that allows arbitrary combinations of latent representations to generate images (see Fig. 1).
Revised: This paper introduces a new generative autoencoder for learning representations of image data sets, in a way that allows arbitrary combinations of latent codes to generate images (see Fig. 1).

Original: We achieve this with an architecture that uses adaptive instance normalization (AdaIn,  Huang & Belongie, 2017), anda training method that lets the model learn a highly disentangled latent space by utilizing progressively growing autoencoders (Heljakka et al., 2019) and novel training principles that we introduce in this paper.
Revised: We achieve this with an architecture that uses adaptive instance normalization (AdaIn, Dumoulin et al., 2017b; Huang & Belongie, 2017), and training methods that let the model learn a highly disentangled latent space by utilizing progressively growing autoencoders (Heljakka et al., 2019).

Original: In a typical autoencoder, input images are encoded into latent space, and the information of the latent variables is then passed through successive layers of decoding until a reconstruction of the input image has been formed.
Revised: In a typical autoencoder, input images are encoded into latent space, and the information of the latent variables is then passed through successive layers of decoding until a reconstruction of the input image has been formed.

Original: In our model, the latent vector independently modulates the statistics of each layer of the decoder, to the same effect as style transfer (Gatys et al., 2016). This allows the layers to work independently of each other.
Revised: In our model, the latent vector independently modulates the statistics of each layer of the decoder—the output of layer n is no longer solely determined by the input from layer n − 1 .

Original: In image generation, the probability mass representing sensible images (such as human faces) lies concentrated on a low-dimensional manifold.
Revised: In image generation, the probability mass representing sensible images (such as human faces) lies concentrated on a low-dimensional manifold.

Original: Even if impressive results have been shown for image generation ( e.g. , by GANs, Goodfellow et al., 2014; Karras et al., 2019), efﬁcient reconstruction and manipulation remain open problems.
Revised: Even if impressive results have been shown for image generation ( e.g. , by GANs, Goodfellow et al., 2014; Karras et al., 2019), efﬁcient reconstruction and manipulation remain open problems.

Original: Deep generative autoencoders provide a principled approach for feature extraction and editing.
Revised: Deep generative autoencoders provide a principled approach for feature extraction, editing, and image synthesis.

Original: Thus, an encoder–decoder structure would be a natural starting point for image synthesis and manipulation, and we show that modulation of decoder layers with AdaIn allows more powerful and disentangled representations and image manipulation.
Revised: We show that modulation of decoder layers with AdaIn further improves these capabilities. It also yields representations that are more disentangled, a property here broadly deﬁned as something that allows for ﬁne control of one semantic (image) sample attribute independently of others.

Original: Previous works on AdaIn are mostly based on GAN models (Karras et al., 2019; Chen et al., 2019).
Revised: Previous works on AdaIn are mostly based on GAN models (Karras et al., 2019; Chen et al., 2019) with no encoder for new input images.

Original: To work on new input images, GANs either need to be extended with a separate encoder, or inverted with optimization-based methods which are prohibitively slow for many applications.

Original: Autoencoders are typically single-pass encoder–decoder structures, where a sample enters from one end and is reconstructed at the other.
Revised: Autoencoders are typically single-pass encoder–decoder structures, where a sample enters from one end and is reconstructed at the other.

Original: The reconstructed samples could be re-introduced to the encoder, repeating the process, and requiring consistency between the passes.
Revised: For robustness, the reconstructed samples could be reintroduced to the encoder, repeating the process, and requiring consistency between the passes.

Original: Unlike stochastic models (like VAEs, Kingma & Welling, 2014; Rezende et al., 2014), our deterministic model  better allows for measuring consistency between the 1 st and 2 nd pass at any network layer.
Revised: In comparison to stochastic models (like VAEs, Kingma & Welling, 2014; Rezende et al., 2014), our deterministic model is better suited to take advantage of this.

Original: Our model allows us to mix the latent codes of separate samples and measure the conservation of layer-speciﬁc information for each.
Revised: We can mix the latent codes of separate samples and measure the conservation of layer-speciﬁc information for each.

Original: This enforces disentanglement of layer-speciﬁc properties, because we can ensure that the latent code introduced on certain layers will affect only those layers on the 2 nd pass.
Revised: This enforces disentanglement of layer-speciﬁc properties, because we can ensure that the latent code introduced from a certain layer onwards on the 1 st pass should not affect the previous layers on the 2 nd pass.

Original: Image autoencoders with convolutional architectures are often used for ﬁnding attributes that are disentangled from each other in the latent space.
Revised: Image autoencoders with convolutional architectures are often used for ﬁnding attributes that are disentangled from each other in the latent space.

Original: Often, they have relatively poor output image quality, and prior information must be fed in either via class-conditioning or by more complex loss functions for the latent variables.
Revised: Often, they have relatively poor output image quality, and prior information must be fed in either via class-conditioning or by more complex loss functions for the latent variables.

Original: Implicit methods such as GANs show good image quality, but have
Revised: Fully implicit methods avoid this, but without a built-in encoding mechanism often cover only a relatively small fraction of the variation in the training data. For the ﬁrst time, we show a full autoencoder architecture that has a state-of-the-art disentanglement

Original: [Figure]
Revised: [Figure]

Original: no built-in encoding mechanism and often cover only a relatively small fraction of the variation in the training data. For the ﬁrst time, we show a full autoencoder architecture that has a state-ofthe-art disentanglement performance even for unsupervised training setup in several data sets, sharp image output quality, good coverage of the variation of the training data, arbitrary mixing of latent representations and a principled approach for incorporating scale-speciﬁc prior information.
Revised: performance even for unsupervised training setup in several data sets, sharp image output quality, good coverage of the variation of the training data, arbitrary mixing of latent representations and a principled approach for incorporating scale-speciﬁc prior information.

Original: Our contributions are as follows.
Revised: Our contributions are as follows.

Original: (i)
Revised: (i)

Original: We introduce the automodulator , a new autoencoder-like model with powerful properties not found in regular autoencoders, including  style transfer.
Revised: We introduce the automodulator , a new autoencoder-like model with powerful properties not found in regular autoencoders, including scale-speciﬁc style transfer (Gatys et al., 2016).

Original: In contrast to architecturally similar ‘style’-based GANs, we can now directly encode and manipulate new inputs.
Revised: In contrast to architecturally similar ‘style’-based GANs, we can now directly encode and manipulate new inputs.

Original: (ii) We present methodology that allows  training  such a model to learn rich latent representation on high resolutions, and speciﬁcally explain a novel multi-pass training principle that enables the model to learn a more disentangled representation of training data than regular autoencoders.
Revised: (ii) We present methodology that allows fully unsupervised training of such a model to learn rich latent representation in high resolutions, and speciﬁcally explain a novel multi-pass training principle that enables the model to learn a more disentangled representation of training data than regular autoencoders.

Original: (iii)
Revised: (iii)

Original: We demonstrate qualitative and quantitative performance and applications on C ELEB A-HQ and LSUN Bedrooms and Cars data sets, taking advantage of the highly disentangled model structure.
Revised: We demonstrate qualitative and quantitative performance and applications on C ELEB A-HQ and LSUN Bedrooms and Cars data sets, taking advantage of the highly disentangled model structure.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work builds upon several lines of previous work in unsupervised representation learning.
Revised: Our work builds upon several lines of previous work in unsupervised representation learning.

Original: The key conceptsrelated to our approach are variational autoencoders (VAEs, Kingma & Welling, 2014) and generative adversarial networks (GANs, Goodfellow et al., 2014) as described below.
Revised: The most relevant concepts are variational autoencoders (VAEs, Kingma & Welling, 2014) and generative adversarial networks (GANs, Goodfellow et al., 2014).

Original: VAEs learn anencoder–decoder model, where the encoder mapsthe data points to a lower dimensional latent space and the decoder maps the latent representations back to the data space.
Revised: In VAEs, an encoder maps data points to a lower dimensional latent space and a decoder maps the latent representations back to the data space.

Original: The model is learnt by minimizing the reconstruction error together with a regularization term that encourages the distribution of latents to match a predeﬁned prior.
Revised: The model is learnt by minimizing the reconstruction error together with a regularization term that encourages the distribution of latents to match a predeﬁned prior.

Original: Latent representations often provide useful features for applications ( e.g. , image analysis and manipulation) and decoding random samples from the prior allows  data generation.
Revised: Latent representations often provide useful features for applications ( e.g. , image analysis and manipulation) and decoding random samples from the prior allows novel data generation.

Original: However, in case of images the generated samples are often blurry and not fully photorealistic and the reconstructions are not perfect either.
Revised: However, in case of images, the generated samples are often blurry and not fully photorealistic, with imperfect reconstructions.

Original: Current state-of-the-art in generative image modeling is represented by  modelsbased on GANs (Brock et al., 2019; Karras et al., 2019). These recent models generate images with better quality than VAE based models.
Revised: Current state-of-the-art in generative image modeling is represented by GAN models (Brock et al., 2019; Karras et al., 2019) which achieve higher image quality than VAE based models.

Original: Nevertheless, the problem with GANsis that that they lack the encoder and hence do not provide a direct way of obtaining the latent representation for a given image.
Revised: Nevertheless, these GANs lack an encoder for obtaining the latent representation for a given image, limiting their usefulness.

Original: This limits the usefulness of the models. In some cases, a given image can be  mapped to the latent spacein a semantically meaningful manner via generator inversion but thisis a costly iterative process  and the result may depend on initialization (Abdal et al., 2019; Creswell & Bharath, 2019).
Revised: In some cases, a given image can be semantically mapped to the latent space via generator inversion but this iterative process is prohibitively slow for many applications, and the result may depend on initialization (Abdal et al., 2019; Creswell & Bharath, 2019).

Original: Thus, to get the best of both worlds, there have been many recent efforts to build hybrid models that combine the properties of VAEs and GANs. Adversarial autoencoders by  Makhzani et al. 
Revised: Bidirectional mapping has been targeted by VAE-GAN hybrids such as Makhzani et al. Mescheder et al.

Original: (2016) are among the ﬁrst efforts and other examples are by Donahue et al.
Revised: (2017); Makhzani (2018), and adversarial models such as Donahue et al.

Original: ; Dumoulin et al.
Revised: Dumoulin et al.

Original: Mescheder et al.
Revised: (2017a); Donahue & Simonyan (2019).

Original: All the aforementioned methods learnthe mappings between the data space and latent space using three deep networks ( i.e. , generator, encoder, and discriminator).
Revised: These models learn mappings between the data space and latent space using combinations of encoders, generators and discriminators.

Original: IntroVAE (Huang et al., 2018) and AGE (Ulyanov et al., 2018) are more compact autoencoders, which both contain only two deep networks, encoder and decoder, learnt using a combination of reconstruction loss and adversarial loss, which encourages the distributions of real and generated images to match with the prior in latent space.
Revised: IntroVAE (Huang et al., 2018) and AGE (Ulyanov et al., 2018) are more compact autoencoders, which both contain only two deep networks, encoder and decoder, learnt using a combination of reconstruction loss and adversarial loss, which encourages the distributions of real and generated images to match with the prior in latent space.

Original: Later, a progressively grown version of AGE,called P IONEER , was proposed by Heljakka et al.
Revised: A progressively grown version of AGE, P IONEER , was proposed by Heljakka et al.

Original: There is also recent work on vector quantized variational autoencoders (VQ-VAE, Razavi et al., 2019) with a discrete latent space.
Revised: There is also recent work on vector quantized variational autoencoders (VQ-VAE, Razavi et al., 2019) with a discrete latent space.

Original: However, while suitable for image compression, the discrete latent representation cannot e.g. be interpolated and hence may not be optimal for semantic image manipulation and learning a disentangled latent space.
Revised: However, while suitable for image compression, the discrete latent representation cannot e.g. be interpolated and hence may not be optimal for semantic image manipulation and learning a disentangled latent space.

Original: The architecture of our generator and AdaIn utilization was inspired by the recent StyleGAN (Karras et al., 2019) but our model does not contain a discriminator.
Revised: The architecture of our generator and AdaIn utilization was inspired by the recent StyleGAN (Karras et al., 2019), but has no discriminator (nor the ‘mapping network’ f ).

Original: Instead, the model simply contains the generator ( i.e. , decoder) and an encoder,which are jointly trained applying a similar progressive growing strategy as Heljakka et al.
Revised: Instead, our model simply contains the generator ( i.e. , decoder) and an encoder, jointly trained applying a similar progressive growing strategy as Heljakka et al.

Original: (2018), but modifying the loss formulation.
Revised: (2018), but with a modiﬁed loss formulation.

Original: Besides the reconstruction loss and adversarial lossborrowed from Ulyanov et al.
Revised: Besides the reconstruction loss and adversarial loss from Ulyanov et al.

Original: (2018) and Heljakka et al.
Revised: (2018) and Heljakka et al.

Original: (2018), we can also recirculate style-mixed reconstructions as ‘second-pass’ training samples in order to better encourage the independence and disentanglement of emerging styles and conservation of layer-speciﬁc information.
Revised: , we can also recirculate style-mixed reconstructions as ‘second-pass’ training samples in order to better encourage the independence and disentanglement of emerging styles and conservation of layerspeciﬁc information.

Original: The recirculation idea is biologically motivated and conceptually related to many works, such as that by Hinton & McClelland (1988).
Revised: The recirculation idea is biologically motivated and conceptually related to many works, such as that by Hinton & McClelland (1988).

Original: Utilizing the outputs of the model as inputs for the next iteration is related to, e.g. , Zamir et al.
Revised: Utilizing the outputs of the model as inputs for the next iteration is related to, e.g. , Zamir et al.

Original: (2017), where feedback is shown to beneﬁt image classiﬁcation, and in RNN-based methods (Rezende et al., 2016; Gregor et al., 2015; 2016).
Revised: (2017), where feedback is shown to beneﬁt image classiﬁcation, and in RNN-based methods (Rezende et al., 2016; Gregor et al., 2015; 2016).

Original:  M ETHODS
Revised:  M ETHODS

Original: We cover how our approach allows for efﬁcient training via latent space reconstruction errors, learning disentanglement via mixed latent codes, cyclic reconstruction errors at any location of the network, and how to utilize known invariances by the architecture.
Revised: We now describe the model architecture, starting from regular P IONEER and building up to the automodulator (Sec. 3.1). We then describe the unsupervised training via our extensions to the regular P IONEER loss function, to support ﬂexible mixing of latents (Sec. 3.2). Finally, we introduce an additional weakly supervised training method to utilize known invariances (Sec. 3.3).

Original: 3. A UTOMODULATOR A RCHITECTURE
Revised: 3. A UTOMODULATOR A RCHITECTURE

Original: Our interest is in unsupervised training of an autoencoder wherein the inputs x are images fed through an encoder φ to form a low-dimensional latent space representation
Revised: Our interest is in unsupervised training of an autoencoder wherein the inputs x are images fed through an encoder φ to form a low-dimensional latent space representation

Original: z (we use z ∈ Rthroughout this paper).
Revised: z (we use z ∈ R 512 , normalized to unity).

Original: This representation can then be decoded back into an image ˆ x through a decoder θ .
Revised: This representation can then be decoded back into an image ˆ x through a decoder θ .

Original: As the encoding and decoding are deterministic in the AGE-based architecture, we retain, in principle, the full information about the image, at every stage of the processing.

Original: This means that when an image has been encoded to a latent vector z , decoded back to image space as ˆ x , and re-encoded as latent vector z (cid:48) , it is possible and desirable to require that z is as close to z (cid:48) .

Original: This allows for using the reconstruction error in latent space as part of the loss function in any AGE-based architecture.

Original: The overall architecture follows the progressively growing  Balanced P IONEERform (Heljakka et al., 2018; 2019).
Revised: For stable training, we adopt the progressively growing architecture of Balanced P IONEER (Heljakka et al., 2018; 2019).

Original: As visualized in Fig. 2, the convolutional layers of the  encoder andthe decoder are faded in gradually during the training, in tandem with the resolutions of training images and generated images.
Revised: The convolutional layers of the symmetric encoder and decoder are faded in gradually during the training, in tandem with the resolutions of training images and generated images (Fig. 2).

Original: This makes for a stable training scheme.

Original: Adaptive Instance Normalization (AdaIn)
Revised: Adaptive Instance Normalization (AdaIn)

Original: We take advantage of AdaIn (Huang & Belongie, 2017) layers inside the model.

Original: AdaIn is a simple method to modulate the channel-wise sample mean µ and variance σ 2 of layers separately with ‘content’ ξ and ‘style’ y (comprising the marginal mean and variance y (cid:44) ( µ y , σ 2 y ) ), as

Original: [Equation]

Original: A traditional decoder architecture starts from a small-resolution image and expand it layer by layer until the full image is formed, feeding the full information of the latent code through the decoder layer by layer.
Revised: A traditional decoder architecture would start from a small-resolution image and expand it layer by layer until the full image is formed, feeding the full information of the latent code via the decoder layers.

Original: In contrast, our decoder function θ  ( ξ  , z )  separately takes a ‘canvas’ variable ξ that denotes the content inputpassed through from the previous layers (see Figs. 2 and 3a).
Revised: In contrast, our decoder is composed of layer-wise functions θ i ( ξ i − 1 , z ) that separately take a ‘canvas’ variable ξ i − 1 denoting the content input from the preceding decoder layer (see Figs. 2 and 3a), and the actual (shared) latent code z .

Original: Layer # starts froma constant input ξ (0) ∈ R 4 × 4 .
Revised: First, each deconvolutional layer #i computes its feature map output χ i from ξ i − 1 as in traditional decoders. Second, we take the AdaIn normalization step that uses z to modulate the channel-wise mean µ and variance σ 2 of the output. To do this, we need a map

Original: Within the decoder, each residual block also comes with its own fully connected afﬁne mapping layer (incorporated in θ ( ξ , z ) )  for Eq.
Revised: g i : z (cid:55)→ ( µ i , σ 2 i ) , arriving at: [Equation] We implement g i for layer # i as a fully connected linear layer, with output size 2 C i for C i channels.

Original: (1), where y is linearly mapped from the input latent vector z .
Revised: Layer #1 starts from a constant input ξ (0) ∈ R 4 × 4 .

Original: We inject input information to the decoder separately for each deconvolution layer, mediated by the afﬁne mapping layers, in order to modulate the statistics of  the deconvolution layer, hence the name automodulator .
Revised: Here, we focus on pyramidal decoders with monotonically increasing resolution and decreasing number of channels, but any deep decoder would be applicable. The AdaIn-based architecture allows the output of layer # i to be not solely determined by the input from layer # i − 1 , enabling ﬁner control over the image information, image mixing schemes (Sec. 3.2), and utilization of scale-speciﬁc invariances in training data (Sec. 3.3). 3. S TRONG CONSERVATION OF C YCLIC I NFORMATION : L AYER - SPECIFIC L OSS M ETRICS In training of regular AGE and P IONEER models, the encoder φ and the decoder θ are trained at separate steps, optimizing either the loss L φ or L θ , correspondingly:

Revised: [Equation]

Original: z ξ (0) x ˆ x 4 ×4 × 4 E n c od e r , φ
Revised: z ξ (0) x ˆ x 4 ×4 × 4 E n c od e r , φ

Original: D ec od e r , θ
Revised: D ec od e r , θ

Original: Encoding z ξ (0) x
Revised: Encoding z ξ (0) x

Original: ˆ x 16 ×16 × 16 E n c od e r , φ
Revised: ˆ x 16 ×16 × 16 E n c od e r , φ

Original: D ec od e r , θ
Revised: D ec od e r , θ

Original: Encoding z ξ (0) x ˆ x 64 ×64 × 64 E n c od e r , φ
Revised: Encoding z ξ (0) x ˆ x 64 ×64 × 64 E n c od e r , φ

Original: D ec od e r , θ
Revised: D ec od e r , θ

Original: Encoding z ξ (0) x ˆ x 256 ×256 × 256 E n c od e r , φ
Revised: Encoding z ξ (0) x ˆ x 256 ×256 × 256 E n c od e r , φ

Original: D ec od e r , θ
Revised: D ec od e r , θ

Original: Encoding
Revised: Encoding

Original: Training progresses This setup follows the same logic as that of Karras et al. , but we do not require an ad hoc disentanglement stack. Here, we focus on pyramidal decoders with monotonically increasing resolution and decreasing number of channels, but any deep decoder would be applicable.
Revised: Training progresses where x is sampled from the training set, ˆ x ∼ q θ ( x | z ) , z ∼ N( 0 , I ) , d X is L1 or L2 distance, and d cos is cosine distance.

Original: The AdaIn-based architecture allows the decoder layers to work independently of each other, enabling ﬁner control over the image information—as covered in  the next section—as well as various image mixing schemes and utilization of scale-speciﬁc invariances in training data.
Revised: Since the model allows sampling from latent space (similarly to VAEs and GANs), the KL divergence terms during training can be calculated from empirical distributions, despite the model itself being fully deterministic. Yet we retain, in principle, the full information about the image, at every stage of the processing.

Original: Suppose we feed the information to certain decoder layers from a latent encoding z A , and  to  the rest of the layers from another encoding z B (cid:54) =
Revised: For an image encoded into a latent vector z , decoded back to image space as ˆ x , and re-encoded as latent vector z (cid:48) , it is possible and desirable to require that z is as close to z (cid:48) as possible, yielding the latent reconstruction error d cos ( z , φ ( θ ( z ))) .

Original: z A .
Revised: Layer-speciﬁc Loss We now show how to encourage the latent space to become hierarchically disentangled with respect to the levels of image detail, allowing one to separately retrieve ‘coarse’ and ‘ﬁne’ aspects of a latent code.

Original: Then, when we re-encode the result and decode again, we enforce intermediate decoder layer outputs to be consistent between the passes.
Revised: This allows for e.g. conditional sampling by ﬁxing the latent code for speciﬁc layers of the decoder, or mixing the scale-speciﬁc features of two or more input images—impossible feats for a traditional autoencoder with mutually entangled decoder layers.

Original: Thus encouraging the latent space to become hierarchically disentangled with respect to the levels of image detail.
Revised: The latent reconstruction error as such can be trivially generalized to any layer of θ .

Original: One of the many beneﬁts is that we can now carry out conditional sampling by ﬁxing the latent code for speciﬁc layers of  the decoder, or mix the scale-speciﬁc features of two or more input images—impossible feats for a traditional autoencoder where the layers of the decoder are too mutually entangled to allow for this.
Revised: We simply pick the layer of measurement, and from there, pass the sample once through a full encoder-decoder cycle. But now, in the automodulator, latent codes can be introduced on a per-layer basis, enabling more powerful reconstruction measurements. (Without loss of generality, here we only consider mixtures of 2 codes.)

Original: This model allows presenting a decoder (Fig. 3b) with N layers split after the j th one as a composition of θ j +1: N ( θ 1: j ( ξ (0) , z A ) , z B ) , in which we can choose z A (cid:54) =
Revised: We can present a decoder (Fig. 3b) with N layers, split after the j th one, as a composition of θ j +1: N ( θ 1: j ( ξ (0) , z A ) , z B ) . Crucially, we can choose z A (cid:54) =

Original: z B (similarly to Karras et al., 2019). Speciﬁcally, consider z A = φ ( x A ) and z B = φ ( x B ) for (image) inputs x A (cid:54) =
Revised: z B (extending the method of Karras et al., 2019), such as z A = φ ( x A ) and z B = φ ( x B ) for (image) inputs x A (cid:54) =

Original: x B .
Revised: x B .

Original: Because the earlier layers  operate on image content at lower resolutions, we can produce images that mix the ‘coarse’ features of z A with ‘ﬁne’ features of z B .
Revised: Because the earlier layers # 1: j operate on image content at lower resolutions, the output fusion image mixes the ‘coarse’ features of z A with ‘ﬁne’ features of z B .

Original: Here we show a 2-way split, but without loss of generality, the split can be made for up to N parts.

Original: On the other hand, in AGE-based models, we can decode any latent vector z back to image space as x , re-encode it as ˆ z , and minimize the latent reconstruction error between z and ˆ z .

Original: Whether the generated images originate from single-source or multi-source (mixed) latents, we can feed them to the encoder for ‘discriminative’ purposes (via

Original: KL divergences) in the same way.

Original: Now, z holds feature information at different levels of detail, some of which are mutually independent.
Revised: Now, z holds feature information at different levels of detail, some of which are mutually independent.

Original: Hence, when re-encoding an image, we should keep the representation of those levels disentangled in z , even if they come from separate source images.
Revised: Hence, when re-encoding an image, we should keep the representation of those levels disentangled in z , even if they originate from separate source images.

Revised: Hence, when we re-encode the fusion image into ˆ z AB , and decode once more, the output of θ 1: j ( ξ (0) , ˆ z AB ) should be unaffected by z B .

Revised: This leads to the following conjecture.

Original: Assume that the described network reconstructs input samples perfectly, i.e. x = θ ( φ ( x )) .
Revised: Assume that the described network reconstructs input samples perfectly, i.e. x = θ ( φ ( x )) .

Original: Consider two latents z A and z B (either z i ∼ N( 0 , I ) or z i ∼ q φ ( z | x i ) ), x
Revised: Any z A and z B can be mixed, decoded and re-encoded as

Original: [Equation]
Revised: [Equation]

Original: = ξ ( N ) and ˆ z AB ∼ q φ ( z | x ) q θ j +1: N ( x | ξ ( j ) , z B ) q θ 1: j ( ξ ( j ) | ξ (0) , z A ) . Now, between ξ ( j ) of the ﬁrst and ξ ( j ) of the second pass (see Fig. 3c), the mutual information is I [ q θ 1: j ( ξ ( j ) | ξ (0) , ˆ z AB ); q θ 1: j ( ξ ( j ) | ξ (0) , z A )] .
Revised: Now, between ξ ( j ) of the ﬁrst and ξ ( j ) of the second pass (see Fig. 3c), the mutual information is [Equation]

Original: Then, sampling from z Aand ˆ z AB ,we have L j = d ( θ 1: j ( ξ (0) , ˆ z AB ) , θ 1: j( ξ (0) , z A )) for some distance function dwhen splitting after layer # j (here,d is the L2 norm).We can minimize it by gradient descent, consequently maximizing mutual information separately for each j .
Revised: With z A , z B ∼ N( 0 , I ) , for each j , we can now maximize (4) by minimizing [Equation] for some distance function d (here, L2 norm).

Original: In other words, the fusion image can be encoded into a new latent vector  in such a way that, at each layer, the decoder will treat the new code similarly to  the original two separate latent codes  (see Fig. 3b).
Revised: In other words, the fusion image can be encoded into a new latent vector ˆ z AB in such a way that, at each layer, the decoder will treat the new code similarly to whichever of the original two separate latent codes was originally used there (see Fig. 3b).

Original: For a perfect network, L j can be viewed as  layer entanglement error.
Revised: For a perfect network, L j can be viewed as a layer entanglement error.

Original: Randomizing j during the training, we can measure L j for any layers of the decoder.
Revised: Randomizing j during the training, we can measure L j for any layers of the decoder.

Original: z ξ (0) xˆ x
Revised: z [Figure] (0) x

Original: [Figure]
Revised: [Figure]

Original: 16 × [Figure] 32 ×64 × [Figure]
Revised: x

Original: [Figure]

Original: 4 ×

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 16 × [Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: E n c od e r , φ
Revised: E n c od e r , φ

Original: D ec od e r , θ
Revised: D ec od e r , θ

Original: Latentencoding
Revised: Latentencoding

Original: Canvasvariable (a
Revised: Canvasvariable (

Original: [Figure]
Revised: [Figure]

Original: Architecture z
Revised: [Figure] z

Original: [Figure]
Revised: [Figure]

Original:  ξ (0)
Revised: z AB ξ (

Original: [Figure]
Revised: [Figure]

Original: ˆ
Revised: ˆ

Original: [Figure]
Revised: [Figure]

Original: AB
Revised: AB

Original: [Figure]
Revised: [Figure]

Original: φ
Revised: φ

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: z
Revised: ξ (

Original: [Figure]
Revised: [Figure]

Original: z
Revised: A ∼ ξ (

Original: [Figure]
Revised: [Figure]

Original:  z AB ξ (0) x
Revised: ) AB z A z B z AB ξ (0) x

Original: [Figure]

Original:  ˆ x AB
Revised: A ˆ x AB

Original: [Figure][Figure]
Revised: [Figure]

Original: E n

Original: [Figure]
Revised: [Figure]

Original: , φ
Revised: 16 ×32 ×64 ×

Original: D ec od e r , θ ξ (2) A ∼ ξ (2) AB z A z B z AB ξ (0) x
Revised: [Figure]

Original: A ˆ x AB 4 ×8 ×16 ×32 ×64 ×128 ×256 ×4 ×8 ×16 ×32 ×64 ×
Revised: [Figure] [Figure] [Figure] 16 ×

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: E n c od e r , φ
Revised: n c od e r , φ

Original: D ec od e r , θ
Revised: D ec od e r , θ

Original: [Figure]
Revised: [Figure]

Original: Fully Unsupervised Automodulator Trainingand Loss For image reconstructionloss, we utilize a progressively-growing variation of the lossfunction d ρ of Barron (2019) that generalizes various norms and exposes robustness as an explicit continuous parameter vector α .
Revised: Final Automodulator Training Loss Our results improved by using a robust image reconstruction loss d ρ of Barron (2019) instead of L1/L2. d ρ generalizes various norms and exposes robustness as an explicit continuous parameter vector α .

Original: The complete loss functions are(see Heljakka et al., 2019, for discussion)
Revised: The complete loss functions are

Original: [Equation]
Revised: [Equation]

Original:  where ˆ x 1: 34 M ∼ q θ ( x | z ) with z ∼ N( 0 , I ) , and ˆ x 34 M : M ∼ q θ ( x | ˆ z AB ) , with a set 3:4 ratio of regular and mixed samples of batch size M  , margin M gap = 0 .
Revised: 6) where ˆ x 1: 34 M ∼ q θ ( x | z ) with z ∼ N( 0 , I ) , and ˆ x 34 M : M ∼ q θ ( x | ˆ z AB ) , with a set 3:4 ratio of regular and mixed samples of batch size M and j ∼ U { 1 , N } . Margin M gap = 0 .

Original: 5  and j ∼ U { 1 , N } . 
Revised: 5 in low resolutions, then 0 . 2 for 128 2 and 0 . 4 for 256 2 (see Heljakka et al., 2019).

Original: In our variation of d ρ ,  we ﬁrst learn the α in the lower resolutions ( e.g. , 4 × 4 ).
Revised: To avoid discontinuities in α , we utilize a progressively-growing variation of d ρ , where we ﬁrst learn the α in the lower resolutions ( e.g. , 4 × 4 ).

Original: Each α i corresponds to one pixel.
Revised: Each α i corresponds to one pixel.

Original: Then, when switching to the higher resolution stage, we take take each parameter α i that corresponds to pixels p x,y in the lower resolution, to initialize the α 1 × 4 j that, in the higher resolution, corresponds to p x,y , p x +1 ,y , p x,y +1 and p x +1 ,y +1 , respectively.
Revised: Then, when switching to the higher resolution stage, we take take each parameter α i that corresponds to pixels p x,y in the lower resolution, to initialize the α 1 × 4 j that, in the higher resolution, corresponds to p x,y , p x +1 ,y , p x,y +1 and p x +1 ,y +1 , respectively.

Original: 3. E NFORCING K NOWN I NVARIANCES AT S PECIFIC L AYERS
Revised: 3. E NFORCING K NOWN I NVARIANCES AT S PECIFIC L AYERS

Original: The architecture and the cyclic training method also allows for a novel principled approach for leveraging known scale-speciﬁc invariances in training data.
Revised: The architecture and the cyclic training method also allow for a novel principled approach to leverage known scale-speciﬁc invariances in training data.

Original: For a known invariance F , assume that we know that for two inputs x 1 and x 2 , and outputs F ( x 1 ) = F ( x 2 ) , so that we have a perfect bijective encoder φ such that we can, with slight abuse of notation, represent the relationship as F ( z 1 ) =
Revised: Assume that images x 1 and x 2 share certain identical characteristics at some scales, but differ on others, with this information further encoded into z 1 and z 2 , correspondingly.

Original: F ( z 2 ) . Furthermore, assume that we know all information about F to be contained on the level of image detail that is represented in decoder layers # j : k . Therefore, by extending the notation, F ( ξ ( j : k ))
Revised: In the automodulator, we could expect to try to represent this information in decoder layers # j : k . Following the reasoning of the previous section, we then must have θ j : k ( ξ j − 1 , z 1 )

Original: =
Revised: =

Original: F ( ξ ( j : k )) .
Revised: θ j : k ( ξ ( j − 1) , z 2 ) .

Original: Finally, assume that  the rest of the information in the images can be represented on layers #1:( j − 1) and #( k + 1): N .
Revised: Let us assume that it is possible to represent the rest of the information in the images of that dataset in layers # 1:( j − 1) and # ( k + 1): N .

Original: This situation occurs, e.g. , when two images are known to differ only interms of their high-frequency properties,which can be represented fully  in the ‘ﬁne’ layers.
Revised: This situation occurs, e.g. , when two images are known to differ only in high-frequency properties, fully representable in the ‘ﬁne’ layers.

Original: Utilizing the independence of the layers, we can now require that
Revised: Utilizing the independence of the layers, our goal is to have

Original: [Equation]
Revised: [Equation]

Original: and vice versa by swapping z 1 and z 2 . We turn these into  optimization targets
Revised: which turns into the optimization target (for some distance function d )

Original: [Equation]
Revised: [Equation]

Original: In other words, consider a set of images such that their differences occur only on certain scales, and are identical at other scales.
Revised: By construction of φ and θ , we immediately see this is equivalent to directly minimizing

Original: Now, we can reserve certain layers to only capture the identical parts.
Revised: [Equation]

Original: Hence, on those layers, we can use their latent codes interchangeably, without affecting the result of the decoding operation. Then, the result of the decoding is only driven by  the latent code that modulates the rest ofthe layers.
Revised: whose complement term L (cid:48) inv you can construct by swapping z 1 with z 2 and x 1 with x 2 . For each known invariant pair x 1 and x 2 of the minibatch, you can now add the terms L inv + L (cid:48) inv to L φ of

Original: Consequently, we push the invariant information to layers # j : k ,
Revised: Eq. Note that in the case of z 1 = thereof

Original: [Figure]
Revised: [Figure]

Original: C o a r s e f ea t u r e s fr o m s ou r ce B
Revised: C o a r s e f ea t u r e s fr o m s ou r ce B

Original: M e d i u m f ea t u r e s fr o m s ou r ce B
Revised: M e d i u m f ea t u r e s fr o m s ou r ce B

Original: F i n e fr o m B
Revised: F i n e fr o m B

Original: [Figure]
Revised: [Figure]

Original:  and the other information away from those layers. Of course, this reduces the number of layers available for the rest of the image information, so that we may need to add extra layers to retain the overall decoder capacity.
Revised: As we push the invariant information to layers # j : k , and the other information away from those layers, this reduces the number of layers available for the rest of the image information. Thus we may need to add extra layers to retain the overall decoder capacity.

Original: Note that in a pyramidal deconvolutional stack where the resolution increases monotonically, the layers of F can only span  one or two consecutive levels of detail, since otherwise there would be ‘middle resolutions’ that the rest of the architecture could not capture. Finally, note that setting z 1 = z 2 with z 1 ∼ q θ ( z | x ∼ X train ) reduces Eqs. (4–5) to the regular sample construction loss, revealing our formulation as a generalization thereof.
Revised: Note that in a pyramidal deconvolutional stack, the layers # j : k can safely span only one or two consecutive levels of detail.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We included both quantitative and qualitative experiments for real-world data sets. First, we assess generation and encoding, whereafter we turn to modulation/style-mixing capabilities.
Revised: [Table]

Original: Metrics and Data Sets We run our experiments on images using C ELEB A-HQ (Karras et al., 2018), FFHQ (Karras et al., 2019), and LSUN Bedrooms and Cars (Yu et al., 2015).
Revised: We run our experiments on images using C ELEB AHQ (Karras et al., 2018), FFHQ (Karras et al., 2019), and LSUN Bedrooms and Cars (Yu et al., 2015).

Original: To quantify the image quality and diversity of random draws from the model at 256 × 256 resolution, we use the Fr´echet inception distance (FID, Heusel et al., 2017), which is comparable across models when sample size is ﬁxed (Binkowski et al., 2018).
Revised: To quantify the image quality and diversity of random draws from the model at 256 × 256 resolution, we use the Fr´echet inception distance (FID, Heusel et al., 2017), which is comparable across models when sample size is ﬁxed (Binkowski et al., 2018).

Original: However, FID is known to remain constant under changing precision-recall characteristics, and hence should be treated with caution (Kynk¨a¨anniemi et al., 2019).
Revised: However, FID is known to remain constant under changing precision-recall characteristics (Kynk¨a¨anniemi et al., 2019).

Original: We use LPIPS (Zhang et al., 2018) as the similarity metric, which has better correspondence to human evaluation than traditional L2 metrics.
Revised: We use LPIPS (Zhang et al., 2018) as the similarity metric, which has better correspondence to human evaluation than traditional L2 metrics.

Original: The degree ofdisentanglement of the latent space  is often considered the most important property of a latent variable model.
Revised: The degree of latent space disentanglement is often considered the most important property of a latent variable model.

Original: We measure this in terms of Perceptual Path Length (PPL, Karras et al., 2019).
Revised: Qualitatively, it is the necessary condition for, e.g. , style mixing capabilities. Quantitatively, Karras et al.

Original: Baseline Methods As baselines, we compare separately against autoencoder and non-autoencoder models.
Revised: (2019) noted that, for a constant-length step in the latent space, a more entangled model will produce a larger overall perceptual change than a less entangled model. The extent of this change, with LPIPS as the perceptual space metric, is the basis of measuring disentanglement as Perceptual Path Length (PPL).

Original:  For the former, we compare to Balanced P IONEER by Heljakka et al.
Revised: Baseline Methods For autoencoders, we compare to Balanced P IONEER by Heljakka et al.

Original: (2019), a vanilla VAE, and a more recent Wasserstein Autoencoder (WAE, Tolstikhin et al., 2017).
Revised: (2019), a vanilla VAE, and a more recent Wasserstein Autoencoder (WAE, Tolstikhin et al., 2017).

Original: For VAE and WAE, we trained both the automodular and traditional architecture, using 128 × 128 to make the task easier.

Original: [Table]

Original: For the non-autoencoders [Table] we compare to GLOW (Kingma & Dhariwal, 2018) and two recent GAN models: StyleGAN and Progressively Growing GAN (PGGAN, Karras et al., 2018).
Revised: For nonautoencoders, we compare to GLOW (Kingma & Dhariwal, 2018) and two recent GAN models: StyleGAN and Progressively Growing GAN (PGGAN, Karras et al., 2018).

Original: Models that only generate samples will produce higher-quality samples than encoder–decoder models

Original: [Table]
Revised: [Table]

Original:  but lack direct reconstruction capabilities.
Revised: Ablation We illustrate the contribution of the style-mixing loss L j and the robust loss d ρ on 64 × C ELEB A-HQ (Table 1). L j increases the FID of regular random samples, but reduces it when two random latents are mixed. d ρ then reduces some of the FID of regular samples.

Original: 4. E NCODING , D ECODING , AND R ANDOM S AMPLING
Revised: 4. E NCODING , D ECODING , AND R ANDOM S AMPLING

Original: In Table 1a
Revised: In Table 2a

Original: [Table]
Revised: [Table]

Original: we compare encoder–decoder performance for the autoencoder models in C ELEB A-HQ on 128 × 128 , with our proposed architecture (‘AdaIn’) and the corresponding regular architecture (‘classic’).
Revised: we compare encoder–decoder performance of the autoencoders on 128 × 128 C ELEB AHQ, with our proposed architecture (‘AdaIn’) and the corresponding regular architecture (‘classic’).

Original: We measure LPIPS, Fr ´ echet Inception Distance (FID), and perceptual path length (PPL).
Revised: We measure LPIPS, FID and PPL.

Original: Our method has the best PPL result, while Balanced P IONEER has the bestFID. FID is based on a 50k batch of generated samples compared to training samples.
Revised: Our method has the best PPL, while Balanced P IONEER has the best FID (50k batch of generated samples compared to training samples).

Original: PPL (with ε = 10 − 4 ) was calculated with 100k samples, cropped to 64 × 64 or 128 × 128 at faces.

Original: Table 1b shows comparison of random sampling (examples in Figs.
Revised: Table 2b shows comparison of random sampling (examples in Figs.

Original: 4b–4c) performance via FID and latent space structure via PPL forvarious data sets of 256 × 256 images on C ELEB A-HQ, FFHQ and 
Revised: 4b–4c) performance via FID and latent space smoothness via PPL for 256 × 256 images on C ELEB A-HQ, FFHQ and LSUN Cars and

Original: LSUN Cars and Bedrooms.
Revised: Bedrooms.

Original: The performance of the automodulator is comparable to the Balanced P IONEER on most data sets, but the FID is worse in general.
Revised: The performance of the automodulator is comparable to the Balanced P IONEER on most data sets, but the FID is worse in general.

Original: The GANs have clearly best FID results on all data sets (NB: a hyper-parameter search with various schemes was used in Karras et al., 2019, to achieve their PPL for FFHQ).
Revised: Models that only generate random samples have clearly best FID results on all data sets (NB: a hyper-parameter search with various schemes was used in Karras et al., 2019, to achieve their PPL for FFHQ).

Original: We train on the actual 60k training set of FFHQ only (unlike StyleGAN that trained on all 70k images).
Revised: We train on the actual 60k training set of FFHQ only (StyleGAN trained on all 70k images).

Original: We also evaluate the 4-way image interpolation capabilities in unseen FFHQ test images (Fig. 12 in the appendix) and observe smooth transitions.
Revised: We also evaluate the 4-way image interpolation capabilities in unseen FFHQ test images (Fig. 12 in the appendix) and observe smooth transitions.

Original: We emphasize that in GANs, such interpolations are often made between the codes of generated samples.
Revised: We emphasize that in GANs, such interpolations are often made between the codes of generated samples.

Original: As such, they cannot tell much about the recall characteristics of those models.
Revised: As such, they cannot tell much about the recall characteristics of those models.

Original: The qualitative style-mixing capability and low PPL may indicate a high degree of latent space disentanglement, although the more advanced network architecture comes at the cost of weaker FID.

Original: 4. S TYLE M IXING
Revised: 4. S TYLE M IXING

Original: We demonstrate the ‘style-mixing’ capabilities of our model.
Revised: We demonstrate the style-mixing capabilities of our model.

Original: For comparison with prior work, we use the source images from the StyleGAN paper (Karras et al., 2019).
Revised: For comparison with prior work, we use the source images from the StyleGAN paper (Karras et al., 2019).

Original: In Fig.
Revised: In Fig.

Original: 4a, we mix speciﬁc input faces so that the ‘coarse’ (latent resolutions 4 × 4 – 8 × 8 ), ‘intermediate’ ( 16 × 16 – 32 × 32 ) or ‘ﬁne’ ( 64 × 64 – 256 × 256 ) layers of  decoder use one input, and the rest of the layers use the other.
Revised: 4a, we mix speciﬁc input faces so that the ‘coarse’ (latent resolutions 4 × 4 – 8 × 8 ), ‘intermediate’ ( 16 × 16 – 32 × 32 ) or ‘ﬁne’ ( 64 × 64 – 256 × 256 ) layers of the decoder use one input, and the rest of the layers use the other.

Original: Importantly, StyleGAN cannot take real inputs, so it can only mix between random images created by the model itself.
Revised: Importantly, StyleGAN cannot take real inputs, so it can only mix between random images created by the model itself.

Original: For new input images, one must run a separate costly optimization process to determine the most ﬁtting latent code.
Revised: For new input images, one must run a separate costly optimization process to determine the most ﬁtting latent code.

Original: For our model, those images appear as completely new test images.
Revised: For our model, those images appear as completely new test images.

Original: Additional style mixing results are included in Figs.
Revised: Additional style mixing results are included in Figs.

Original: 13–14 in the appendix.
Revised: 13–14 in the appendix.

Original: 4. E NFORCING I NVARIANCES 
Revised: 4. L EVERAGING I NVARIANCES IN A W EAKLY S UPERVISED S ETUP

Revised: We now consider the cases where there is knowledge of some scale-speciﬁc invariances within the training data.

Original: To demonstrate scale-speciﬁc invariances, we utilize the simplest image transformation possible: horizontal ﬂipping.
Revised: Here, we demonstrate a proof-of-concept experiment that uses the simplest image transformation possible: horizontal ﬂipping.

Original: For the C ELEB A-HQ face data, this provides us with pairs of images that share every other property except the azimuth rotation angle of the face.
Revised: For the C ELEB A-HQ face data, this provides us with pairs of images that share every other property except the azimuth rotation angle of the face, making the face identity invariant amongst each pair.

Original: Since the original rotation of faces in the set varies, the ﬂip-augmented data set contains faces rotated across a wide continuum of angles.
Revised: Since the original rotation of faces in the set varies, the ﬂip-augmented data set contains faces rotated across a wide continuum of angles.

Original: For further simplicity, we make an artiﬁcially strong hypothesis that the 2D projected face shape is the only relevant feature at 4 × 4 scale, and does not need to affect scales ﬁner than 8 × 8 .
Revised: For further simplicity, we make an artiﬁcially strong hypothesis that the 2D projected face shape is the only relevant feature at 4 × 4 scale, and does not need to affect scales ﬁner than 8 × 8 .

Original: This lets us enforce the L inv loss for layers 1–2.
Revised: This lets us enforce the L inv loss for layers #1–2.

Original: Since we do not want to restrict the scale

Original: [Figure] rce [Figure] 8 × 8only for the shape features, we add an extra 8 × 8resolution layer to the regular exponentially widening layer hierarchy, so that layer #3 also operates at 8 × 8 , layer
Revised: Since we do not want to restrict the scale 8 × 8 for the shape features alone, we add an extra 8 × 8 layer after layer #2 of the regular stack, so that layers #2–3 both operate at 8 × 8 , layer

Original: #4  at 16 × 16 , etc.
Revised: #4 only at 16 × 16 , etc.

Original: Now, take z 2 that corresponds to the horizontally ﬂipped counterpart of z 1 . This means that θ 3: N ( ξ (2) , z 1 ) =
Revised: Now, with z 2 that corresponds to the horizontally ﬂipped counterpart of z 1 , we have θ 3: N ( ξ (2) , z 1 ) =

Original: θ 3: N ( ξ (2) , z 2 ) .
Revised: θ 3: N ( ξ (2) , z 2 ) .

Original: With our previous architecture choice, we have j = 1 and can drop the innermost part of Eq.
Revised: Our choices amount to j = 3 , k = N , [Figure] u rce [Figure]

Revised: S

Revised: [Figure]

Revised: S

Revised: [Figure]

Revised: We generate images with the ‘non-coarse’ styles from the source images (left), and the ‘coarse’ styles from each top row image, in turn.

Revised: With ‘Enforced id. invariance’, the top row only drives the face pose while conserving identity.

Revised: In comparison, the ‘Regular’ training lets the top row also affect other characteristics, including identity.

Revised: Here, the actual data that deﬁned the invariance was too simple to allow learning ﬁne-grained control of the hair at coarse levels, making it blurry.

Revised: allowing us to drop the outermost part of Eq.

Original: Hence, our additional encoder loss terms are
Revised: Hence, our additional encoder loss terms are

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Fig.
Revised: Fig.

Original: 5 shows the results after training with the new loss (50% of the training samples ﬂipped in each minibatch).
Revised: 5 shows the results after training with the new loss (50% of the training samples ﬂipped in each minibatch).

Original: The model forcesthe two ﬁrst decoder layers  to  affect the face pose only.
Revised: With the invariance enforcement, the model forces decoder layers #1–2 to only affect the pose.

Original: When modulating the rest of the layers with the original latent, we  receive variations of the original facesuch thatthey only differ in terms of the pose, while the face identity is conserved.
Revised: We generate images by driving those layers with faces at different poses, while modulating the rest of the layers with the face whose identity we seek to conserve. We receive variations of the original face that only differ in terms of the pose, unlike in regular automodulator training.

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: In this paper, we proposed the ﬁrst generative autoencoder model with a hierarchical latent representation that supports controllable image generation and editing,including conditional image sampling by ﬁxing styles of speciﬁc layers, and style-mixing of real images.
Revised: In this paper, we proposed the ﬁrst generative autoencoder model with a hierarchical latent representation that supports controllable image generation and editing, conditional image sampling by ﬁxing styles of speciﬁc layers, and style-mixing of real images.

Original: In our model, the latent vector independently modulates each decoder layer.
Revised: In our model, the latent vector independently modulates each decoder layer.

Original: The model outperforms other generative autoencoders in terms of latent space disentanglement and matches them in faithfulness of reconstructions, with slight reduction of output sample quality.
Revised: The model outperforms other generative autoencoders in terms of latent space disentanglement and matches them in faithfulness of reconstructions, with slight reduction of output sample quality.

Original: We use the term automodulator to denote any autoencoder that uses the latent code only to modulate the statistical properties of the information that ﬂows through the layers of the decoder.
Revised: We use the term automodulator to denote any autoencoder that uses the latent code only to modulate the statistical properties of the information that ﬂows through the layers of the decoder.

Original: Such decoders could also include, e.g. , 3D or graph convolutions.
Revised: Such decoders could also include, e.g. , 3D or graph convolutions.

Original: Potential future applications include introducing completely independent ‘plugin’ layers or modules in the decoder, trained afterwards on top of the pretrained base automodulator, leveraging the mutual independence of the layers.
Revised: Potential future applications include introducing completely independent ‘plugin’ layers or modules in the decoder, trained afterwards on top of the pretrained base automodulator, leveraging the mutual independence of the layers.

Original: The afﬁne maps themselves could also be re-used across domains, potentially offering mixing of different domains.
Revised: The afﬁne maps themselves could also be re-used across domains, potentially offering mixing of different domains.

Original: Such examples highlight that the range of applications of our model is far wider than the initial ones shown here, making the family of automodulators a viable alternative to state-of-the-art autoencoders and GANs.
Revised: Such examples highlight that the range of applications of our model is far wider than the initial ones shown here, making the family of automodulators a viable alternative to state-of-the-art autoencoders and GANs.

Original: Upon acceptance for publication, our source code will be released at http://github.com/anonymized.
Revised: Upon acceptance for publication, our source code will be released at http://github.com/anonymized.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: In the appendix, we include further details on training and complement the results in the main paper with examples of random samples, reconstruction, latent space interpolations, and style mixing.
Revised: In the appendix, we include further details on training and complement the results in the main paper with examples of random samples, reconstruction, latent space interpolations, and style mixing.

Original: A. T RAINING D ETAILS
Revised: A. T RAINING D ETAILS

Original: The training method follows largely those in Heljakka et al.
Revised: The training method follows largely those in Heljakka et al.

Original: (2019), with progressively growing symmetric encoder and decoder, and decreasing the batch size when moving to higher resolutions.
Revised: (2019), with progressively growing symmetric encoder and decoder, and decreasing the batch size when moving to higher resolutions.

Original: The encoder and decoder consist of 7 blocks each containting two residual blocks with a 3 × 3 ﬁlter.
Revised: The encoder and decoder consist of 7 blocks each containting two residual blocks with a 3 × 3 ﬁlter.

Original: In the encoder, these are followed by a spectral normalization operation (Miyato et al., 2018) and binomial ﬁltering.
Revised: In the encoder, these are followed by a spectral normalization operation (Miyato et al., 2018) and binomial ﬁltering.

Original: In the decoder, by AdaIn normalization and binomial ﬁltering.
Revised: In the decoder, by AdaIn normalization and binomial ﬁltering.

Original: A leaky ReLU ( p = 0 . 2 ) is used as activation.
Revised: A leaky ReLU ( p = 0 . 2 ) is used as activation.

Original: In the encoder, each block halves the resolution of the convolution map, while in the decoder, each block doubles it.
Revised: In the encoder, each block halves the resolution of the convolution map, while in the decoder, each block doubles it.

Original: The output of the ﬁnal encoder layers is ﬂattened into a 512-dimensional latent block.
Revised: The output of the ﬁnal encoder layers is ﬂattened into a 512-dimensional latent block.

Original: As in Karras et al.
Revised: As in Karras et al.

Original: , the block is mapped by afﬁne mapping layers so that each convolutional layer C in the decoder block is preceded by its own fully connected layer that maps the latent to two vectors each of length N , when N equals the number of channels in C .
Revised: , the block is mapped by afﬁne mapping layers so that each convolutional layer C in the decoder block is preceded by its own fully connected layer that maps the latent to two vectors each of length N , when N equals the number of channels in C .

Original: Each resolution phase until 32 × 32 for all data sets use a learning rate α = 0 . 0005 and thereafter 0 .
Revised: Each resolution phase until 32 × 32 for all data sets use a learning rate α = 0 . 0005 and thereafter 0 .

Original: Optimization is done with ADAM ( β 1 = 0 , β 2 = 0 .
Revised: Optimization is done with ADAM ( β 1 = 0 , β 2 = 0 .

Original: 99 , (cid:15) =
Revised: 99 , (cid:15) =

Original: 10 − 8 ).
Revised: 10 − 8 ).

Original: After the ﬁrst two resolution steps, the KL margin is turned on and ﬁxed to 0.5.
Revised: After the ﬁrst two resolution steps, the KL margin is turned on and ﬁxed to 0.5.

Original: The length of training phases amounts to 2.4M training samples until 64 × 64 resolution phase, which lasts for 10.4M samples.
Revised: The length of training phases amounts to 2.4M training samples until 64 × 64 resolution phase, which lasts for 10.4M samples.

Original: For FFHQ and CelebAHQ, the 128 × 128 phase uses 13.0M samples while LSUN Bedrooms and Cars use 10.0M samples.
Revised: For FFHQ and CelebAHQ, the 128 × 128 phase uses 13.0M samples while LSUN Bedrooms and Cars use 10.0M samples.

Original: The ﬁnal 256 × 256 phase uses 7–10M samples for each data set.
Revised: The ﬁnal 256 × 256 phase uses 7–10M samples for each data set.

Original: The training of the ﬁnal stage was generally cut off when reasonable FID results had been obtained.
Revised: The training of the ﬁnal stage was generally cut off when reasonable FID results had been obtained.

Original: More training and learning rate optimization would likely improve results.
Revised: More training and learning rate optimization would likely improve results.

Original: With two Titan V100 GPUs for pretraining stages and four GPUs for the 256 × 256 stage, the training time for C ELEB A-HQ and FFHQ were 18 days each, and for LSUN Bedfrooms 17 days and Cars 19 days.
Revised: With two Titan V100 GPUs for pretraining stages and four GPUs for the 256 × 256 stage, the training time for C ELEB A-HQ and FFHQ were 18 days each, and for LSUN Bedfrooms 17 days and Cars 19 days.

Original: For evaluating the model after training, a moving expontential running average of generator weights was used, as in both Karras et al.
Revised: For evaluating the model after training, a moving expontential running average of generator weights was used, as in both Karras et al.

Original: (2018) and Heljakka et al.
Revised: (2018) and Heljakka et al.

Original: For all data sets, training/test set splits were used as given or deﬁned by data set authors, except for LSUN Cars, where we used 4,968,695 samples for training and 552,061 for testing.
Revised: For all data sets, training/test set splits were used as given or deﬁned by data set authors, except for LSUN Cars, where we used 4,968,695 samples for training and 552,061 for testing.

Original: Note that in regular GAN training, complete data sets are often used without train/test split, leading them to use larger training sets.
Revised: Note that in regular GAN training, complete data sets are often used without train/test split, leading them to use larger training sets.

Original: For baselines, we used pre-trained models for StyleGAN, PGGAN, P IONEER , and GLOW with default settings provided by the authors.
Revised: For baselines, we used pre-trained models for StyleGAN, PGGAN, P IONEER , and GLOW with default settings provided by the authors.

Original: We trained the VAE and WAE models manually.
Revised: We trained the VAE and WAE models manually.

Original: For all
Revised: For all

Original: VAE baselines the weight for KLD loss was 0.005.
Revised: VAE baselines the weight for KLD loss was 0.005.

Original: For all WAE baseline, we used the WAE-MMD algorithm.
Revised: For all WAE baseline, we used the WAE-MMD algorithm.

Original: The weights for the MMD loss with automodular architecture (WAE-AdaIn) was 4 and with Balanced P IONEER (WAE-classic) architecure it was 2.
Revised: The weights for the MMD loss with automodular architecture (WAE-AdaIn) was 4 and with Balanced P IONEER (WAE-classic) architecure it was 2.

Original: For VAEs, the learning rate for the encoder was 0.0001, and for the generator 0.0005.
Revised: For VAEs, the learning rate for the encoder was 0.0001, and for the generator 0.0005.

Original: For WAEs, the learning rate for both was 0.0002.
Revised: For WAEs, the learning rate for both was 0.0002.

Original: For evaluating the encoding and decoding performance, we used 10k unseen test images from the
Revised: For evaluating the encoding and decoding performance, we used 10k unseen test images from the

Original: FFHQ data set, cropped the input and reconstruction to 128 × 128 as in Karras et al.
Revised: FFHQ data set, cropped the input and reconstruction to 128 × 128 as in Karras et al.

Original: and evaluated the LPIPS distance between the inputs and reconstructions.
Revised: and evaluated the LPIPS distance between the inputs and reconstructions.

Original: We evaluated 50k random samples in all data sets and compare against the provided training set.
Revised: We evaluated 50k random samples in all data sets and compare against the provided training set.

Original: The GLOW model has not been shown to work with 256 × 256 resolution on LSUN (the authors show qualitative result only for 128 × 128 ).
Revised: The GLOW model has not been shown to work with 256 × 256 resolution on LSUN (the authors show qualitative result only for 128 × 128 ).

Original: Training of P IONEER did not converge on FFHQ, however we believe this is an issue with the default hyper-parameters not suitable for FFHQ.
Revised: Training of P IONEER did not converge on FFHQ, however we believe this is an issue with the default hyper-parameters not suitable for FFHQ.

Original: The Perceptual Path Length (PPL) was calculated with 100k samples, cropped to 128 × 128 ( ε = 10 − 4 ).
Revised: For Perceptual Path Length (PPL), we repeatedly take a random vector of length ε = 10 − 4 in the latent space, generate images at its endpoints, crop them around mid-face to 128 × 128 or 64 × 64 , and measure the LPIPS between them (Karras et al., 2019). PPL equals the scaled expectation of this value (for a sample of 100k vectors).

Original: Pre-trained models for PGGAN and GLOW were used with default settings provided by the authors.
Revised: Pre-trained models for PGGAN and GLOW were used with default settings provided by the authors.

Original: Note that we train on the actual 60k training images of FFHQ only (unlike StyleGAN that trained on all 70k images).
Revised: Note that we train on the actual 60k training images of FFHQ only (unlike StyleGAN that trained on all 70k images).

Original: A. R ANDOM S AMPLES
Revised: A. R ANDOM S AMPLES

Original: Our model is capable of fully random sampling by specifying z ∼ N( 0 , I ) to be draws from a unit Gaussian.
Revised: Our model is capable of fully random sampling by specifying z ∼ N( 0 , I ) to be draws from a unit Gaussian.

Original: Fig.
Revised: Fig.

Original: 6–8 show samples from an automodulator trained with the FFHQ/C ELEB A- HQ/LSUN data sets up to resolution 128 × 128 .
Revised: 6–8 show samples from an automodulator trained with the FFHQ/C ELEB AHQ/LSUN data sets up to resolution 256 × 256 .

Original: The samples here indicate the full range of samples and face features the model can support.
Revised: The samples here indicate the full range of samples and face features the model can support

Original: [Figure]
Revised: [Figure]

Original: A. R ECONSTRUCTIONS
Revised: A. R ECONSTRUCTIONS

Original: We include examples of the reconstruction capabilities of the automodulator at 256 × 256 in for uncurated test set samples from the FFHQ and C ELEB A-HQ data sets.
Revised: We include examples of the reconstruction capabilities of the automodulator at 256 × 256 in for uncurated test set samples from the FFHQ and C ELEB A-HQ data sets.

Original: These examples are provided in Figs.
Revised: These examples are provided in Figs.

Original: A. C ONDITIONAL S AMPLING
Revised: A. C ONDITIONAL S AMPLING

Original: The automodulator directly allows for conditional sampling in the sense of ﬁxing a latent encoding z A , but allowing some of the modulations come from a random encoding z B ∼ N( 0 , I ) .
Revised: The automodulator directly allows for conditional sampling in the sense of ﬁxing a latent encoding z A , but allowing some of the modulations come from a random encoding z B ∼ N( 0 , I ) .

Original: In Fig.
Revised: In Fig.

Original: 11, we show conditional sampling of 128 × 128 random face images based on ‘coarse’ (latent resolutions 4 × 4 – 8 × 8 ) and ‘intermediate’ ( 16 × 16 – 32 × 32 ) latent features of the ﬁxed input.
Revised: 11, we show conditional sampling of 128 × 128 random face images based on ‘coarse’ (latent resolutions 4 × 4 – 8 × 8 ) and ‘intermediate’ ( 16 × 16 – 32 × 32 ) latent features of the ﬁxed input.

Original: The input image controls the coarse features (such as head shape, pose, gender) on the top and more ﬁne features (expressions, accessories, eyebrows) on the bottom.
Revised: The input image controls the coarse features (such as head shape, pose, gender) on the top and more ﬁne features (expressions, accessories, eyebrows) on the bottom.

Original: A. S TYLE M IXING AND I NTERPOLATION
Revised: A. S TYLE M IXING AND I NTERPOLATION

Original: The well disentangled latent space allows for interpolations between encoded images.
Revised: The well disentangled latent space allows for interpolations between encoded images.

Original: We show regular latent space interpolations between the reconstructions of new input images (Fig. 12).
Revised: We show regular latent space interpolations between the reconstructions of new input images (Fig. 12).

Original: As two more systematic style mixing examples, we include style mixing results based on both
Revised: As two more systematic style mixing examples, we include style mixing results based on both

Original: FFHQ and LSUN Cars.
Revised: FFHQ and LSUN Cars.

Original: The source images are unseen real test images, not self-generated images.
Revised: The source images are unseen real test images, not self-generated images.

Original: In Figs. 13 and 14 we show a matrix of cross-mixing either ‘coarse’ (latent resolutions 4 × 4 – 8 × 8 ) or ‘intermediate’ ( 16 × 16 – 32 × 32 ) latent features.
Revised: In Figs. 13 and 14 we show a matrix of cross-mixing either ‘coarse’ (latent resolutions 4 × 4 – 8 × 8 ) or ‘intermediate’ ( 16 × 16 – 32 × 32 ) latent features.

Original: Mixing coarse features results in large-scale changes, such as pose, while the intermediate features drive ﬁner details, such as color.
Revised: Mixing coarse features results in large-scale changes, such as pose, while the intermediate features drive ﬁner details, such as color.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]