Original: Self-Supervised Variational Auto-Encoders
Revised: Self-Supervised Variational Auto-Encoders

Original: Anonymous Authors

Original: Anonymous Institution

Original: Abstract
Revised: Abstract

Original: Density estimation, compression, and data generation are crucial tasks in artiﬁcial intelligence. Variational Auto-Encoders (VAEs) constitute a single framework to achieve these goals.
Revised: Variational Auto-Encoders (VAEs) constitute a single framework to achieve density estimation, compression, and data generation.

Original: Here, we present a novel class of generative models, called self-supervised Variational Auto-Encoder (selfVAE), that utilizes deterministic and discrete transformations of data.
Revised: Here, we present a novel class of generative models, called self-supervised Variational Auto-Encoder (selfVAE), that utilizes deterministic and discrete transformations of data.

Original: This class of models allows performing both conditional and unconditional sampling while simplifying the objective function.
Revised: The models allow performing both conditional and unconditional sampling while simplifying the objective function.

Original: First, we use a single self-supervised transformation as a latent variable, where a transformation is either downscaling or edge detection.
Revised: First, we use a single self-supervised transformation as a latent variable, where a transformation is either downscaling or edge detection.

Original: Next, we consider a hierarchical architecture, i.e., multiple transformations, and we show its beneﬁts compared to the VAE.
Revised: Next, we consider a hierarchical architecture, i.e., multiple transformations, and we show its beneﬁts compared to the VAE.

Original: The ﬂexibility of selfVAE in data reconstruction ﬁnds a particularly interesting use case in data compression tasks, where we can trade-oﬀ memory for better data quality, and vice-versa.
Revised: The ﬂexibility of selfVAE in data reconstruction ﬁnds a particularly interesting use case in data compression tasks, where we can trade-oﬀ memory for better data quality, and vice-versa.

Original: We present the performance of our approach on three benchmark image data (Cifar10, Imagenette64, and CelebA).
Revised: We present the performance of our approach on Cifar10, Imagenette64, and CelebA.

Original: 1. Introduction
Revised: 1. Introduction

Original: The framework of variational autoencoders (VAEs) provides a principled approach for learning latent-variable models.
Revised: The framework of variational autoencoders (VAEs) provides a principled approach for learning latent-variable models.

Original: As it utilizes a meaningful low-dimensional latent space with density estimation capabilities, it forms an attractive solution for generative modelling tasks.
Revised: As it utilizes a meaningful low-dimensional latent space with density estimation capabilities, it forms an attractive solution for generative modeling tasks.

Original: However, its performance in terms of the test log-likelihood and quality of generated samples is often disappointing, thus, many modiﬁcations were proposed.
Revised: However, its performance in terms of the test log-likelihood and quality of generated samples is often disappointing, thus, many modiﬁcations were proposed.

Original: In general, one can obtain a tighter lower bound, and, thus, a more powerful and ﬂexible model, by advancing over the following three components: the encoder (Rezende et al., 2014; van den Berg et al., 2018; Hoogeboom et al., 2020;
Revised: In general, one can obtain a tighter lower bound and, thus, a more powerful and ﬂexible model, by advancing over the following three components: the encoder (Rezende et al., 2014; van den Berg et al., 2018; Hoogeboom et al., 2020;

Original: Maaløe et al., 2016), the prior (or marginal over latents) (Chen et al., 2016;
Revised: Maaløe et al., 2016), the prior (or marginal over latents) (Chen et al., 2016;

Original: Habibian et al., 2019; Lavda et al., 2020; Lin and Clark, 2020; Tomczak and Welling, 2017) and the decoder (Gulrajani et al., 2016).
Revised: Habibian et al., 2019; Lavda et al., 2020; Lin and Clark, 2020; Tomczak and Welling, 2017) and the decoder (Gulrajani et al., 2016).

Original: Recent studies have shown that by employing deep hierarchical architectures and by carefully designing building blocks of the neural networks,
Revised: Recent studies have shown that by employing deep hierarchical architectures and by carefully designing building blocks of the neural networks,

Original: VAEs can successfully model high-dimensional data and reach state-of-the-art test likelihoods (Zhao et al., 2017; Maaløe et al., 2019; Vahdat and Kautz, 2020).
Revised: VAEs can successfully model high-dimensional data and reach state-of-the-art test likelihoods (Zhao et al., 2017; Maaløe et al., 2019; Vahdat and Kautz, 2020).

Original: In this work, we present a novel class of VAEs, called self-supervised Variational AutoEncoders , where we introduce additional variables to VAEs that result from discrete and deterministic transformations of observed images.
Revised: In this work, we present a novel class of VAEs, called self-supervised Variational AutoEncoders , where we introduce additional variables to VAEs that result from discrete and deterministic transformations of observed images.

Original: Since the transformations are deterministic, and they provide a speciﬁc aspect of images (e.g., contextual information through detecting edges or downscaling), we refer to them as self-supervised representations .
Revised: Since the transformations are deterministic, and they provide a speciﬁc aspect of images (e.g., contextual information through detecting edges or downscaling), we refer to them as self-supervised representations .

Original: The introduction of the discrete and deterministic variables allows to train deep hierarchical models eﬃciently by decomposing the task of learning a highly complex distribution into training smaller and conditional distributions.
Revised: The introduction of the discrete and deterministic variables allows training deep hierarchical models eﬃciently by decomposing the task of learning a highly complex distribution into training smaller and conditional distributions.

Original: In this way, the model allows to integrate the prior knowledge about the data, but still enables to synthesize unconditional samples.
Revised: In this way, the model allows to integrate the prior knowledge about the data, but still enables to synthesize unconditional samples.

Original: Furthermore, the discrete and deterministic variables could be used to conditionally reconstruct data, which could be of great use in data compression and super-resolution tasks.
Revised: Furthermore, the discrete and deterministic variables could be used to conditionally reconstruct data, which could be of great use in data compression and super-resolution tasks.

Original: 2. Our approach Background
Revised: 2. Our approach Background

Original: Let x ∈
Revised: Let x ∈

Original: X
Revised: X

Original: D be a vector of observable variables, where X ⊆ R or X ⊆ Z , and z ∈ R M denote a vector of latent variables.
Revised: D be a vector of observable variables, where X ⊆ R or X ⊆ Z , and z ∈ R M denote a vector of latent variables.

Original: Since calculating p ϑ ( x ) = (cid:82) p ϑ ( x , z )d z is computationally intractable for non-linear stochastic dependencies, a variational family of distributions could be used for approximate inference.
Revised: Since calculating p ϑ ( x ) = (cid:82) p ϑ ( x , z )d z is computationally intractable for non-linear stochastic dependencies, a variational family of distributions could be used for approximate inference.

Original: Then, the following objective function could be derived, namely, the evidence lower bound (ELBO) (Jordan et al., 1999):
Revised: Then, the following objective function could be derived, namely, the evidence lower bound (ELBO) (Jordan et al., 1999):

Original: [Equation]
Revised: [Equation]

Original: (1) where q φ ( z | x ) is the variational posterior (or the encoder ), p θ ( x | z ) is the conditional likelihood function (or the decoder ) and
Revised: (1) where q φ ( z | x ) is the variational posterior (or the encoder ), p θ ( x | z ) is the conditional likelihood function (or the decoder ) and

Original: p λ
Revised: p λ

Original: ( z ) is the prior (or marginal ), φ , θ and λ denote parameters.
Revised: ( z ) is the prior (or marginal ), φ , θ and λ denote parameters.

Original: The expectation is approximated by Monte Carlo sampling while exploiting the reparameterization trickin order to obtain unbiased gradient estimators.
Revised: The expectation is approximated by Monte Carlo sampling while exploiting the reparameterization trick to obtain unbiased gradient estimators.

Original: The models are parameterized by neural networks.
Revised: The models are parameterized by neural networks.

Original: This generative framework is known as Variational Auto-Encoder (VAE) (Kingma and Welling, 2013; Rezende et al., 2014).
Revised: This generative framework is known as Variational Auto-Encoder (VAE) (Kingma and Welling, 2013; Rezende et al., 2014).

Original: a) Generative Model yu zy 2 z… …
Revised: a) Generative Model yu zy 2 z… …

Original: y k z k … x yu zy 2 z… …
Revised: y k z k … x yu zy 2 z… …

Original: y k z k x b) Inference Model i)
Revised: y k z k x b) Inference Model i)

Original: Stochastic dependencies of self-supervised VAE ii)
Revised: Stochastic dependencies of self-supervised VAE ii)

Original: Hierarchical ssVAE self-supervised VAE z
Revised: Hierarchical ssVAE self-supervised VAE z

Original: u y x
Revised: u y x

Original: z d
Revised: z d

Original: Self-supervised representations
Revised: Self-supervised representations

Original: The idea of self-supervised learning is about utilizing original unlabeled data to create additional context information.
Revised: The idea of self-supervised learning is about utilizing original unlabeled data to create additional context information.

Original: It could be achieved in multiple manners, e.g., by adding noise to data (Vincent et al., 2008) or masking data during training (Zhang et al., 2017).
Revised: It could be achieved in multiple manners, e.g., by adding noise to data (Vincent et al., 2008) or masking data during training (Zhang et al., 2017).

Original: Self-supervised learning could also be seen as turning an unsupervised model into a supervised by, e.g., treating predicting next pixels as a classiﬁcation task (Hénaﬀ et al., 2019;
Revised: Self-supervised learning could also be seen as turning an unsupervised model into a supervised by, e.g., treating predicting next pixels as a classiﬁcation task (Hénaﬀ et al., 2019;

Original: Oord et al., 2018).
Revised: Oord et al., 2018).

Original: These are only a few examples of a quickly growing research line (Liu et al., 2020).
Revised: These are only a few examples of a quickly growing research line (Liu et al., 2020).

Original: Here, we propose to use non-trainable transformations to obtain information about image data.
Revised: Here, we propose to use non-trainable transformations to obtain information about image data.

Original: Our main hypothesis is that since working with highly-quality images is challenging, we could alleviate this problem by additionally considering partial information about them.
Revised: Our main hypothesis is that since working with highly-quality images is challenging, we could alleviate this problem by additionally considering partial information about them.

Original: Fitting a model to images of lower quality, and then enhancing them to match the target distribution seems to be overall an easier task (Chang et al., 2004;
Revised: Fitting a model to images of lower quality, and then enhancing them to match the target distribution seems to be overall an easier task (Chang et al., 2004;

Original: Gatopoulos et al., 2020).
Revised: Gatopoulos et al., 2020).

Original: By incorporating compressed transformations (i.e., the self-supervised representations) that still contain global information, with the premise that it would be easier to approximate, the process of modelling a high-dimensional complex density breaks down into simpler tasks.
Revised: By incorporating compressed transformations (i.e., the self-supervised representations) that still contain global information, with the premise that it would be easier to approximate, the process of modeling a high-dimensional complex density breaks down into simpler tasks.

Original: In this way, the expressivity of the model will grow and gradually result into richer, better generations.
Revised: In this way, the expressivity of the model will grow and gradually result in richer, better generations.

Original: A positive eﬀect of the proposed framework is that the model allows us to integrate prior knowledge through the image transformations, without losing its unconditional generative functionality.
Revised: A positive eﬀect of the proposed framework is that the model allows us to integrate prior knowledge through the image transformations, without losing its unconditional generative functionality.

Original: Overall, we end up with a two-level VAE with three latent variables, where one is a data transformation that can be obtained in a self-supervised fashion.
Revised: Overall, we end up with a two-level VAE with three latent variables, where one is a data transformation that can be obtained in a self-supervised fashion.

Original: In Figure 1 a schematic representation of the proposed approach with downscaling is presented.
Revised: In Figure 1 a schematic representation of the proposed approach with downscaling is presented.

Original: A number of exemplary image transformations are presented in Figure 4 (in the appendix).
Revised: A number of exemplary image transformations are presented in Figure 4 (in the appendix).

Original: We can see that with these transformations, even though we discard a lot of information, the global structure is preserved.
Revised: We can see that with these transformations, even though we discard a lot of information, the global structure is preserved.

Original: As a result, in practice the model should have the ability to extract a general concept of the data at the ﬁrst stage, and add local information afterwards.
Revised: As a result, in practice, the model should have the ability to extract a general concept of the data at the ﬁrst stage, and add local information afterward.

Original: In this work, we focus on downscaling (Figure 4.b, c & d in the appendix) and edge detection or sketching (Fig. 4.i in the appendix).
Revised: In this work, we focus on downscaling (Figure 4.b, c & d in the appendix) and edge detection or sketching (Fig. 4.i in the appendix).

Original: Model formulation
Revised: Model formulation

Original: In our model, we consider representations that result from deterministic and discrete transformations of an image.
Revised: In our model, we consider representations that result from deterministic and discrete transformations of an image.

Original: Formally, we introduce a transformation d : X D → X C that takes x and returns an image representation y , e.g., a downscaled image.
Revised: Formally, we introduce a transformation d : X D → X C that takes x and returns an image representation y , e.g., a downscaled image.

Original: Since we lose information about the original image, z could be seen as a variable that compensates lost details in x .
Revised: Since we lose information about the original image, z could be seen as a variable that compensates lost details in x .

Original: Further we propose to introduce an additional latent variable, u ∈ R N to model y and z .
Revised: Further we propose to introduce an additional latent variable, u ∈ R N to model y and z .

Original: We can deﬁne the joint distribution of x and y as follows: p ( x , y )
Revised: We can deﬁne the joint distribution of x and y as follows: p ( x , y )

Original: =
Revised: =

Original: p ( y | x ) p ( x ) , where p ( y | x ) =
Revised: p ( y | x ) p ( x ) , where p ( y | x ) =

Original: δ ( y − d ( x )) due to the deterministic transformation d ( · ) , where δ ( · ) is the Kronecker delta.
Revised: δ ( y − d ( x )) due to the deterministic transformation d ( · ) , where δ ( · ) is the Kronecker delta.

Original: Thus, the empirical distribution is δ ( y − d ( x ))
Revised: Thus, the empirical distribution is δ ( y − d ( x ))

Original: p data ( x ) .
Revised: p data ( x ) .

Original: However, since we are interested in decomposing the problem of modeling a complex distribution p ( x ) , we propose to model p ( x | y ) p ( y ) instead, and utilize the variational inference of the form Q ( u , z | x , y )
Revised: However, since we are interested in decomposing the problem of modeling a complex distribution p ( x ) , we propose to model p ( x | y ) p ( y ) instead, and utilize the variational inference of the form Q ( u , z | x , y )

Original: =
Revised: =

Original: q ( u | y ) q ( z | x ) that yields:
Revised: q ( u | y ) q ( z | x ) that yields:

Original: [Equation]
Revised: [Equation]

Original: In the appendix (see Section B), we present the speciﬁc choices of the distributions.
Revised: In the appendix (see Section B), we present the speciﬁc choices of the distributions.

Original: In order to highlight the self-supervised part in our model, we refer to it as the self-supervised 
Revised: To highlight the self-supervised part in our model, we refer to it as the self-supervised Variational

Original: Variational Auto-Encoder (or selfVAE for short).
Revised: Auto-Encoder (or selfVAE for short).

Original: Generation and Reconstruction in selfVAE
Revised: Generation and Reconstruction in selfVAE

Original: As generative models, VAEs can be used to synthesize novel content through the following process: z ∼ p ( z ) → x ∼ p ( x | z ) , but also to reconstruct a data sample
Revised: As generative models, VAEs can be used to synthesize novel content through the following process: z ∼ p ( z ) → x ∼ p ( x | z ) , but also to reconstruct a data sample

Original: x ∗ by using the following scheme:
Revised: x ∗ by using the following scheme:

Original: z ∼ q ( z | x ∗ ) → x ∼ p ( x | z ) .
Revised: z ∼ q ( z | x ∗ ) → x ∼ p ( x | z ) .

Original: Interestingly, our approach allows to utilize more operations regarding data generation and reconstruction.
Revised: Interestingly, our approach allows to utilize more operations regarding data generation and reconstruction.

Original: First, analogously to VAEs, the selfVAE allows to generate data by applying the following hierarchical sampling process ( generation ): u ∼ p ( u ) −→ y ∼ p ( y
Revised: First, analogously to VAEs, the selfVAE allows to generate data by applying the following hierarchical sampling process ( generation ): u ∼ p ( u ) −→ y ∼ p ( y

Original: | u ) −→ z ∼ p ( z | u , y ) −→ x ∼ p ( x | y , z ) .
Revised: | u ) −→ z ∼ p ( z | u , y ) −→ x ∼ p ( x | y , z ) .

Original: However, we can use the ground-truth y (i.e, y ∗ =
Revised: However, we can use the ground-truth y (i.e, y ∗ =

Original: d ( x ∗ ) ), and sample or infer z .
Revised: d ( x ∗ ) ), and sample or infer z .

Original: Then, the generative process for the former ( conditional generation ) is: z ∼ q ( z | x ∗ ) −→ x ∼ p ( x | y ∗ , z ) , and for the latter ( conditional reconstruction ): u ∼ q ( u | y ∗ )
Revised: Then, the generative process for the former ( conditional generation ) is: z ∼ q ( z | x ∗ ) −→ x ∼ p ( x | y ∗ , z ) , and for the latter ( conditional reconstruction ): u ∼ q ( u | y ∗ )

Original: −→ z ∼ p ( z | u , y ∗ ) , −→ x ∼ p ( x | y ∗ , z ) .
Revised: −→ z ∼ p ( z | u , y ∗ ) , −→ x ∼ p ( x | y ∗ , z ) .

Original: If y is a downscaling transformation of the input image, selfVAE can be used in a manner similar to the super-resolution (Gatopoulos et al., 2020).
Revised: If y is a downscaling transformation of the input image, selfVAE can be used in a manner similar to the super-resolution (Gatopoulos et al., 2020).

Original: Alternatively, we can sample (or generate) y instead, and choose to sample or infer z .
Revised: Alternatively, we can sample (or generate) y instead, and choose to sample or infer z .

Original: In this way, we can reconstruct an image in two ways, namely, reconstruction 1 : y ∗ = 
Revised: In this way, we can reconstruct an image in two ways, namely, reconstruction 1 : y ∗ = d ( x [Equation]

Original: d ( x ∗ ) −→ u ∼ q ( u | y ∗ ) −→ y ∼ p ( y | u) −→ z ∼ p ( z |u , y ) −→ x ∼ p ( x | z , y ) , and reconstruction 2 : (cid:16) y 
Revised: ) −→ u ∼ q [Equation] u | y [Equation]

Revised: −→ y

Original: [Equation]
Revised: [Equation]

Revised: p

Original:  u ) (cid:17) , then z ∼ q ( z |  x ∗ ) 
Revised: [Equation] y | u [Equation] −→ z ∼ p ( z | u , y ) −→ x ∼ p ( x | z , y ) , and reconstruction 2 : (cid:16)

Original:  −→ x ∼ p ( x | y , z ).
Revised: [Equation] (cid:17) then z ∼ q ( z | x ∗ ) −→ x ∼ p ( x | y , z )

Original: The presented versions of generating and reconstructing images could be useful in the compression task.
Revised: The presented versions of generating and reconstructing images could be useful in the compression task.

Original: As we will see in the experiments, each option creates a diﬀerent ratio of the reconstruction quality against the memory that we need to allocate to send information.
Revised: As we will see in the experiments, each option creates a diﬀerent ratio of the reconstruction quality against the memory that we need to allocate to send information.

Original: However, every inferred variable needs to be sent, thus, more sampling corresponds to lower memory requirements.
Revised: However, every inferred variable needs to be sent, thus, more sampling corresponds to lower memory requirements.

Original: u … … x a) Generative Model y K u … … …
Revised: u … … x a) Generative Model y K u … … …

Original: x y K 1 − y K
Revised: x y K 1 − y K

Original: yy K 1 − yb)
Revised: yy K 1 − yb)

Original: Inference Model u
Revised: Inference Model u

Original: z y x i)
Revised: z y x i)

Original: Stochastic dependencies of self-supervised VAE ii) Stochastic dependencies of the hierarchical ssVAE z K 1 − z K
Revised: Stochastic dependencies of self-supervised VAE ii) Stochastic dependencies of the hierarchical ssVAE z K 1 − z K

Original: z 1 zz K 1 − z K d d d
Revised: z 1 zz K 1 − z K d d d

Original: Hierarchical self-supervised VAE
Revised: Hierarchical self-supervised VAE

Original: The proposed approach can be further extended and generalized by introducing multiple transformations, in the way that it is illustrated in Figure 2.
Revised: The proposed approach can be further extended and generalized by introducing multiple transformations, in the way that it is illustrated in Figure 2.

Original: By incorporating a single (or multiple) self-supervised representation(s) of the data, the process of modelling a high-dimensional complex density breaks down into K simpler modeling tasks.
Revised: By incorporating a single (or multiple) self-supervised representation(s) of the data, the process of modeling a high-dimensional complex density breaks down into K simpler modeling tasks.

Original: Thus, we obtain a K -level VAE architecture, where the overall expressivity of the model grows even further and gradually results into generations of higher quality.
Revised: Thus, we obtain a K -level VAE architecture, where the overall expressivity of the model grows even further and gradually results in generations of higher quality.

Original: Some transformations cannot be applied multiple times (e.g., edge detection), however, others could be used sequentially, e.g., downscaling.
Revised: Some transformations cannot be applied multiple times (e.g., edge detection), however, others could be used sequentially, e.g., downscaling.

Original: We take K self-supervised data transformations d k ( · ) that give K representations denoted by y 1: K , and the following variational distributions:
Revised: We take K self-supervised data transformations d k ( · ) that give K representations denoted by y 1: K , and the following variational distributions:

Original: [Equation]
Revised: [Equation]

Revised: K

Original: [Equation]
Revised: [Equation]

Revised: k

Revised: [Equation]

Revised: [Equation]

Original: that yields the following objective:
Revised: that yields the following objective:

Original: [Equation]
Revised: [Equation]

Original: 3. Experiments
Revised: 3. Experiments

Original: Setup
Revised: Setup

Original: We evaluate the proposed model on CIFAR-10, Imagenette64 1 and CelebA. Encoders and decoders consist of building blocks composed of DenseNets (Huang et al., 2016), channelwise attention (Zhang et al., 2018), and ELUs (Clevert et al., 2015) as activation functions.
Revised: We evaluate the proposed model on CIFAR-10, Imagenette64 1 and CelebA. Encoders and decoders consist of building blocks composed of DenseNets (Huang et al., 2016), channelwise attention (Zhang et al., 2018), and ELUs (Clevert et al., 2015) as activation functions.

Original: The dimensionality of all the latent variables were kept at 8 × 8 × 16 = 1024 and all models were trained using AdaMax (Kingma and Ba, 2014) with data-dependent initialization (Salimans and Kingma, 2016).
Revised: The dimensionality of all the latent variables were kept at 8 × 8 × 16 = 1024 and all models were trained using AdaMax (Kingma and Ba, 2014) with data-dependent initialization (Salimans and Kingma, 2016).

Original: Regarding the selfVAEs, in CIFAR-10 we used an architecture with a single downscaled transformation (selfVAE-downscale), while on the remaining two datasets (CelebA and Imagenette64) we used a hierarchical 3-leveled selfVAE with downscaling, and a selfVAE with sketching.
Revised: Regarding the selfVAEs, in CIFAR-10 we used an architecture with a single downscaled transformation (selfVAE-downscale), while on the remaining two datasets (CelebA and Imagenette64) we used a hierarchical 3-leveled selfVAE with downscaling, and a selfVAE with sketching.

Original: All models were employed with the bijective prior (RealNVP) comparable in terms of the number of parameters (the range of the weights of all models was from 32 M to 42 M).
Revised: All models were employed with the bijective prior (RealNVP) comparable in terms of the number of parameters (the range of the weights of all models was from 32 M to 42 M).

Original: For more details, please refer to the appendix sections D and C.
Revised: For more details, please refer to the appendix sections D and C.

Original: We approximate the negative log-likelihood using 512 IW-samples (Burda et al., 2015) and express the scores in bits per dimension ( bpd ).
Revised: We approximate the negative log-likelihood using 512 IW-samples (Burda et al., 2015) and express the scores in bits per dimension ( bpd ).

Original: Additionally, for CIFAR-10, we use the
Revised: Additionally, for CIFAR-10, we use the

Original: Fréchet Inception Distance (FID) (Heusel et al., 2017).
Revised: Fréchet Inception Distance (FID) (Heusel et al., 2017).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Quantitative results
Revised: Quantitative results

Original: We present the results of the experiments on the benchmark datasets in Table 1.
Revised: We present the results of the experiments on the benchmark datasets in Table 1.

Original: First, we notice that on CIFAR-10 our implementation of the VAE is still lacking behind other generative models in terms of bpd , however, it is better or comparable in terms of FID.
Revised: First, we notice that on CIFAR-10 our implementation of the VAE is still lacking behind other generative models in terms of bpd , however, it is better or comparable in terms of FID.

Original: The selfVAE-downscale achieves worse bpd than the VAE.
Revised: The selfVAE-downscale achieves worse bpd than the VAE.

Original: A possible explanation may lie in the small image size ( 32 × 32 ), as the beneﬁts of breaking down the learning process in two or more steps are not obvious given the small target dimensional space.
Revised: A possible explanation may lie in the small image size ( 32 × 32 ), as the beneﬁts of breaking down the learning process in two or more steps are not obvious given the small target dimensional space.

Original: Nevertheless, the selfVAE-downscale achieves signiﬁcantly better FID scores than the other generative models.
Revised: Nevertheless, the selfVAE-downscale achieves signiﬁcantly better FID scores than the other generative models.

Original: This result could follow from the fact that downscaling allows to maintain context information about the original image and, as a result, a general coherence is of higher quality.
Revised: This result could follow from the fact that downscaling allows maintaining context information about the original image and, as a result, a general coherence is of higher quality.

Original: Interestingly, on the two other datasets, a three-level selfVAE-downscale achieves signiﬁcantly better bpd scores than the VAE with the bijective prior.
Revised: Interestingly, on the two other datasets, a three-level selfVAE-downscale achieves signiﬁcantly better bpd scores than the VAE with the bijective prior.

Original: This indicates the beneﬁt of employing a multi-leveled self-supervised framework against the VAE in higher-dimensional data, where the plain model fails to scale eﬃciently.
Revised: This indicates the beneﬁt of employing a multi-leveled self-supervised framework against the VAE in higher-dimensional data, where the plain model fails to scale eﬃciently.

Original: It seems that the hierarchical structure of self-supervised random variables allows to encode the missing information more eﬃciently in z k , in contrast to the vanilla VAE, where all information about images must be coded in z .
Revised: It seems that the hierarchical structure of self-supervised random variables allows encoding the missing information more eﬃciently in z k , in contrast to the vanilla VAE, where all information about images must be coded in z .

Original: Qualitative results
Revised: Qualitative results

Original: We present generations on CIFAR-10 and Imagenette64 in Figure 6 and on CelebA in Figure 7, and reconstructions on CIFAR-10 and CelebA in Figure 3.
Revised: We present generations on CIFAR-10 and Imagenette64 in Figure 6 and on CelebA in Figure 7, and reconstructions on CIFAR-10 and CelebA in Figure 3.

Original: We ﬁrst notice that the generations from selfVAE seem to be more coherent, in contrast with these from VAE that produces overall more contextless and distorted generations.
Revised: We ﬁrst notice that the generations from selfVAE seem to be more coherent, in contrast with these from VAE that produces overall more contextless and distorted generations.

Original: This result seems to be in line with the FID scores.
Revised: This result seems to be in line with the FID scores.

Original: Especially for CelebA, we observe impressive synthesis quality, great sampling diversity and coherent generations (Figure 7).
Revised: Especially for CelebA, we observe impressive synthesis quality, great sampling diversity, and coherent generations (Figure 7).

Original: On the
Revised: On the

Original: Imagenette64 dataset, we can also observe crisper generations for our method compared to
Revised: Imagenette64 dataset, we can also observe crisper generations for our method compared to

Revised: [Figure]

Original: VA E s e l f VA E - do w n s ca l e - 2 l v l
Revised: s e l f VA E - do w n s ca l e - 2 l v l

Original: Reconstruction Data Sample
Revised: [Figure] Data Sample

Original: Reconstruction Reconstruction
Revised: Reconstruction Reconstruction

Original: Conditional Generation
Revised: Conditional Generation

Original: ConditionalReconstruction
Revised: ConditionalReconstruction

Original: Self-Supervised Representation
Revised: Self-Supervised Representation

Original: Self-Supervised Reconstruction
Revised: Self-Supervised Reconstruction

Original: [Figure]
Revised: [Figure]

Original: 8% C o m p r e ss i on ra ti o : 8% 16% 25% 33% 8% 16%
Revised: [Figure] p [Figure] e ss i on r

Original: [Figure]
Revised: [Figure]

Original: Reconstruction
Revised: ti o : 8% 16% 25% 33% 8% 16%

Original: [Figure]
Revised: [Figure]

Original: ConditionalReconstruction
Revised: ConditionalReconstruction

Original: Reconstruction Reconstruction
Revised: Reconstruction Reconstruction

Original: [Figure]
Revised: [Figure]

Original: s e l f VA E - do w n s ca l e - 3 l v l s e l f VA E - s k e t c h
Revised: s e l f VA E - do w n s ca l e - 3 l v l s e l f VA E - s k e t c h

Original: [Figure]
Revised: [Figure]

Original: the VAE (Figure 6).
Revised: the VAE (Figure 6).

Original: Furthermore, the hierarchical selfVAE seems to be ofa great potential for compression purposes.
Revised: Furthermore, the hierarchical selfVAE seems to be of great potential for compression purposes.

Original: In contrast to the VAE, which is restricted to using a single way of reconstructing an image, the selfVAE allows four various options with diﬀerent quality/memory ratios (Figure 3).
Revised: In contrast to the VAE, which is restricted to using a single way of reconstructing an image, the selfVAE allows four various options with diﬀerent quality/memory ratios (Figure 3).

Original: In the selfVAE-sketch, we can retrieve the image with high accuracy by using only 16% of the original data, as it manages to encode all the texture of the image to z (Figure 8).
Revised: In the selfVAE-sketch, we can retrieve the image with high accuracy by using only 16% of the original data, as it manages to encode all the texture of the image to z (Figure 8).

Original: This shows the advantage of choosing prior knowledge into the learning process.
Revised: This shows the advantage of choosing prior knowledge into the learning process.

Original: Lastly, the latents learn to add extra information, which deﬁnes the end result, and we can alter details of an image like facial expressions (Figure 9.ii).
Revised: Lastly, the latents learn to add extra information, which deﬁnes the end result, and we can alter details of an image like facial expressions (Figure 9.ii).

Original: 4. Conclusion
Revised: 4. Conclusion

Original: In this paper, we showed that taking deterministic and discrete transformations results in coherent generations of high visual quality, and allows to integrate prior knowledge without loosing its unconditional generative functionality.
Revised: In this paper, we showed that taking deterministic and discrete transformations results in coherent generations of high visual quality, and allows to integrate prior knowledge without losing its unconditional generative functionality.

Original: The experimental results seem to conﬁrm that hierarchical architectures perform better and allow to obtain both better bpd scores and better generations and reconstructions.
Revised: The experimental results seem to conﬁrm that hierarchical architectures perform better and allow to obtain both better bpd scores and better generations and reconstructions.

Original: In the experiments, we considered two classes of image transformations, namely, downscaling and edge detection ( sketching ).
Revised: In the experiments, we considered two classes of image transformations, namely, downscaling and edge detection ( sketching ).

Original: However, there is a vast of possible other transformations (see Figure 4), and we leave investigating them for future work.
Revised: However, there is a vast of possible other transformations (see Figure 4), and we leave investigating them for future work.

Original: Moreover, we ﬁnd the proposed approach interesting for the compression task.
Revised: Moreover, we ﬁnd the proposed approach interesting for the compression task.

Original: A similar approach with a multi-scale auto-encoder for image compression was proposed, e.g, by Mentzer et al.
Revised: A similar approach with a multi-scale auto-encoder for image compression was proposed, e.g, by Mentzer et al.

Original: (2019) or Razavi et al.
Revised: (2019) or Razavi et al.

Original: However, we still use a probabilistic framework and indicate that various non-trainable image transformations (not only multiple scales) could be of great potential.
Revised: However, we still use a probabilistic framework and indicate that various non-trainable image transformations (not only multiple scales) could be of great potential.

Original: Appendix A. Non-trainable image transformations
Revised: Appendix A. Non-trainable image transformations

Original: In Figure 4, we present various examples of discrete and deterministic (and non-trainable) image transformations.
Revised: In Figure 4, we present various examples of discrete and deterministic (and non-trainable) image transformations.

Original: [Figure]
Revised: [Figure]

Original: Appendix B. Speciﬁc choices of distributions in selfVAE
Revised: Appendix B. Speciﬁc choices of distributions in selfVAE

Original: In this paper
Revised: In this paper

Original: [Equation]
Revised: [Equation]

Original: we use the following distributions for the generative part of the selfVAE:
Revised: we use the following distributions for the generative part of the selfVAE:

Original: [Equation]
Revised: [Equation]

Original: where Dlogistic is deﬁned as the discretized logistic distribution (Salimans et al., 2017)
Revised: where Dlogistic is deﬁned as the discretized logistic distribution (Salimans et al., 2017)

Original: [Equation]
Revised: [Equation]

Original: and we utilize a ﬂow-based model for p λ ( u ) .
Revised: and we utilize a ﬂow-based model for p λ ( u ) .

Original: Further, we consider the following distributions for the variational part:
Revised: Further, we consider the following distributions for the variational part:

Original: [Equation]
Revised: [Equation]

Original: Appendix C. Datasets CIFAR- 
Revised: Appendix C. Datasets CIFAR-

Original: The CIFAR-10 dataset is a well-known image benchmark data containing 60.000 training examples and 10.000 validation examples.
Revised: The CIFAR-10 dataset is a well-known image benchmark data containing 60.000 training examples and 10.000 validation examples.

Original: From the training data, we put aside 15% randomly selected images as the test set.
Revised: From the training data, we put aside 15% randomly selected images as the test set.

Original: We augment the training data by using random horizontal ﬂips and random aﬃne transformations and normalize the data uniformly in the range (0, 1).
Revised: We augment the training data by using random horizontal ﬂips and random aﬃne transformations and normalize the data uniformly in the range (0, 1).

Original: Imagenette
Revised: Imagenette

Original: Imagenette64 3 is a subset of 10 classes from the downscaled Imagenet dataset.
Revised: Imagenette64 3 is a subset of 10 classes from the downscaled Imagenet dataset.

Original: We downscaled the dataset to 64 px × 64 px images.
Revised: We downscaled the dataset to 64 px × 64 px images.

Original: Similarly to CIFAR-10, e put aside 15% randomly selected training images as the test set.
Revised: Similarly to CIFAR-10, e put aside 15% randomly selected training images as the test set.

Original: We used the same data augmentation as in CIFAR-
Revised: We used the same data augmentation as in CIFAR-

Original: CelebA
Revised: CelebA

Original: The Large-scale CelebFaces Attributes (CelebA) Dataset consists of 202 .images of celebrities.
Revised: The Large-scale CelebFaces Attributes (CelebA) Dataset consists of 202 .images of celebrities.

Original: We cropped original images on the 40 vertical and 15 horizontal component of the top left corner of the crop box, which height and width were cropped to 148.
Revised: We cropped original images on the 40 vertical and 15 horizontal component of the top left corner of the crop box, which height and width were cropped to 148.

Original: Besides the uniform normalization of the image, no other augmentation was applied.
Revised: Besides the uniform normalization of the image, no other augmentation was applied.

Original: Appendix D. Neural Network Architecture
Revised: Appendix D. Neural Network Architecture

Original: The choice of the NN architecture is crucial for the performance and the scalability of the overall framework, and usually architectures that showcased great performance in discriminate tasks (i.e. classiﬁcation) are used in generative modelling tasks as well.
Revised: The choice of the NN architecture is crucial for the performance and the scalability of the overall framework, and usually architectures that showcased great performance in discriminate tasks (i.e. classiﬁcation) are used in generative modelling tasks as well.

Original: However, the internal representations that the networks have to discover are fundamentally diﬀerent, and little attention has been given into designing a NN speciﬁcally for an auto-encoder setting.
Revised: However, the internal representations that the networks have to discover are fundamentally diﬀerent, and little attention has been given into designing a NN speciﬁcally for an auto-encoder setting.

Original: For example, in classiﬁcation tasks the network extracts speciﬁc representation of a particular object, in contrast with the generative models, where we aim for discovering the semantic structure of the data.
Revised: For example, in classiﬁcation tasks the network extracts speciﬁc representation of a particular object, in contrast with the generative models, where we aim for discovering the semantic structure of the data.

Original: Thus, as we argue that we can beneﬁt from a carefully designed architecture, in this section we present our approach.
Revised: Thus, as we argue that we can beneﬁt from a carefully designed architecture, in this section we present our approach.

Original: For building blocks of the network, we employed densely connected convolutional networks instead of residual ones.
Revised: For building blocks of the network, we employed densely connected convolutional networks instead of residual ones.

Original: The motivation for this choice is that since DenseNets encourage feature reuse, it will help preserve visual information from the very ﬁrst layer eﬀectively, while requiring less trainable parameters.
Revised: The motivation for this choice is that since DenseNets encourage feature reuse, it will help preserve visual information from the very ﬁrst layer eﬀectively, while requiring less trainable parameters.

Original: Thus, the network could discover easier generic graphical features and local pixel correlations.
Revised: Thus, the network could discover easier generic graphical features and local pixel correlations.

Original: The concatenation of the ﬁlters will also alleviate the vanishing-gradient problem and allow us to build deep architectures.
Revised: The concatenation of the ﬁlters will also alleviate the vanishing-gradient problem and allow us to build deep architectures.

Original: Additionally, exponential linear units (ELUs) are used everywhere as activation functions.
Revised: Additionally, exponential linear units (ELUs) are used everywhere as activation functions.

Original: In contrast to
Revised: In contrast to

Original: ReLUs, ELUs have negative values which allows them to push mean unit activations closer to zero, which speeds up the learning process.
Revised: ReLUs, ELUs have negative values which allows them to push mean unit activations closer to zero, which speeds up the learning process.

Original: This is due to a reduced bias shift eﬀect ; bias that is introduced to the units from those of the previous layer which have a non-zero mean activation.
Revised: This is due to a reduced bias shift eﬀect ; bias that is introduced to the units from those of the previous layer which have a non-zero mean activation.

Original: Typically, every convolution operation precedes a batch normalization layer, as they empirically exhibit a boost in performance in discriminate tasks.
Revised: Typically, every convolution operation precedes a batch normalization layer, as they empirically exhibit a boost in performance in discriminate tasks.

Original: However, their performance is known to degrade for small batch sizes, as the variance of the activation noise that they contribute is inversely proportional to the number of data that is processed.
Revised: However, their performance is known to degrade for small batch sizes, as the variance of the activation noise that they contribute is inversely proportional to the number of data that is processed.

Original: This noise injection, in combination with their intensive memory demands, can be critical drawbacks when we process image data, especially high-dimensional ones.
Revised: This noise injection, in combination with their intensive memory demands, can be critical drawbacks when we process image data, especially high-dimensional ones.

Original: We instead use weight normalization, where even though it separates the weight vector from its direction just like batch normalization, they do not make use of the variance.
Revised: We instead use weight normalization, where even though it separates the weight vector from its direction just like batch normalization, they do not make use of the variance.

Original: This allows them to get the desired output even in small mini-batches, while allocating a small proportion of memory.
Revised: This allows them to get the desired output even in small mini-batches, while allocating a small proportion of memory.

Original: We empirically ﬁnd out that indeed using weight normalization reduces the overﬁtting of the model.
Revised: We empirically ﬁnd out that indeed using weight normalization reduces the overﬁtting of the model.

Original: In addition, we used a data-dependant initialization of the model parameters, by sampling the ﬁrst batch of the training set.
Revised: In addition, we used a data-dependant initialization of the model parameters, by sampling the ﬁrst batch of the training set.

Original: This will allow the parameters to be adjusted by the output of the previous layers, taking into account and thus resulting into a faster learning process.
Revised: This will allow the parameters to be adjusted by the output of the previous layers, taking into account and thus resulting into a faster learning process.

Original: An important element of the auto-encoding scheme is the process of feature downscaling and upscaling.
Revised: An important element of the auto-encoding scheme is the process of feature downscaling and upscaling.

Original: Despite its success in classiﬁcation tasks, pooling is a ﬁxed operation that replacing it with a stride-convolution layer can also be seen as generalization, as the scaling process is now learned.
Revised: Despite its success in classiﬁcation tasks, pooling is a ﬁxed operation that replacing it with a stride-convolution layer can also be seen as generalization, as the scaling process is now learned.

Original: This will increase the models’ expressibility with the cost of adding a negligible amount of learning parameters.
Revised: This will increase the models’ expressibility with the cost of adding a negligible amount of learning parameters.

Original: For the upscaling operation, even though various methods have been proposed (Shi et al., 2016), we found out that the plain transposed convolution generalised better than the others, while requiring far less trainable parameters.
Revised: For the upscaling operation, even though various methods have been proposed (Shi et al., 2016), we found out that the plain transposed convolution generalised better than the others, while requiring far less trainable parameters.

Original: Finally, inspired from the recent advantages on super-resolution neural network architectures, we used channel-wise attention blocks (CA) at the end of every DenseNet block (Zhang et al., 2018).
Revised: Finally, inspired from the recent advantages on super-resolution neural network architectures, we used channel-wise attention blocks (CA) at the end of every DenseNet block (Zhang et al., 2018).

Original: The CA blocks will help the network to focus on more informative features, by exploiting the inter-dependencies among feature channels.
Revised: The CA blocks will help the network to focus on more informative features, by exploiting the inter-dependencies among feature channels.

Original: Thus, it performs feature recalibration in a global way, where the per-channel summary statistics are calculated and then used to selectively emphasise informative feature-maps as well as suppress useless ones (e.g. redundant feature-maps).
Revised: Thus, it performs feature recalibration in a global way, where the per-channel summary statistics are calculated and then used to selectively emphasise informative feature-maps as well as suppress useless ones (e.g. redundant feature-maps).

Original: This is done through a global average pooling, that squeezes global spatial information into a channel statistical descriptor, followed by a gating mechanism, where it learns nonlinear interactions between the input channels.
Revised: This is done through a global average pooling, that squeezes global spatial information into a channel statistical descriptor, followed by a gating mechanism, where it learns nonlinear interactions between the input channels.

Original: The core building blocks and the network of an auto-encoding network are illustrated in
Revised: The core building blocks and the network of an auto-encoding network are illustrated in

Original: Figure D.
Revised: Figure D.

Original: [Table]
Revised: [Table]

Original: Appendix E. Additional results
Revised: Appendix E. Additional results

Original: [Figure]
Revised: [Figure]

Original: C
Revised: C

Original: I F
Revised: I F

Original: A
Revised: A

Original: RI m a g e n e tt e
Revised: RI m a g e n e tt e

Original: [Figure]
Revised: [Figure]

Original: ond l
Revised: ond l

Original: [Figure]
Revised: [Figure]

Original: v e l F i r s t l e v e l
Revised: v e l F i r s t l e v e l

Original: T h
Revised: T h

Original: i r d l e v e l -
Revised: i r d l e v e l -

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: i ) s e l f VA
Revised: i ) s e l f VA

Original: E - do w n s ca l e - 3 l v ii ) s e l f VA
Revised: E - do w n s ca l e - 3 l v ii ) s e l f VA

Original: E - s k e t c h iii ) VA ER ea l NV P
Revised: E - s k e t c h iii ) VA ER ea l NV P

Original: [Figure]
Revised: [Figure]

Original: l e ss s a m p li
Revised: l e ss s a m p li

Original: ng v a r i a n ce
Revised: ng v a r i a n ce

Original: [Figure]
Revised: [Figure]

Original: s e l f VA Edo w n s ca l e - 3 l v
Revised: s e l f VA Edo w n s ca l e - 3 l v