Original: D ECOUPLING G LOBAL AND L OCAL R EPRESENTATIONS VIA I NVERTIBLE G ENERATIVE F LOWS
Revised: D ECOUPLING G LOBAL AND L OCAL R EPRESENTATIONS VIA I NVERTIBLE G ENERATIVE F LOWS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this work, we propose a new generative model that is capable of automatically decoupling global and local representations of images in an entirely unsupervised setting, by embedding a generative ﬂow in the VAE framework to model the decoder.
Revised: In this work, we propose a new generative model that is capable of automatically decoupling global and local representations of images in an entirely unsupervised setting, by embedding a generative ﬂow in the VAE framework to model the decoder.

Original: Speciﬁcally, the proposed model utilizes the variational auto-encoding framework to learn a (low-dimensional) vector of latent variables to capture the global information of an image, which is fed as a conditional input to a ﬂow-based invertible decoder with architecture borrowed from style transfer literature.
Revised: Speciﬁcally, the proposed model utilizes the variational auto-encoding framework to learn a (low-dimensional) vector of latent variables to capture the global information of an image, which is fed as a conditional input to a ﬂow-based invertible decoder with architecture borrowed from style transfer literature.

Original: Experimental results on standard image benchmarks demonstrate the effectiveness of our model in terms of density estimation, image generation and unsupervised representation learning.
Revised: Experimental results on standard image benchmarks demonstrate the effectiveness of our model in terms of density estimation, image generation and unsupervised representation learning.

Original: Importantly, this work demonstrates that with only architectural inductive biases, a generative model with a plain log-likelihood objective is capable of learning decoupled representations, requiring no explicit supervision.
Revised: Importantly, this work demonstrates that with only architectural inductive biases, a generative model with a likelihood-based objective is capable of learning decoupled representations, requiring no explicit supervision.

Original: The code for our model is available at anonymous link .
Revised: The code for our model is available at anonymous link .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Unsupervised learning of probabilistic models and meaningful representation learning are two central yet challenging problems in machine learning.
Revised: Unsupervised learning of probabilistic models and meaningful representation learning are two central yet challenging problems in machine learning.

Original: Formally, let X ∈ X be the random variables of the observed data, e.g., X is an image.
Revised: Formally, let X ∈ X be the random variables of the observed data, e.g., X is an image.

Original: One goal of generative models is to learn the parameter θ such that the model distribution P θ ( X ) can best approximate the true distribution P ( X ) .
Revised: One goal of generative models is to learn the parameter θ such that the model distribution P θ ( X ) can best approximate the true distribution P ( X ) .

Original: Throughout the paper, uppercase letters represent random variables and lowercase letters their realizations.
Revised: Throughout the paper, uppercase letters represent random variables and lowercase letters their realizations.

Original: Unsupervised (disentangled) representation learning, besides data distribution estimation and data generation, is also a principal component in generative models.
Revised: Unsupervised (disentangled) representation learning, besides data distribution estimation and data generation, is also a principal component in generative models.

Original: The goal is to identify and disentangle the underlying causal factors, to tease apart the underlying dependencies of the data, so that it becomes easier to understand, to classify, or to perform other tasks (Bengio et al., 2013).
Revised: The goal is to identify and disentangle the underlying causal factors, to tease apart the underlying dependencies of the data, so that it becomes easier to understand, to classify, or to perform other tasks (Bengio et al., 2013).

Original: Unsupervised representation learning has spawned signiﬁcant interests and a number of techniques (Chen et al., 2017a;
Revised: Unsupervised representation learning has spawned signiﬁcant interests and a number of techniques (Chen et al., 2017a;

Original: Devlin et al., 2019; Hjelm et al., 2019) has emerged over the years to address this challenge.
Revised: Devlin et al., 2019; Hjelm et al., 2019) has emerged over the years to address this challenge.

Original: Among these generative models, VAE (Kingma & Welling, 2014; Rezende et al., 2014) and Generative (Normalizing) Flows (Dinh et al., 2014) have stood out for their simplicity and effectiveness.
Revised: Among these generative models, VAE (Kingma & Welling, 2014; Rezende et al., 2014) and Generative (Normalizing) Flows (Dinh et al., 2014) have stood out for their simplicity and effectiveness.

Original: 1. V ARIATIONAL A UTO -E NCODERS (VAE S )
Revised: 1. V ARIATIONAL A UTO -E NCODERS (VAE S )

Original: VAE, as a member of latent variable models (LVMs), gains popularity for its capability of automatically learning meaningful (low-dimensional) representations from raw data.
Revised: VAE, as a member of latent variable models (LVMs), gains popularity for its capability of automatically learning meaningful (low-dimensional) representations from raw data.

Original: In the framework of
Revised: In the framework of

Original: VAEs, a set of latent variables Z ∈ Z are introduced, and the model distribution P θ ( X ) is deﬁned as the marginal of the joint distribution between X and Z :
Revised: VAEs, a set of latent variables Z ∈ Z are introduced, and the model distribution P θ ( X ) is deﬁned as the marginal of the joint distribution between X and Z :

Original: [Equation]
Revised: [Equation]

Original: where the joint distribution p θ ( x, z ) is factorized as the product of a prior p θ ( z ) over the latent Z , and the “generative” distribution p θ ( x | z ) .
Revised: where the joint distribution p θ ( x, z ) is factorized as the product of a prior p θ ( z ) over the latent Z , and the “generative” distribution p θ ( x | z ) .

Original: µ ( z ) is the base measure on the latent space Z .
Revised: µ ( z ) is the base measure on the latent space Z .

Original: In general, this marginal likelihood is intractable to compute or differentiate directly, and Variational
Revised: In general, this marginal likelihood is intractable to compute or differentiate directly, and Variational

Original: Inference (Wainwright et al., 2008) provides a solution to optimize the evidence lower bound (ELBO), an alternative objective by introducing a parametric inference model q φ ( z | x ) :
Revised: Inference (Wainwright et al., 2008) provides a solution to optimize the evidence lower bound (ELBO), an alternative objective by introducing a parametric inference model q φ ( z | x ) :

Original: [Equation]
Revised: [Equation]

Original: where ELBO could be seen as an autoencoding loss with q φ ( z | x ) being the encoder and p θ ( x | z ) being the decoder, with the ﬁrst term in the RHS in (
Revised: where ELBO could be seen as an autoencoding loss with q φ ( z | x ) being the encoder and p θ ( x | z ) being the decoder, with the ﬁrst term in the RHS in (2)

Original: [Figure]
Revised: [Figure]

Original: 1. G ENERATIVE F LOWS
Revised: 1. G ENERATIVE F LOWS

Original: Put simply, generative ﬂows (a.k.a., normalizing ﬂows) work by transforming a simple distribution,
Revised: Put simply, generative ﬂows (a.k.a., normalizing ﬂows) work by transforming a simple distribution,

Original: P (Υ) (e.g. a simple Gaussian) into a complex one (e.g. the complex distribution of data P ( X ) ) through a chain of invertible transformations.
Revised: P (Υ) (e.g. a simple Gaussian) into a complex one (e.g. the complex distribution of data P ( X ) ) through a chain of invertible transformations.

Original: Formally, a generative ﬂow deﬁnes a bijection function f : X → Υ (with g = f − 1 ), where υ ∈ Υ is a set of latent variables with simple prior distribution p Υ ( υ ) .
Revised: Formally, a generative ﬂow deﬁnes a bijection function f : X → Υ (with g = f − 1 ), where υ ∈ Υ is a set of latent variables with simple prior distribution p Υ ( υ ) .

Original: It provides us with an invertible transformation between X and Υ , whereby the generative process over X is deﬁned straightforwardly:
Revised: It provides us with an invertible transformation between X and Υ , whereby the generative process over X is deﬁned straightforwardly:

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: An important insight behind generative ﬂows is that given this bijection function, the change of the variable formula deﬁnes the model distribution on X by:
Revised: An important insight behind generative ﬂows is that given this bijection function, the change of the variable formula deﬁnes the model distribution on X by:

Original: [Equation]
Revised: [Equation]

Original: where ∂f θ ( x ) ∂x is the Jacobian of f θ at x .
Revised: where ∂f θ ( x ) ∂x is the Jacobian of f θ at x .

Original: A stacked sequence of such invertible transformations is called a generative (normalizing) ﬂow (Rezende & Mohamed, 2015):
Revised: A stacked sequence of such invertible transformations is called a generative (normalizing) ﬂow (Rezende & Mohamed, 2015):

Original: [Equation]
Revised: [Equation]

Original: where f = f 1 ◦ f 2 ◦ · · · ◦ f K is a ﬂow of K transformations (omitting θ for brevity).
Revised: where f = f 1 ◦ f 2 ◦ · · · ◦ f K is a ﬂow of K transformations (omitting θ for brevity).

Original: 1. P ROBLEMS OF VAE S AND G ENERATIVE F LOWS
Revised: 1. P ROBLEMS OF VAE S AND G ENERATIVE F LOWS

Original: Despite their impressive successes, VAEs and generative ﬂows still suffer their own problems.
Revised: Despite their impressive successes, VAEs and generative ﬂows still suffer their own problems.

Original: Posterior Collapse in VAEs As discussed in Bowman et al.
Revised: Posterior Collapse in VAEs As discussed in Bowman et al.

Original: (2015), without further assumptions, the ELBO objective in (2) may not guide the model towards the intended role for the latent variables Z , or even learn uninformative Z with the observation that the KL term KL( q φ ( Z | X )
Revised: (2015), without further assumptions, the ELBO objective in (2) may not guide the model towards the intended role for the latent variables Z , or even learn uninformative Z with the observation that the KL term KL( q φ ( Z | X )

Original: || p θ ( Z )) varnishes to zero.
Revised: || p θ ( Z )) varnishes to zero.

Original: The essential reason of this posterior collapse problem is that, under absolutely unsupervised setting, the marginal likelihood-based objective incorporates no (direct) supervision on the latent space to characterize the latent variable Z with preferred properties w.r.t.
Revised: The essential reason of this posterior collapse problem is that, under absolutely unsupervised setting, the marginal likelihood-based objective incorporates no (direct) supervision on the latent space to characterize the latent variable Z with preferred properties w.r.t.

Original: representation learning.
Revised: representation learning.

Original: Local Dependency in Generative Flows Generative ﬂows suffer from the limitation of expressiveness and local dependency.
Revised: Local Dependency in Generative Flows Generative ﬂows suffer from the limitation of expressiveness and local dependency.

Original: Most generative ﬂows tend to capture the dependency among features only locally, and are incapable of realistic synthesis of large images compared to GANs (Goodfellow et al., 2014).
Revised: Most generative ﬂows tend to capture the dependency among features only locally, and are incapable of realistic synthesis of large images compared to GANs (Goodfellow et al., 2014).

Original: Unlike latent variable models, e.g. VAEs, which represent the high-dimensional data as coordinates in a latent low-dimensional space, the long-term dependencies that usually describe the global features of the data can only be propagated through a composition of transformations.
Revised: Unlike latent variable models, e.g. VAEs, which represent the high-dimensional data as coordinates in a latent low-dimensional space, the long-term dependencies that usually describe the global features of the data can only be propagated through a composition of transformations.

Original: Previous studies attempted to enlarge the receptive ﬁeld by using a special design of parameterization like masked convolutions (Ma et al., 2019a) or attention mechanism (Ho et al., 2019).
Revised: Previous studies attempted to enlarge the receptive ﬁeld by using a special design of parameterization like masked convolutions (Ma et al., 2019a) or attention mechanism (Ho et al., 2019).

Original: In this paper, we propose a simple and effective generative model to simultaneously tackle the aforementioned challenges of VAEs and generative ﬂows by leveraging their properties to complement each other.
Revised: In this paper, we propose a simple and effective generative model to simultaneously tackle the aforementioned challenges of VAEs and generative ﬂows by leveraging their properties to complement each other.

Original: By embedding a generative ﬂow in the VAE framework to model the decoder, the proposed model is able to learn decoupled representations which capture global and local information of images respectively in an entirely unsupervised manner.
Revised: By embedding a generative ﬂow in the VAE framework to model the decoder, the proposed model is able to learn decoupled representations which capture global and local information of images respectively in an entirely unsupervised manner.

Original: The key insight is to utilize the inductive biases from the model architecture design — leveraging the VAE framework equipped with a compression encoder to extract the global information in a low-dimensional representation, and a ﬂow-based decoder which favors local dependencies to store the residual information into a local high-dimensional representation (§2).
Revised: The key insight is to utilize the inductive biases from the model architecture design — leveraging the VAE framework equipped with a compression encoder to extract the global information in a low-dimensional representation, and a ﬂow-based decoder which favors local dependencies to store the residual information into a local high-dimensional representation (§2).

Original: Experimentally, on four benchmark datasets for images, we demonstrate the effectiveness of our model on two aspects: (i) density estimation and image generation, by consistently achieving signiﬁcant improvements over Glow (Kingma & Dhariwal, 2018), (ii) decoupled representation learning, by performing classiﬁcation on learned representations the switch operation (see examples in Figure 1).
Revised: Experimentally, on four benchmark datasets for images, we demonstrate the effectiveness of our model on two aspects: (i) density estimation and image generation, by consistently achieving signiﬁcant improvements over Glow (Kingma & Dhariwal, 2018), (ii) decoupled representation learning, by performing classiﬁcation on learned representations the switch operation (see examples in Figure 1).

Original: Perhaps most strikingly, we demonstrate the feasibility of decoupled representation learning via the plain likelihood-based generation, using only architectural inductive biases (§3).
Revised: Perhaps most strikingly, we demonstrate the feasibility of decoupled representation learning via the plain likelihood-based generation, using only architectural inductive biases (§3).

Original:  G ENERATIVE M ODEL FOR D ECOUPLED R EPRESENTATION L EARNING
Revised:  G ENERATIVE M ODEL FOR D ECOUPLED R EPRESENTATION L EARNING

Original: We ﬁrst illustrate the high-level insights of the architecture design of our generative model (shown in
Revised: We ﬁrst illustrate the high-level insights of the architecture design of our generative model (shown in

Original: Figure 2) before detailing each component in the following sections.
Revised: Figure 2) before detailing each component in the following sections.

Original: In the training process of our generative model, we  ﬁrst feed the input image x into the encoder q φ ( z | x )in the VAE framework to compute the latent variable z .
Revised: In the training process of our generative model, we optimize ELBO (RHS in (2)) in the VAE framework . Speciﬁcally, we ﬁrst feed the input image x into the encoder q φ ( z | x ) to compute the latent variable z .

Original: The encoder is designed to be a compression network, which compresses the high-dimensional image into a low-dimensional vector (§2.1).
Revised: The encoder is designed to be a compression network, which compresses the high-dimensional image into a low-dimensional vector (§2.2).

Original: Through this compression process, the local information of an image x is enforced to be discarded, yielding representation z that captures the global information.
Revised: Through this compression process, the local information of an image x is enforced to be discarded, yielding representation z that captures the global information.

Original: Then we feed z as a conditional input to a ﬂow-based decoder, which transforms x into the representation υ with the same dimension (§2.2).
Revised: Then we feed z as a conditional input to a ﬂow-based decoder, which transforms x into the representation υ with the same dimension (§2.3).

Original: Since the decoder is invertible, with z and υ , we can exactly reconstruct the original image x .
Revised: Since the decoder is invertible, with z and υ , we can exactly reconstruct the original image x .

Original: It indicates that z and υ maintain all the information of x , and the reconstruction process can be regarded as an additional operation — adding z and υ to recover x .
Revised: It indicates that z and υ maintain all the information of x , and the reconstruction process can be regarded as an additional operation — adding z and υ to recover x .

Original: In this way, we expect that the local information discarded in the compression process will be restored in υ .
Revised: In this way, we expect that the local information discarded in the compression process will be restored in υ .

Revised: In the generative process, we combine the sampling procedures of VAEs and generative ﬂows: we ﬁrst sample a value of z from the prior distribution p ( z ) , and a value of υ from p Υ

Revised: ( υ )

Revised: ; second, we input z and υ into the invertible function f − 1 modeled by the generative ﬂow decoder to generate an image x = f − 1 θ ( υ, z ) =

Revised: g θ

Revised: ( υ, z ) .

Original: 2. C OMPRESSION E NCODER
Revised: 2. C OMPRESSION E NCODER

Original: Following previous work, the variational posterior distribution q φ ( z | x ) , a.k.a encoder, models the latent variable Z as a diagonal Gaussian with learned mean and variance:
Revised: Following previous work, the variational posterior distribution q φ ( z | x ) , a.k.a encoder, models the latent variable Z as a diagonal Gaussian with learned mean and variance:

Original: [Equation]
Revised: [Equation]

Original: where µ ( · ) and σ ( · ) are neural networks.
Revised: where µ ( · ) and σ ( · ) are neural networks.

Original: In the context of 2D images where x is a tensor of shape [ h × w × c ] with spatial dimensions ( h, w ) and channel dimension c , the compression encoder maps each image x to a d z -dimensional vector.
Revised: In the context of 2D images where x is a tensor of shape [ h × w × c ] with spatial dimensions ( h, w ) and channel dimension c , the compression encoder maps each image x to a d z -dimensional vector.

Original: d z is the dimension of the latent space.
Revised: d z is the dimension of the latent space.

Original: In this work, the motivation of the encoder is to compress the high-dimensional data x to lowdimensional latent variable z , i.e. h × w × c (cid:29) d z , to enforce the latent representation z to capture the global features of x .

Original: Furthermore, unlike previous studies on VAE based generative models for natural images (Kingma et al., 2016; Chen et al., 2017a; Ma et al., 2019b) that represented latent codes z as low-resolution feature maps 1 , we represent z as an unstructured 1-dimensional vector to erase the local spatial dependencies.

Original: Concretely, we implement the encoder with a similar architecture in ResNet (He et al., 2016).

Original: The spatial downsampling is implemented by a 2-strided ResNet block

Original: [Figure]
Revised: [Figure]

Original: Local Representation υ low
Revised: Global Representation z

Original: [Figure]

Original: dimensional discarding local information
Revised: Local Representation υ low-dimensional discarding local information

Original: Same size with X
Revised: Same size with X

Original: [Figure]
Revised: [Figure]

Original: Flow-based Decoder p θ ( x | z ) 
Revised: Flow-based Decoder p θ ( x | z ) x

Original: [Figure]
Revised: = f −1 θ ( υ ; z ) υ = f θ ( x ; z ) In this work, the motivation of the encoder is to compress the high-dimensional data x to lowdimensional latent variable z , i.e. h × w × c (cid:29) d z , to enforce the latent representation z to capture the global features of x . Furthermore, unlike previous studies on VAE based generative models for natural images (Kingma et al., 2016; Chen et al., 2017a; Ma et al., 2019b) that represented latent codes z as low-resolution feature maps 1 , we represent z as an unstructured 1-dimensional vector to erase the local spatial dependencies.

Original: ( υ ; z ) υ = f θ ( x ; z ) with 3 × 3 ﬁlters.
Revised: Concretely, we implement the encoder with a similar architecture in ResNet (He et al., 2016). The spatial downsampling is implemented by a 2-strided ResNet block with 3 × 3 ﬁlters.

Original: On top of these ResNet blocks, there is one more fully-connected layer with number of output units equal to d z × 2 to generate µ ( x ) and log σ 2 ( x ) (details in Appendix B).
Revised: On top of these ResNet blocks, there is one more fully-connected layer with number of output units equal to d z × 2 to generate µ ( x ) and log σ 2 ( x ) (details in Appendix B).

Original: Zero initialization.
Revised: Zero initialization.

Original: Following Ma et al. (2019c), we initialize the weights of the last fullyconnected layer that generates the µ and log σ 2 values with zeros.
Revised: Following Ma et al. (2019c), we initialize the weights of the last fullyconnected layer that generates the µ and log σ 2 values with zeros.

Original: This ensures that the posterior distribution is initialized as a simple normal distribution, which has been demonstrated helpful for training very deep neural networks more stably in the framework of VAEs.
Revised: This ensures that the posterior distribution is initialized as a simple normal distribution, which has been demonstrated helpful for training very deep neural networks more stably in the framework of VAEs.

Original: 2. I NVERTIBLE D ECODER BASED ON G ENERATIVE F LOW
Revised: 2. I NVERTIBLE D ECODER BASED ON G ENERATIVE F LOW

Original: The ﬂow-based decoder deﬁnes a (conditionally) invertible function υ = f θ ( x ; z ) , where υ follows a standard normal distribution υ ∼ N (0 , I ) .
Revised: The ﬂow-based decoder deﬁnes a (conditionally) invertible function υ = f θ ( x ; z ) , where υ follows a standard normal distribution υ ∼ N (0 , I ) .

Original: Conditioned on the latent variable z output from the encoder, we can reconstruct x with the inverse function x = f − 1 θ ( υ ; z ) .
Revised: Conditioned on the latent variable z output from the encoder, we can reconstruct x with the inverse function x = f − 1 θ ( υ ; z ) .

Original: The ﬂow-based decoder adopts the main backbone architecture of Glow (Kingma & Dhariwal, 2018), where each step of ﬂow consists of the same three types of elementary ﬂows — actnorm, invertible 1 × 1 convolution and coupling (details in Appendix A).
Revised: The ﬂow-based decoder adopts the main backbone architecture of Glow (Kingma & Dhariwal, 2018), where each step of ﬂow consists of the same three types of elementary ﬂows — actnorm, invertible 1 × 1 convolution and coupling (details in Appendix A).

Original: Conditional Inputs in Afﬁne Coupling Layers.
Revised: Conditional Inputs in Afﬁne Coupling Layers.

Original: To incorporate z as a conditional input to the decoder, we modify the neural networks for the scale and bias terms:
Revised: To incorporate z as a conditional input to the decoder, we modify the neural networks for the scale and bias terms:

Original: [Equation]
Revised: [Equation]

Original: where s() and b() take both x a and z as input.
Revised: where s() and b() take both x a and z as input.

Original: Speciﬁcally, each coupling layer includes three convolution layers where the ﬁrst and last convolutions are 3 × 3 , while the center convolution is 1 × 1 .
Revised: Speciﬁcally, each coupling layer includes three convolution layers where the ﬁrst and last convolutions are 3 × 3 , while the center convolution is 1 × 1 .

Original: ELU (Clevert et al., 2015) is used as the activation function throughout the ﬂow architecture:
Revised: ELU (Clevert et al., 2015) is used as the activation function throughout the ﬂow architecture:

Original: [Equation]
Revised: [Equation]

Original: where FC() refers to a linear full-connected layer and ⊕ is addition operation per channel between a 2D image and a 1D vector.
Revised: where FC() refers to a linear full-connected layer and ⊕ is addition operation per channel between a 2D image and a 1D vector.

Original: Importantly, z is fed as conditional input to every coupling layers, unlike previous work (Agrawal & Dukkipati, 2016; Morrow & Chiu, 2019) where z is only used to learn the mean and variance of the underlying Gaussian of υ .

Original: This design is inspired by the generator in Style-GAN (Karras et al., 2019), where the style-vector is added to each block of the generator.

Original: We conduct experiments to show the importance of this architectural design (see §3.1).

Original: Reﬁned Architecture of Glow.

Original: In this work, we reﬁne the organization of these three elementary ﬂows in one step (see Figure 3a) to reduce the total number of invertible 1 × 1 convolution ﬂows.

Original: Coupling layer (B)
Revised: Coupling layer (B)

Original: Coupling layer (A)
Revised: Coupling layer (A)

Original: Actnorm
Revised: Actnorm

Original: Coupling layer (D)
Revised: Coupling layer (D)

Original: Coupling layer (C)
Revised: Coupling layer (C)

Original: Actnorm
Revised: Actnorm

Original: Invertible 1x1 conv split step squeeze  /  ×  ×(  − 1)
Revised: Invertible 1x1 conv split step squeeze  /  ×  ×(  − 1)

Original:  ×  /2 split A split B split C split D (a) One step of our ﬂow
Revised:  ×  /2 split A split B split C split D (a) One step of our ﬂow

Original: Coupling layer (B)
Revised: Coupling layer (B)

Original: Coupling layer (A)
Revised: Coupling layer (A)

Original: Actnorm
Revised: Actnorm

Original: Coupling layer (D)
Revised: Coupling layer (D)

Original: Coupling layer (C)
Revised: Coupling layer (C)

Original: Actnorm
Revised: Actnorm

Original: Invertible 1x1 conv split step squeeze  /  ×  ×(  − 1)
Revised: Invertible 1x1 conv split step squeeze  /  ×  ×(  − 1)

Original:  ×  /2 split A split B split C split D (b) Four types of coupling layer splits
Revised:  ×  /2 split A split B split C split D (b) Four types of coupling layer splits

Original: Coupling layer (B)
Revised: Coupling layer (B)

Original: Coupling layer (A)
Revised: Coupling layer (A)

Original: Actnorm
Revised: Actnorm

Original: Coupling layer (D)
Revised: Coupling layer (D)

Original: Coupling layer (C)
Revised: Coupling layer (C)

Original: Actnorm
Revised: Actnorm

Original: Invertible 1x1 conv split step squeeze  /  ×  ×(  −1)
Revised: Invertible 1x1 conv split step squeeze  /  ×  ×(  −1)

Original:  ×  /2 split A split B split C split D (c) Fine-grained multi-scale architecture
Revised:  ×  /2 split A split B split C split D (c) Fine-grained multi-scale architecture

Revised: Importantly, z is fed as conditional input to every coupling layers, unlike previous work (Agrawal & Dukkipati, 2016; Morrow & Chiu, 2019) where z is only used to learn the mean and variance of the underlying Gaussian of υ .

Revised: This design is inspired by the generator in Style-GAN (Karras et al., 2019), where the style-vector is added to each block of the generator.

Revised: We conduct experiments to show the importance of this architectural design (see §3.1).

Revised: Reﬁned Architecture of Glow.

Revised: In this work, we reﬁne the organization of these three elementary ﬂows in one step (see Figure 3a) to reduce the total number of invertible 1 × 1 convolution ﬂows.

Original: The reason is that the cost and the numerical stability of computing or differentiating the determinant of the weight matrix becomes the practical bottleneck when the channel dimension c is considerably large for high-resolution images.
Revised: The reason is that the cost and the numerical stability of computing or differentiating the determinant of the weight matrix becomes the practical bottleneck when the channel dimension c is considerably large for high-resolution images.

Original: To reduce the number of invertible 1 × 1 convolution ﬂows while maintaining the permutation effect along the channel dimension, we use four split patterns for the split() function in (6) (see Figure 3b).
Revised: To reduce the number of invertible 1 × 1 convolution ﬂows while maintaining the permutation effect along the channel dimension, we use four split patterns for the split() function in (6) (see Figure 3b).

Original: The splits perform on the channel dimension with continuous and alternate patterns, respectively.
Revised: The splits perform on the channel dimension with continuous and alternate patterns, respectively.

Original: For each pattern of the split, we alternate x a and x b .
Revised: For each pattern of the split, we alternate x a and x b .

Original: Coupling layers with different split types alternate in one step of our ﬂow, as illustrated in Figure 3a.
Revised: Coupling layers with different split types alternate in one step of our ﬂow, as illustrated in Figure 3a.

Original: We further replace the original multi-scale architecture with the ﬁne-grained multi-scale architecture (Figure 3c) proposed in Ma et al. (2019a), with the same value of M = 4 .
Revised: We further replace the original multi-scale architecture with the ﬁne-grained multi-scale architecture (Figure 3c) proposed in Ma et al. (2019a), with the same value of M = 4 .

Original: Experimental improvements over
Revised: Experimental improvements over

Original: Glow demonstrate the effectiveness of our reﬁned architecture (§3.1).
Revised: Glow demonstrate the effectiveness of our reﬁned architecture (§3.1).

Original: 2. D ISCUSSION
Revised: 2. T ACKLING THE T WO P ORBLEMS IN VAE S AND G ENERATIVE F LOWS

Revised: Resolving Local Dependency in Generative Flows with Global Information from z . As discussed in §1.3, the ﬂow-based decoder suffers the limitation of expressiveness and local dependency.

Original: From the high-level view of the VAE encoder, the (indirect) supervision of learning global latent representation z comes from two sources of architectural inductive bias.
Revised: In the VAE framework of our model, the latent codes z provides the decoder with the imperative global information, which is essential to resolve the limitation of expressiveness due to local dependency.

Original: First, the compression architecture, which takes a high-dimensional image as input and outputs a low-dimensional vector, encourages the encoder to discard local dependencies of the image.
Revised: On the other hand, the ﬂow-based decoder favors to store local dependencies, encouraging the encoder to extract global information that is complementary to it. Resolving Posterior Collapse in VAEs with Flow-based Decoders. As discussed in previous work (Chen et al., 2017a), one possible reason for posterior collapse in VAEs is that the decoder model is sufﬁciently expressive such that it completely ignores latent variables z , and a solution to posterior collapse is to limit the capacity of the decoder. This suggests generative ﬂow an ideal model for the decoder since they are insufﬁciently powerful to trigger the posterior collapse problem.

Original: Second, the preference of the ﬂow-based decoder for capturing local dependencies reinforces global information modeling of the encoder, since the all the  information of the input imagex needs to be preserved by  z and υ .
Revised: Architectural Inductive Biases for Decoupled Representation Learning. From the high-level view of our model, we utilize these complementary properties of the architectures of the encoder and decoder as inductive bias to attempt to decouple the global and local information of an image by storing them in separate representations. The (indirect) supervision of learning global latent representation z comes from two sources of architectural inductive bias. First, the compression

Original: From the perspective of the ﬂow-based decoder, the latent codes z provides the  decoder with the imperative global information, which is essential to resolve the limitation of expressiveness due to local dependency. In this work, we utilize these complementary properties of the architectures of the encoder and decoder as inductive bias to attempt to decouple theglobal and local information of an image  by storing them in separate representations.
Revised: [Table] architecture, which takes a high-dimensional image as input and outputs a low-dimensional vector, encourages the encoder to discard local dependencies of the image. Second, the preference of the ﬂow-based decoder for capturing local dependencies reinforces global information modeling of the encoder, since the all the information of the input image x needs to be preserved by z and υ .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To evaluate our generative model, we conduct two groups of experiments on four benchmark datasets that are commonly used to evaluate deep generative models: CIFAR-10 (Krizhevsky & Hinton, 2009), 64 × 64 downsampled version ImageNet (Oord et al., 2016), the bedroom category in LSUN (Yu et al., 2015) and the CelebA-HQ dataset (Karras et al., 2018) 2 .
Revised: To evaluate our generative model, we conduct two groups of experiments on four benchmark datasets that are commonly used to evaluate deep generative models: CIFAR-10 (Krizhevsky & Hinton, 2009), 64 × 64 downsampled version ImageNet (Oord et al., 2016), the bedroom category in LSUN (Yu et al., 2015) and the CelebA-HQ dataset (Karras et al., 2018) 2 .

Original: Unlike previous studies which performed experiments on 5-bit images from the LSUN and CelebA-HQ datasets, all the samples from the four datasets are 8-bit images in our experiments.
Revised: Unlike previous studies which performed experiments on 5-bit images from the LSUN and CelebA-HQ datasets, all the samples from the four datasets are 8-bit images in our experiments.

Original: All the models are trained by using afﬁne coupling layers and uniform dequantization (Uria et al., 2013).
Revised: All the models are trained by using afﬁne coupling layers and uniform dequantization (Uria et al., 2013).

Original: Additional details on datasets, model architectures, and results of the conducted experiments are provided in Appendix C.[Table]
Revised: Additional details on datasets, model architectures, and results of the conducted experiments are provided in Appendix C.

Original:  G ENERATIVE M ODELING
Revised: 3. G ENERATIVE M ODELING

Original: We begin our experiments with an evaluation on the performance of generative modeling, leaving the experiments of evaluating the quality of the decoupled global and local representations to §3.2.
Revised: We begin our experiments with an evaluation on the performance of generative modeling, leaving the experiments of evaluating the quality of the decoupled global and local representations to §3.2.

Original: The baseline model we compare with is the reﬁned Glow model, which is the exact architecture used in our ﬂow-based decoder, except the conditional input z .
Revised: The baseline model we compare with is the reﬁned Glow model, which is the exact architecture used in our ﬂow-based decoder, except the conditional input z .

Original: Thus, the comparison with this baseline illustrates the effect of the decoupled representations on image generation.
Revised: Thus, the comparison with this baseline illustrates the effect of the decoupled representations on image generation.

Original: For the reﬁned Glow model, we adjust the number of steps in each level so that there are similar numbers of coupling layers and parameters with the original Glow model for a fair comparison.
Revised: For the reﬁned Glow model, we adjust the number of steps in each level so that there are similar numbers of coupling layers and parameters with the original Glow model for a fair comparison.

Original: Density Estimation.
Revised: Density Estimation.

Original: Table 1 provides the negative log-likelihood scores in bits/dim (BPD) on the four benchmark datasets, along with the top-performing autoregressive models and ﬂow-based generative models.
Revised: Table 1 provides the negative log-likelihood scores in bits/dim (BPD) on the four benchmark datasets, along with the top-performing autoregressive models and ﬂow-based generative models.

Original: For a comprehensive comparison, we report results on 5-bit images from the
Revised: For a comprehensive comparison, we report results on 5-bit images from the

Original: LSUN and CelebA-HQ datasets with additive coupling layers.
Revised: LSUN and CelebA-HQ datasets with additive coupling layers.

Original: Our reﬁned Glow model obtains better performance than the original one in Kingma & Dhariwal (2018), demonstrating the effectiveness of the reﬁned architecture.
Revised: Our reﬁned Glow model obtains better performance than the original one in Kingma & Dhariwal (2018), demonstrating the effectiveness of the reﬁned architecture.

Original: The proposed generative model achieves state-of-the-art BPD on all the four standard benchmarks in the non-autoregressive category, except the 5-bit CelebA-HQ dataset.
Revised: The proposed generative model achieves state-of-the-art BPD on all the four standard benchmarks in the non-autoregressive category, except the 5-bit CelebA-HQ dataset.

Original: Sample Quality For quantitative evaluation of sample quality, we report the Fréchet Inception
Revised: Sample Quality For quantitative evaluation of sample quality, we report the Fréchet Inception

Original: Distance (FID) (Heusel et al., 2017) on CIFAR-10 in Table 2.
Revised: Distance (FID) (Heusel et al., 2017) on CIFAR-10 in Table 2.

Original: Results marked with † and ‡ are taken  from † Ostrovski et al.
Revised: Results marked with † and ‡ are taken [Table] from † Ostrovski et al.

Original: and ‡ Heusel et al.
Revised: and ‡ Heusel et al.

Original: (2017), respectively.
Revised: (2017), respectively.

Original: Table 2 also provides scores of two energy-based models, EBM (Du & Mordatch, 2019) and NCSN (Song & Ermon, 2019).
Revised: Table 2 also provides scores of two energy-based models, EBM (Du & Mordatch, 2019) and NCSN (Song & Ermon, 2019).

Original: We see that our model obtains better FID scores than all the other explicit density models.
Revised: We see that our model obtains better FID scores than all the other explicit density models.

Original: In particular, the improvement over the reﬁned Glow model on FID score demonstrates that learning decoupled representations is also helpful for realistic image synthesis.
Revised: In particular, the improvement over the reﬁned Glow model on FID score demonstrates that learning decoupled representations is also helpful for realistic image synthesis.

Original: Qualitatively, Figure 4 showcases some random samples for 8-bit CelebA-HQ 256 × 256 at temperature 0.7.
Revised: Qualitatively, Figure 4 showcases some random samples for 8-bit CelebA-HQ 256 × 256 at temperature 0.7.

Original: More image samples, including samples on other datasets, are provided in Appendix F.
Revised: More image samples, including samples on other datasets, are provided in Appendix F.

Original: [Table]
Revised: [Table]

Original: Ours 3.

Original: Effect of feeding z to every coupling layer.
Revised: Effect of feeding z to every coupling layer.

Original: As mentioned in §2.2, we feed latent codes z to every coupling layer in the ﬂow-based decoder.
Revised: As mentioned in §2.3, we feed latent codes z to every coupling layer in the ﬂow-based decoder.

Original: To investigate the importance of this design, we perform experiments on CIFAR-10 to compare our model with the baseline model where z is only used in the underlying Gaussian of υ (Agrawal & Dukkipati, 2016; Morrow & Chiu, 2019).
Revised: To investigate the importance of this design, we perform experiments on CIFAR-10 to compare our model with the baseline model where z is only used in the underlying Gaussian of υ (Agrawal & Dukkipati, 2016; Morrow & Chiu, 2019).

Original: Table 3 gives the performance on BPD and FID score.
Revised: Table 3 gives the performance on BPD and FID score.

Original: Our model outperforms the baseline on both the two metrics, demonstrating the effectiveness of this design in our decoder.[Table]
Revised: Our model outperforms the baseline on both the two metrics, demonstrating the effectiveness of this design in our decoder.

Original: 3. D ECOUPLED R EPRESENTATION L EARNING
Revised: 3. D ECOUPLED R EPRESENTATION L EARNING

Original: Image Classiﬁcation As discussed above, good latent representation
Revised: Image Classiﬁcation As discussed above, good latent representation

Original: z need to capture global features that characterize the entire image, and disentangle the underlying causal factors.
Revised: z need to capture global features that characterize the entire image, and disentangle the underlying causal factors.

Original: From this perspective, we follow the widely adopted downstream linear evaluation protocol (Oord et al., 2018;
Revised: From this perspective, we follow the widely adopted downstream linear evaluation protocol (Oord et al., 2018;

Original: Hjelm et al., 2019) to train a linear classiﬁer for image classiﬁcation on the learned representations using all available training labels.
Revised: Hjelm et al., 2019) to train a linear classiﬁer for image classiﬁcation on the learned representations using all available training labels.

Original: The classiﬁcation accuracy is a measure of the linear separability, which is commonly used as a proxy for disentanglement and mutual information between representations and class labels.
Revised: The classiﬁcation accuracy is a measure of the linear separability, which is commonly used as a proxy for disentanglement and mutual information between representations and class labels.

Original: We perform linear classiﬁcation on CIFAR-10 using a support vector machine (SVM).
Revised: We perform linear classiﬁcation on CIFAR-10 using a support vector machine (SVM).

Original: Table 4 lists the classiﬁcation accuracy of SVM on the representations of z and υ , together with AAE (Makhzani et al., 2015), VAE (Kingma & Welling, 2014), BiGAN (Donahue et al., 2017) and Deep InfoMax (Hjelm et al., 2019).
Revised: Table 4 lists the classiﬁcation accuracy of SVM on the representations of z and υ , together with AAE (Makhzani et al., 2015), VAE (Kingma & Welling, 2014), BiGAN (Donahue et al., 2017) and Deep InfoMax (Hjelm et al., 2019).

Original: Results marked with † are token from Hjelm et al.
Revised: Results marked with † are token from Hjelm et al.

Original: Raw pixel is the baseline that directly training a classiﬁer on the raw pixels of an image.
Revised: Raw pixel is the baseline that directly training a classiﬁer on the raw pixels of an image.

Original: The classiﬁcation accuracy on the representation z is signiﬁcantly better than that on υ , indicating that z captures more global information, while υ captures more local dependencies.
Revised: The classiﬁcation accuracy on the representation z is signiﬁcantly better than that on υ , indicating that z captures more global information, while υ captures more local dependencies.

Original: Moreover, the accuracy of z outperforms Deep InfoMax, which is one of the state-of-the-art unsupervised representation learning methods via mutual information maximization.
Revised: Moreover, the accuracy of z outperforms Deep InfoMax, which is one of the state-of-the-art unsupervised representation learning methods via mutual information maximization.

Original:  Our generative model leads to the two-dimensional interpolation , where we linearly interpolate the two latent spaces z and υ between two real images:
Revised: Two-dimensional Interpolation Our generative model leads to the two-dimensional interpolation , where we linearly interpolate the two latent spaces z and υ between two real images:

Original: [Equation]
Revised: [Equation]

Original: where α, β ∈ [0 , 1] .
Revised: where α, β ∈ [0 , 1] .

Original: z 1 , υ 1 and z 2 , υ 2 are the global and local representations of images x 1 and x 2 , respectively.
Revised: z 1 , υ 1 and z 2 , υ 2 are the global and local representations of images x 1 and x 2 , respectively.

Original: Figure 5 shows one interpolation example from CelebA-HQ, where the images on theleft top and right bottom corners are the real images 3 .
Revised: Figure 5 shows one interpolation example from CelebA-HQ, where the images on the

Original: The switch operation is two special cases of the two-dimensional interpolation with ( α = 1 , β = 0) and ( α = 0 , β = 1) . More examples of interpolation and switch operation are provided in Appendix D.
Revised: [Table]

Original: Discussion
Revised: Discussion

Original: The results on image classiﬁcation and two-dimensional interpolation empirical demonstrate that our model relive the posterior collapse problem in VAEs, by learning meaningful information in the latent space.
Revised: The results on image classiﬁcation and two-dimensional interpolation empirical demonstrate that our model relive the posterior collapse problem in VAEs, by learning meaningful information in the latent space.

Original: The reason is that we use generative ﬂow as the decoder in the VAE framework.

Original: As discussed in previous work (Chen et al., 2017a), A potential solution to posterior collapse is to limit the capacity of the decoder.

Original: As generative ﬂows suffer from the limitation of expressiveness and local dependency § 1 . 3 , they are insufﬁciently powerfull to reconstruct the input image without encoding useful information in the latent variable.

Original: Due to the entirely unsupervised setting, however, there are no intuitive explanations or theoretical guarantees for what information is captured by the global and local representations, respectively.
Revised: Due to the entirely unsupervised setting, however, there are no intuitive explanations or theoretical guarantees for what information is captured by the global and local representations, respectively.

Original: It is an interesting direction for future work to explore how to learn more interpretable representations by investigating connections to architectural inductive biases, or leveraging weak or distant supervision.[Table]
Revised: It is an interesting direction for future work to explore how to learn more interpretable representations by investigating connections to architectural inductive biases, or leveraging weak or distant supervision.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Combination of VAEs and Generative Flows.
Revised: Combination of VAEs and Generative Flows.

Original: In the literature of combining VAEs and generative ﬂows, one direction of research is to use generative ﬂows as an inference machine in variational inference for continuous latent variable models (Kingma et al., 2016; Van Den Berg et al., 2018).
Revised: In the literature of combining VAEs and generative ﬂows, one direction of research is to use generative ﬂows as an inference machine in variational inference for continuous latent variable models (Kingma et al., 2016; Van Den Berg et al., 2018).

Original: Another direction is to incorporate generative ﬂows in the VAE framework as a trainable component, such as the prior (Chen et al., 2017a) or the decoder (Agrawal & Dukkipati, 2016; Morrow & Chiu, 2019; Mahajan et al., 2019).
Revised: Another direction is to incorporate generative ﬂows in the VAE framework as a trainable component, such as the prior (Chen et al., 2017a) or the decoder (Agrawal & Dukkipati, 2016; Morrow & Chiu, 2019; Mahajan et al., 2019).

Original: Recently, two contemporaneous work (Huang et al., 2020; Chen et al., 2020) explore the idea of constructing an invertible ﬂow-based model on an augmented input space by augmenting the original data with an additional random variable.
Revised: Recently, two contemporaneous work (Huang et al., 2020; Chen et al., 2020) explore the idea of constructing an invertible ﬂow-based model on an augmented input space by augmenting the original data with an additional random variable.

Original: The main difference between these work and ours is the purpose of introducing the latent variables and using generative ﬂows.
Revised: The main difference between these work and ours is the purpose of introducing the latent variables and using generative ﬂows.

Original: In
Revised: In

Original: Huang et al.
Revised: Huang et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2020), the latent variables are utilized to augment the input with extra dimensions to improve the expressivenss of the bijective mapping in generative ﬂows.
Revised: (2020), the latent variables are utilized to augment the input with extra dimensions to improve the expressivenss of the bijective mapping in generative ﬂows.

Original: Our generative model, on the other hand, aims to learn representations with decoupled information, and the design of the latent variables and the ﬂow-based decoder architecture is to accomplish this goal.
Revised: Our generative model, on the other hand, aims to learn representations with decoupled information, and the design of the latent variables and the ﬂow-based decoder architecture is to accomplish this goal.

Revised: The proposed generative model is closely related with generative ﬂows with picecewise invertible transformations, such as RAD (Dinh et al., 2019) and CIFs (Cornish et al., 2020), and can be regarded as inﬁnite of ﬂows (Papamakarios et al., 2019).

Original: Disentangled Representation Learning.
Revised: Disentangled Representation Learning.

Original: Disentanglement learning (Bengio et al., 2013; Mathieu et al., 2016) recently becomes a popular topic in representation learning.
Revised: Disentanglement learning (Bengio et al., 2013; Mathieu et al., 2016) recently becomes a popular topic in representation learning.

Original: Creating representations where each dimension is independent and corresponds to a particular attribute have been explored in several approaches, including VAE variants (Alemi et al., 2017; Higgins et al., 2017; Kim & Mnih, 2018;
Revised: Creating representations where each dimension is independent and corresponds to a particular attribute have been explored in several approaches, including VAE variants (Alemi et al., 2017; Higgins et al., 2017; Kim & Mnih, 2018;

Original: Chen et al., 2018; Mathieu et al., 2019), adversarial training (Mathieu et al., 2016;
Revised: Chen et al., 2018; Mathieu et al., 2019), adversarial training (Mathieu et al., 2016;

Original: Karras et al., 2019) and mutual information maximization/regularization (Chen et al., 2016; Hjelm et al., 2019;
Revised: Karras et al., 2019) and mutual information maximization/regularization (Chen et al., 2016; Hjelm et al., 2019;

Original: Sanchez et al., 2019).
Revised: Sanchez et al., 2019).

Original: Of particular relevance to this work are approaches that explore disentanglement in the context of VAEs, which aim to achieve independence or generalized decomposition (Mathieu et al., 2016) of the latent space.
Revised: Of particular relevance to this work are approaches that explore disentanglement in the context of VAEs, which aim to achieve independence or generalized decomposition (Mathieu et al., 2016) of the latent space.

Original: Different from these work which attempted to learn factorial representations for disentanglement, we aim to learn two separate representations to decouple the global and local information.
Revised: Different from these work which attempted to learn factorial representations for disentanglement, we aim to learn two separate representations to decouple the global and local information.

Original: Neural Style Transfer.
Revised: Neural Style Transfer.

Original: From the visualization, the switch operation of our model is also (empirically) related to neural style transfer of natural images (Gatys et al., 2015; Johnson et al., 2016; Jing et al., 2019) — the global and local representations in our model appear to correspond to the style and content representations in neural style transfer models.
Revised: From the visualization, the switch operation of our model is also (empirically) related to neural style transfer of natural images (Gatys et al., 2015; Johnson et al., 2016; Jing et al., 2019) — the global and local representations in our model appear to correspond to the style and content representations in neural style transfer models.

Original: The main difference is that the global and local representations in our model are learned in unsupervised manner, while the content and style representations in neural style transfer models are usually extracted from a pre-trained classiﬁcation network (VGG network) (Simonyan & Zisserman, 2014).
Revised: The main difference is that the global and local representations in our model are learned in unsupervised manner, while the content and style representations in neural style transfer models are usually extracted from a pre-trained classiﬁcation network (VGG network) (Simonyan & Zisserman, 2014).

Original: It is an interesting direction of future work to further investigate the relation between our learned decoupled representations and the content and style representations in neural style transfer models.
Revised: It is an interesting direction of future work to further investigate the relation between our learned decoupled representations and the content and style representations in neural style transfer models.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose a simple and effective generative model that embeds a generative ﬂow as decoder in the VAE framework.
Revised: In this paper, we propose a simple and effective generative model that embeds a generative ﬂow as decoder in the VAE framework.

Original: Simple as it appears to be, our model is capable of automatically decoupling global and local representations of images in an entirely unsupervised setting.
Revised: Simple as it appears to be, our model is capable of automatically decoupling global and local representations of images in an entirely unsupervised setting.

Original: Experimental results on standard image benchmarks demonstrate the effectiveness of our model on generative modeling and representation learning.
Revised: Experimental results on standard image benchmarks demonstrate the effectiveness of our model on generative modeling and representation learning.

Original: Importantly, we demonstrate the feasibility of decoupled representation learning via the plain likelihood-based generation, using only architectural inductive biases.
Revised: Importantly, we demonstrate the feasibility of decoupled representation learning via the plain likelihood-based generation, using only architectural inductive biases.

Original: Moreover, the two-dimensional interpolation supported by our model, with the switch operation as a special case, is an important step towards controllable image manipulation.
Revised: Moreover, the two-dimensional interpolation supported by our model, with the switch operation as a special case, is an important step towards controllable image manipulation.

Original: A PPENDIX : D ECOUPLING G LOBAL AND L OCAL R EPRESENTATIONS VIA I NVERTIBLE G ENERATIVE F LOWS A P RELIMINARY I NTRODUCTION OF G LOW
Revised: A PPENDIX : D ECOUPLING G LOBAL AND L OCAL R EPRESENTATIONS VIA I NVERTIBLE G ENERATIVE F LOWS A P RELIMINARY I NTRODUCTION OF G LOW

Original: Flow-based generative models focus on certain types of transformations f θ that allow (i) the inverse functions g θ and Jacobian determinants to be tractable and efﬁcient to compute and (ii) f θ to be expressive.
Revised: Flow-based generative models focus on certain types of transformations f θ that allow (i) the inverse functions g θ and Jacobian determinants to be tractable and efﬁcient to compute and (ii) f θ to be expressive.

Original: Most work within this line of research is dedicated to designing invertible transformations to enhance the expressiveness while maintaining the computational efﬁciency (Kingma & Dhariwal, 2018;
Revised: Most work within this line of research is dedicated to designing invertible transformations to enhance the expressiveness while maintaining the computational efﬁciency (Kingma & Dhariwal, 2018;

Original: Ma et al., 2019a; Ho et al., 2019; Chen et al., 2019), among which Glow (Kingma & Dhariwal, 2018) has stood out for its simplicity and effectiveness.
Revised: Ma et al., 2019a; Ho et al., 2019; Chen et al., 2019), among which Glow (Kingma & Dhariwal, 2018) has stood out for its simplicity and effectiveness.

Original: The following brieﬂy describes the three types of transformations that comprise Glow, which (in a reﬁned version) is adopted as the backbone architecture of the ﬂow-based decoder in our generative model (detailed in Appendix B).
Revised: The following brieﬂy describes the three types of transformations that comprise Glow, which (in a reﬁned version) is adopted as the backbone architecture of the ﬂow-based decoder in our generative model (detailed in Appendix B).

Original: Actnorm.
Revised: Actnorm.

Original: Kingma & Dhariwal (2018) proposed an activation normalization layer (Actnorm) as an alternative for batch normalization (Ioffe & Szegedy, 2015) to alleviate the challenges in model training.
Revised: Kingma & Dhariwal (2018) proposed an activation normalization layer (Actnorm) as an alternative for batch normalization (Ioffe & Szegedy, 2015) to alleviate the challenges in model training.

Original: Similar to batch normalization, Actnorm performs an afﬁne transformation of the activations using a scale and bias parameter per channel for 2D images, such that
Revised: Similar to batch normalization, Actnorm performs an afﬁne transformation of the activations using a scale and bias parameter per channel for 2D images, such that

Original: [Equation]
Revised: [Equation]

Original: where both x and y are in shape [ h × w × c ] with spatial dimensions ( h, w ) and channel dimension c .
Revised: where both x and y are in shape [ h × w × c ] with spatial dimensions ( h, w ) and channel dimension c .

Original: Invertible 1 × 1 convolution.
Revised: Invertible 1 × 1 convolution.

Original: To incorporate a permutation along the channel dimension, Glow includes a trainable invertible 1 × 1 convolution layer to generalize the permutation operation as:
Revised: To incorporate a permutation along the channel dimension, Glow includes a trainable invertible 1 × 1 convolution layer to generalize the permutation operation as:

Original: [Equation]
Revised: [Equation]

Original: where W is the weight matrix with shape c × c .
Revised: where W is the weight matrix with shape c × c .

Original: Afﬁne Coupling Layers.
Revised: Afﬁne Coupling Layers.

Original: Following Dinh et al.
Revised: Following Dinh et al.

Original: , Glow includes afﬁne coupling layers in its architecture of:
Revised: , Glow includes afﬁne coupling layers in its architecture of:

Original: [Equation]
Revised: [Equation]

Original: where s( x a ) and b( x a ) are outputs of two neural networks with x a as input.
Revised: where s( x a ) and b( x a ) are outputs of two neural networks with x a as input.

Original: The split() and concat() functions perform operations along the channel dimension.
Revised: The split() and concat() functions perform operations along the channel dimension.

Original: B I MPLEMENTATION D ETAILS B. C OMPRESSION ENCODER
Revised: B I MPLEMENTATION D ETAILS B. C OMPRESSION ENCODER

Original: The encoder ﬁrst compresses the input image of size [ h × h × c ] to the low-resolution tensor of size 4 × 4 × c (cid:48) .
Revised: The encoder ﬁrst compresses the input image of size [ h × h × c ] to the low-resolution tensor of size 4 × 4 × c (cid:48) .

Original: Then, with a fully-connected layer, the encoder transforms the output tensor to a vector of dimension d z .
Revised: Then, with a fully-connected layer, the encoder transforms the output tensor to a vector of dimension d z .

Original: Concretely, to compress the high-resolution images to low-resolution tensors, the encoder consists of levels of ResNet blocks (He et al., 2016).
Revised: Concretely, to compress the high-resolution images to low-resolution tensors, the encoder consists of levels of ResNet blocks (He et al., 2016).

Original: At each level, there are two ResNet blocks with the same number of hidden units and strides 1 and 2, respectively.
Revised: At each level, there are two ResNet blocks with the same number of hidden units and strides 1 and 2, respectively.

Original: Thus, after each level the input is compressed to half of the spatial dimensions: from h × h to h 2 × h 2 .
Revised: Thus, after each level the input is compressed to half of the spatial dimensions: from h × h to h 2 × h 2 .

Original: ELU (Clevert et al., 2015) is used as the activation function throughout the encoder architecture.
Revised: ELU (Clevert et al., 2015) is used as the activation function throughout the encoder architecture.

Original: B. S CALE TERM IN AFFINE COUPLING LAYERS
Revised: B. S CALE TERM IN AFFINE COUPLING LAYERS

Original: To model the scale term s in (6), a straight-forward way is to take the output of the neural network as the logarithm of s .
Revised: To model the scale term s in (6), a straight-forward way is to take the output of the neural network as the logarithm of s .

Original: Formally, let u denote as the output from the neural network described in (7).
Revised: Formally, let u denote as the output from the neural network described in (7).

Original: Then we can compute s by taking the exponential function: s = exp( u ) .
Revised: Then we can compute s by taking the exponential function: s = exp( u ) .

Original: In practice, however, we found this formulation leads to numerical issues in model training.
Revised: In practice, however, we found this formulation leads to numerical issues in model training.

Original: In our implementation, we calculate s in the following way:
Revised: In our implementation, we calculate s in the following way:

Original: [Equation]
Revised: [Equation]

Original: where the constant α ∈ (0 , 1) .
Revised: where the constant α ∈ (0 , 1) .

Original: In this formulation, we restrict s in the range of [1 − α, 1 + α ] .
Revised: In this formulation, we restrict s in the range of [1 − α, 1 + α ] .

Original: For
Revised: For

Original: ImageNet, we set α = 0 .
Revised: ImageNet, we set α = 0 .

Original: 5 while for other datasets we used α = 1 . 0 .
Revised: 5 while for other datasets we used α = 1 . 0 .

Original: In the experiments, we found this formulation not only improved the numerical stability but also achieved better performance on density estimation and FID scores.
Revised: In the experiments, we found this formulation not only improved the numerical stability but also achieved better performance on density estimation and FID scores.

Original: B. P RIOR DISTRIBUTION IN VAE S
Revised: B. P RIOR DISTRIBUTION IN VAE S

Original: In this work, the prior distribution p θ ( z ) in VAE is modeled with a generative ﬂow with architecture similar to Glow.
Revised: In this work, the prior distribution p θ ( z ) in VAE is modeled with a generative ﬂow with architecture similar to Glow.

Original: The generative ﬂow also consists of three elementary invertible transformations: actnorm, invertible linear layer and afﬁne coupling layer.
Revised: The generative ﬂow also consists of three elementary invertible transformations: actnorm, invertible linear layer and afﬁne coupling layer.

Original: The actnorm and invertible linear layer is similar to those in Ma et al.
Revised: The actnorm and invertible linear layer is similar to those in Ma et al.

Original: (2019c), with the difference that we did not use the multi-head mechanism.
Revised: (2019c), with the difference that we did not use the multi-head mechanism.

Original: The afﬁne coupling layer is similar to the one in Glow, which applies the split function across the dimension d z .
Revised: The afﬁne coupling layer is similar to the one in Glow, which applies the split function across the dimension d z .

Original: The neural networks for the scale and bias terms in afﬁne coupling layers are implemented with multi-layer perceptrons (MLP).
Revised: The neural networks for the scale and bias terms in afﬁne coupling layers are implemented with multi-layer perceptrons (MLP).

Original: C E XPERIMENTAL D ETAILS C. P REPROCESSING
Revised: C E XPERIMENTAL D ETAILS C. P REPROCESSING

Original: We used random horizontal ﬂipping for CIFAR10, and CelebA-HQ 256.
Revised: We used random horizontal ﬂipping for CIFAR10, and CelebA-HQ 256.

Original: For CIFAR-10, we also used random cropping after reﬂection padding with 4 pixels.
Revised: For CIFAR-10, we also used random cropping after reﬂection padding with 4 pixels.

Original: For LSUN 128, we ﬁrst centre cropped the original image, then downsampled to size 128 × 128 .
Revised: For LSUN 128, we ﬁrst centre cropped the original image, then downsampled to size 128 × 128 .

Original: C. O PTIMIZATION
Revised: C. O PTIMIZATION

Original: Parameter optimization is performed with the Adam optimizer (Kingma & Ba, 2014) with β = (0 . 9 , 0 . 999) and (cid:15) = 1e − 8 .
Revised: Parameter optimization is performed with the Adam optimizer (Kingma & Ba, 2014) with β = (0 . 9 , 0 . 999) and (cid:15) = 1e − 8 .

Original: Warmup training is applied to all the experiments: the learning rate linearly increases to the initial learning rate 1e − 3 .
Revised: Warmup training is applied to all the experiments: the learning rate linearly increases to the initial learning rate 1e − 3 .

Original: Then we use exponential decay to decrease the learning rate with decay rate is 0 . 999997 .
Revised: Then we use exponential decay to decrease the learning rate with decay rate is 0 . 999997 .

Original: C. H YPER - PARAMETERS
Revised: C. H YPER - PARAMETERS

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C OMPARISON OF M ODEL S IZE AND T RAINING S PEED
Revised: C OMPARISON OF M ODEL S IZE AND T RAINING S PEED

Original: Table 6 provides the number of parameters of different models on CIFAR-10, together with the corresponding training time over one epoch (measured on four Tesla V100 GPUs).
Revised: Table 6 provides the number of parameters of different models on CIFAR-10, together with the corresponding training time over one epoch (measured on four Tesla V100 GPUs).

Original: With similar model size, our reﬁned Glow model obtains better performance and faster speed, demonstrating the effectiveness and efﬁciency of the reﬁned architecture.
Revised: With similar model size, our reﬁned Glow model obtains better performance and faster speed, demonstrating the effectiveness and efﬁciency of the reﬁned architecture.

Original: Due to introducing the encoder, our
Revised: Due to introducing the encoder, our

Original: VAE-based model contains a little bit more parameters and the training speed is a little slower than the reﬁned Glow.
Revised: VAE-based model contains a little bit more parameters and the training speed is a little slower than the reﬁned Glow.

Original: Same as training a standard VAE model, ELBO is used as the training objective.
Revised: Same as training a standard VAE model, ELBO is used as the training objective.

Original: [Table]
Revised: [Table]

Original: Similar to VAE-based models, exact inference is not preserved and negative log-likelihood (NLL) is approximated with ELBO in model evaluation.
Revised: Similar to VAE-based models, exact inference is not preserved and negative log-likelihood (NLL) is approximated with ELBO in model evaluation.

Original: However, since EBLO provides a way to estimate the upper bound of NLL, the scores in Table 1 are comparable to previous work.
Revised: However, since EBLO provides a way to estimate the upper bound of NLL, the scores in Table 1 are comparable to previous work.

Original: D E XAMPLES FOR TWO - DIMENSIONAL INTERPOLATION
Revised: D E XAMPLES FOR TWO - DIMENSIONAL INTERPOLATION

Original: E M ORE SAMPLES FOR SWITCH OPERATION
Revised: E M ORE SAMPLES FOR SWITCH OPERATION

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: CIFAR
Revised: CIFAR

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original:  I 
Revised: 10 & I MAGE

Original: [Figure]
Revised: [Figure]

Revised: ET