Original: P REVENTING POSTERIOR COLLAPSE WITH δ -VAE S
Revised: P REVENTING POSTERIOR COLLAPSE WITH δ -VAE S

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Due to the phenomenon of “posterior collapse”, current latent variable generative models pose a challenging design choice which trades-off optimizing the ELBO but handicapping the decoder’s capacity and expressivity, or changing the loss to something that is not directly minimizing the description length of the data.
Revised: Due to the phenomenon of “posterior collapse”, current latent variable generative models pose a challenging design choice which trades-off optimizing the ELBO but handicapping the decoder’s capacity and expressivity, or changing the loss to something that is not directly minimizing the description length of the data.

Original: In this paper we propose an alternative that utilizes the best, most powerful generative models as decoders, whilst optimizing the proper variational lower bound all while ensuring that the latent variables preserve and encode useful information.
Revised: In this paper we propose an alternative that utilizes the best, most powerful generative models as decoders, whilst optimizing the proper variational lower bound all while ensuring that the latent variables preserve and encode useful information.

Original: δ -VAEs proposed here achieve this by constraining the variational family for the posterior to have a minimum distance to the prior.
Revised: δ -VAEs proposed here achieve this by constraining the variational family for the posterior to have a minimum distance to the prior.

Original: For sequential latent variable models, our approach resembles the classic representation learning approach of slow feature analysis.
Revised: For sequential latent variable models, our approach resembles the classic representation learning approach of slow feature analysis.

Original: We demonstrate the efﬁcacy of our approach at modeling text on LM1B and modeling images: learning representations, improving sample quality, and achieving state of the art log-likelihood on CIFAR-10 and ImageNet 32 × 32 .
Revised: We demonstrate the efﬁcacy of our approach at modeling text on LM1B and modeling images: learning representations, improving sample quality, and achieving state of the art log-likelihood on CIFAR-10 and ImageNet 32 × 32 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep latent variable models trained with amortized variational inference have led to advances in representation learning on high-dimensional datasets (Kingma & Welling, 2013; Rezende et al., 2014).
Revised: Deep latent variable models trained with amortized variational inference have led to advances in representation learning on high-dimensional datasets (Kingma & Welling, 2013; Rezende et al., 2014).

Original: These latent variable models typically have simple decoders, where the mapping from the latent variable to the input space is unimodal.
Revised: These latent variable models typically have simple decoders, where the mapping from the latent variable to the input space is unimodal.

Original: This typically results in representations that are good at capturing the global structure, but fail at capturing more complex local structure (e.g., texture (Larsen et al., 2016)).
Revised: This typically results in representations that are good at capturing the global structure, but fail at capturing more complex local structure (e.g., texture (Larsen et al., 2016)).

Original: In parallel, advances in autoregressive models have led to drastic improvements in density modeling and sample quality without explicit latent variables (van den Oord et al., 2016b).
Revised: In parallel, advances in autoregressive models have led to drastic improvements in density modeling and sample quality without explicit latent variables (van den Oord et al., 2016b).

Original: While these models are good at capturing local statistics, they often fail to produce globally coherent structures (Ostrovski et al., 2018).
Revised: While these models are good at capturing local statistics, they often fail to produce globally coherent structures (Ostrovski et al., 2018).

Original: Combining the powerful tractable densities of autoregressive models with the representation learning capabilities of latent variable models could result in higher-quality generative models with useful latent representations.
Revised: Combining the powerful tractable densities of autoregressive models with the representation learning capabilities of latent variable models could result in higher-quality generative models with useful latent representations.

Original: While much prior work has attempted to pair the advantages of these two model by using the autoregressive model as the decoder (mapping from latents to inputs), a common problem remains.
Revised: While much prior work has attempted to pair the advantages of these two model by using the autoregressive model as the decoder (mapping from latents to inputs), a common problem remains.

Original: The “posterior collapse” which stems from the autoregressive decoder being too expressive to model the data density, and causes the model to learn a trivial posterior collapsing to the prior.
Revised: The “posterior collapse” which stems from the autoregressive decoder being too expressive to model the data density, and causes the model to learn a trivial posterior collapsing to the prior.

Original: In this case, the decoder can ignore the latents (which are effectively pure noise).
Revised: In this case, the decoder can ignore the latents (which are effectively pure noise).

Original: This phenomenon has been observed by numerous previous works and has been referred to as optimisation challenges of VAEs by Bowman et al.
Revised: This phenomenon has been observed by numerous previous works and has been referred to as optimisation challenges of VAEs by Bowman et al.

Original: (2015), the information preference property by Chen et al.
Revised: (2015), the information preference property by Chen et al.

Original: (2016), and the posterior collapse problems by several others (e.g., van den Oord et al.
Revised: (2016), and the posterior collapse problems by several others (e.g., van den Oord et al.

Original: Kim et al.
Revised: Kim et al.

Original: We adopt the later name throughout the paper.
Revised: We adopt the later name throughout the paper.

Original: Ideally, an approach that mitigates posterior collapse would not alter the ELBO training objective, and would allow the practitioner to leverage the most recent advances in powerful autoregressive decoders to improve performance.
Revised: Ideally, an approach that mitigates posterior collapse would not alter the ELBO training objective, and would allow the practitioner to leverage the most recent advances in powerful autoregressive decoders to improve performance.

Original: To the best of our knowledge, no prior work has succeeded at this goal.
Revised: To the best of our knowledge, no prior work has succeeded at this goal.

Original: Existing approaches change the objective (Higgins et al., 2017; Alemi et al., 2017; Zhao et al., 2017; Chen et al., 2016; Lucas & Verbeek, 2017), weaken the decoder (Bowman et al., 2015;
Revised: Existing approaches change the objective (Higgins et al., 2017; Alemi et al., 2017; Zhao et al., 2017; Chen et al., 2016; Lucas & Verbeek, 2017), weaken the decoder (Bowman et al., 2015;

Original: Gulrajani et al., 2016), or place strong restrictions on the variational family (van den Oord et al., 2017; Xu & Durrett, 2018; Guu et al., 2017).
Revised: Gulrajani et al., 2016), or place strong restrictions on the variational family (van den Oord et al., 2017; Xu & Durrett, 2018; Guu et al., 2017).

Original: Additionally, these approaches are often challenging to tune and highly sensitive to hyperparameters (Alemi et al., 2017; Chen et al., 2016).
Revised: Additionally, these approaches are often challenging to tune and highly sensitive to hyperparameters (Alemi et al., 2017; Chen et al., 2016).

Original: In this paper, we propose δ -VAEs, a simple framework for choosing variational families that prevent posterior collapse without altering the ELBO training objective or weakening the decoder.
Revised: In this paper, we propose δ -VAEs, a simple framework for choosing variational families that prevent posterior collapse without altering the ELBO training objective or weakening the decoder.

Original: By re- stricting the parameters or family of the posterior, we ensure that there is a minimum KL divergence, δ , between the posterior and the prior.
Revised: By re- stricting the parameters or family of the posterior, we ensure that there is a minimum KL divergence, δ , between the posterior and the prior.

Original: We demonstrate the effectiveness of this approach at learning latent-variable models with powerful decoders on images (CIFAR-10, and ImageNet 32 × 32 ), and text (LM1B).
Revised: We demonstrate the effectiveness of this approach at learning latent-variable models with powerful decoders on images (CIFAR-10, and ImageNet 32 × 32 ), and text (LM1B).

Original: We achieve SOTA log-likelihood results with image models by additionally introducing a sequential latent-variable model with an anti-causal encoder structure.
Revised: We achieve SOTA log-likelihood results with image models by additionally introducing a sequential latent-variable model with an anti-causal encoder structure.

Original: Our experiments demonstrate the utility of δ -VAEs at learning useful representations for downstream tasks without sacriﬁcing performance on density modeling.
Revised: Our experiments demonstrate the utility of δ -VAEs at learning useful representations for downstream tasks without sacriﬁcing performance on density modeling.

Original:  M ITIGATING POSTERIOR COLLAPSE WITH δ -VAE S
Revised:  M ITIGATING POSTERIOR COLLAPSE WITH δ -VAE S

Original: Our proposed δ -VAE builds upon the framework of variational autoencoders (VAEs) (Kingma &
Revised: Our proposed δ -VAE builds upon the framework of variational autoencoders (VAEs) (Kingma &

Original: Welling, 2013; Rezende et al., 2014) for training latent-variable models with amortized variational inference.
Revised: Welling, 2013; Rezende et al., 2014) for training latent-variable models with amortized variational inference.

Original: Our goal is to train a generative model p ( x, z ) to maximize the marginal likelihood log p ( x ) on a dataset.
Revised: Our goal is to train a generative model p ( x, z ) to maximize the marginal likelihood log p ( x ) on a dataset.

Original: As the marginal likelihood requires computing an intractable integral over the unobserved latent variable z , VAEs introduce an encoder network q ( z | x ) and optimize a tractable lower bound (the ELBO): log p ( x ) ≥ E z ∼ q ( z | x ) [log p ( x | z )] −  KL ( q ( z | x ) (cid:107) p ( z )) .
Revised: As the marginal likelihood requires computing an intractable integral over the unobserved latent variable z , VAEs introduce an encoder network q ( z | x ) and optimize a tractable lower bound (the ELBO): log p ( x ) ≥ E z ∼ q ( z | x ) [log p ( x | z )] − D KL ( q ( z | x ) (cid:107) p ( z )) .

Original: The ﬁrst term is the reconstruction term, while the second term (KL) is the rate term, as it measures how many nats on average are required to send through the latent variables from the encoder ( q ( z | x ) ) to the decoder ( p ( z | x ) ) (Matthew D. Hoffman, 2016;
Revised: The ﬁrst term is the reconstruction term, while the second term (KL) is the rate term, as it measures how many nats on average are required to send through the latent variables from the encoder ( q ( z | x ) ) to the decoder ( p ( z | x ) ) (Matthew D. Hoffman, 2016;

Original: Alemi et al., 2017).
Revised: Alemi et al., 2017).

Original: The problem of posterior collapse is when the rate term,  KL ( q ( z | x ) (cid:107) p ( z )) reduces to 0.
Revised: The problem of posterior collapse is when the rate term, D KL ( q ( z | x ) (cid:107) p ( z )) reduces to 0.

Original: In this case, the approximate posterior q ( z | x ) is trivially equal to the prior p ( z ) , thus the latent variables do not convey any information about the input x .
Revised: In this case, the approximate posterior q ( z | x ) is trivially equal to the prior p ( z ) , thus the latent variables do not convey any information about the input x .

Original: A necessary condition if we want representations to be meaningful is to have the rate term be positive.
Revised: A necessary condition if we want representations to be meaningful is to have the rate term be positive.

Original: In this paper we propose to address the posterior collapse problem with structural constraints on the prior and posterior such that the divergence between them is lower bounded by design.
Revised: In this paper we propose to address the posterior collapse problem with structural constraints on the prior and posterior such that the divergence between them is lower bounded by design.

Original: This can be achieved by choosing families of distributions for the prior and approximate posterior, p λ ( z ) and q φ ( z | x ) such that min λ,φ D KL ( q φ ( z | x ) (cid:107) p λ ( z )) ≥ δ .
Revised: This can be achieved by choosing families of distributions for the prior and approximate posterior, p λ ( z ) and q φ ( z | x ) such that min λ,φ D KL ( q φ ( z | x ) (cid:107) p λ ( z )) ≥ δ .

Original: We refer to δ as the committed rate of the model.
Revised: We refer to δ as the committed rate of the model.

Original: Note that a trivial choice for p and q to have a non-zero committed rate is to set them to Gaussian distributions with ﬁxed (but different) variance term.
Revised: Note that a trivial choice for p and q to have a non-zero committed rate is to set them to Gaussian distributions with ﬁxed (but different) variance term.

Original: We study a variant of this case in the experiments, and provide more details of this setup in the Appendix D. In the following section we describe our choices for p λ and q φ , but others should also be explored in future work.
Revised: We study a variant of this case in the experiments, and provide more details of this setup in the Appendix D. In the following section we describe our choices for p λ and q φ , but others should also be explored in future work.

Original: 2. δ -VAE WITH S EQUENTIAL L ATENT V ARIABLES
Revised: 2. δ -VAE WITH S EQUENTIAL L ATENT V ARIABLES

Original: Intuitively, data such as speech, natural images and text exhibit strong spatiotemporal continuity.
Revised: Intuitively, data such as speech, natural images and text exhibit strong spatiotemporal continuity.

Original: Our aim is to model variations in such data through latent variables, so that we have control over not just the global characteristics of the generated samples (e.g., existence of an object), but also can inﬂuence their ﬁner, often shifting attributes such as texture and pose in the case of natural images, tone, volume and accent in the case of speech, or style and sentiment in the case of natural language.
Revised: Our aim is to model variations in such data through latent variables, so that we have control over not just the global characteristics of the generated samples (e.g., existence of an object), but also can inﬂuence their ﬁner, often shifting attributes such as texture and pose in the case of natural images, tone, volume and accent in the case of speech, or style and sentiment in the case of natural language.

Original: Sequences of latent variables can be an effective modeling tool for expressing the occurrence and evolution of such features throughout the sequence.
Revised: Sequences of latent variables can be an effective modeling tool for expressing the occurrence and evolution of such features throughout the sequence.

Original: To deﬁne the δ -VAE in this sequential setting, we model the posterior distribution of each timestep as q ( z t | x ) =
Revised: To deﬁne the δ -VAE in this sequential setting, we model the posterior distribution of each timestep as q ( z t | x ) =

Original: N ( z t ; µ t ( x ) , σ t ( x )) .
Revised: N ( z t ; µ t ( x ) , σ t ( x )) .

Original: Therefore z i and z j are conditionally independent given x .
Revised: Therefore z i and z j are conditionally independent given x .

Original: For the prior, we use the ﬁrst-order linear autoregressive process, known as AR(1).
Revised: For the prior, we use the ﬁrst-order linear autoregressive process, known as AR(1).

Original: This means that at each timestep, z t = α z t − 1 + (cid:15) t where (cid:15) t is a Gaussian noise with zero mean and σ 2 (cid:15) constant variance.
Revised: This means that at each timestep, z t = α z t − 1 + (cid:15) t where (cid:15) t is a Gaussian noise with zero mean and σ 2 (cid:15) constant variance.

Original: The conditional probability for the latent variable can be expressed as p ( z t | z <t ) =
Revised: The conditional probability for the latent variable can be expressed as p ( z t | z <t ) =

Original: N ( z t ; α z t − 1 , σ (cid:15) ) .
Revised: N ( z t ; α z t − 1 , σ (cid:15) ) .

Original: This process is wide-sense stationary (that is, having constant sufﬁcient statistics through its time evolution) if | α | < 1 .
Revised: This process is wide-sense stationary (that is, having constant sufﬁcient statistics through its time evolution) if | α | < 1 .

Original: If so, then z t has zero mean and variance of σ 2 (cid:15) 1 − α 2 .
Revised: If so, then z t has zero mean and variance of σ 2 (cid:15) 1 − α 2 .

Original: It is thus convenient to choose σ (cid:15) =
Revised: It is thus convenient to choose σ (cid:15) =

Original: √ 1 − α 2 .
Revised: √ 1 − α 2 .

Original: The mismatch in the correlation structure of the prior and the posterior results in the following positive lower-bound on the KL-divergence between the two distributions (see Appendix C for derivation):
Revised: The mismatch in the correlation structure of the prior and the posterior results in the following positive lower-bound on the KL-divergence between the two distributions (see Appendix C for derivation):

Original: [Figure]
Revised: [Figure]

Original: (1) where n is the length of the sequence and d is the dimension of the latent variable at each timestep.
Revised: (1) where n is the length of the sequence and d is the dimension of the latent variable at each timestep.

Original: The amount of committed rate between the prior and the posterior is easily controlled by equating the right hand side of the inequality in equation 1 to a given rate c and solving for α .
Revised: The amount of committed rate between the prior and the posterior is easily controlled by equating the right hand side of the inequality in equation 1 to a given rate δ and solving for α .

Original: In Fig.
Revised: In Fig.

Original: 1, we show the scaling of the minimum rate as a function of α and the behavior of δ -VAE in 2d.
Revised: 1, we show the scaling of the minimum rate as a function of α and the behavior of δ -VAE in 2d.

Original: [Figure]
Revised: [Figure]

Original: 2.1. R ELATION TO P ROBABILISTIC S LOWNESS P RIOR
Revised: 2.1. R ELATION TO P ROBABILISTIC S LOWNESS P RIOR

Original: The linear autoregressive process we have chosen as our main prior over the latent variables speciﬁes the degree of temporal correlation in the latent space.
Revised: The linear autoregressive process we have chosen as our main prior over the latent variables speciﬁes the degree of temporal correlation in the latent space.

Original: The transitions in the latent space are smoother the closer α is to one.
Revised: The transitions in the latent space are smoother the closer α is to one.

Original: On the other hand in the limit of α approaching 0 , the prior becomes the same as the independent standard Gaussian prior where there are no correlations between timesteps.
Revised: On the other hand in the limit of α approaching 0 , the prior becomes the same as the independent standard Gaussian prior where there are no correlations between timesteps.

Original: This exact prior form has indeed been suggested (Turner & Sahani, 2007) to be, in the limit of inﬁnte sequence length, the probabilistic counterpart to Slow Feature Analysis (Wiskott & Sejnowski, 2002).
Revised: This exact prior form has indeed been suggested (Turner & Sahani, 2007) to be, in the limit of inﬁnte sequence length, the probabilistic counterpart to Slow Feature Analysis (Wiskott & Sejnowski, 2002).

Original: SFA has been shown to be an effective method for learning invariant spatio-temporal features (Wiskott & Sejnowski, 2002).
Revised: SFA has been shown to be an effective method for learning invariant spatio-temporal features (Wiskott & Sejnowski, 2002).

Original: In our models, we infer latent variables with multiple dimensions per timestep, each with a different slowness ﬁlter imposed by a different value of α , corresponding to features with different speed of variation.
Revised: In our models, we infer latent variables with multiple dimensions per timestep, each with a different slowness ﬁlter imposed by a different value of α , corresponding to features with different speed of variation.

Original: 2. A NTI -C AUSAL E NCODER N ETWORK
Revised: 2. A NTI -C AUSAL E NCODER N ETWORK

Original: Having a high capacity autoregressive network as the decoder implies that it can accurately estimate p ( x t | x <t ) .
Revised: Having a high capacity autoregressive network as the decoder implies that it can accurately estimate p ( x t | x <t ) .

Original: Given this premise, what kind of complementary information can latent variables provide?
Revised: Given this premise, what kind of complementary information can latent variables provide?

Original: Encoding information about the past seems wasteful as the autoregressive decoder has full access to past observations already, so encoding them in latent variables seems wasteful.
Revised: Encoding information about the past seems wasteful as the autoregressive decoder has full access to past observations already.

Original: On the other hand, if we impose conditional independence between observations and latent variables at other timesteps given the current one (i.e., p ( x t | z ) = p ( x t | z t ) ), there will then be at best (by the data processing inequality (Cover & Thomas, 2006))
Revised: On the other hand, if we impose conditional independence between observations and latent variables at other timesteps given the current one (i.e., p ( x t | z ) = p ( x t | z t ) ), there will then be at best (by the data processing inequality (Cover & Thomas, 2006))

Original: a break-even situation between the KL cost of encoding information in z t and the resulting improvement in the reconstruction loss.
Revised: a break-even situation between the KL cost of encoding information in z t and the resulting improvement in the reconstruction loss.

Original: There is therefore no advantage for the model to utilize the latent variable even if it would transmit to the decoder the unobserved x t .
Revised: There is therefore no advantage for the model to utilize the latent variable even if it would transmit to the decoder the unobserved x t .

Original: The situation is, however, different when z t can inform the decoder at multiple timesteps, encoding mutual information between x t and x >t .
Revised: The situation is, however, different when z t can inform the decoder at multiple timesteps, encoding mutual information between x t and x >t .

Original: In this setting, the decoder pays the KL cost for the mutual information once, but is able to leverage the transmitted information multiple times to reduce its entropy about future predictions.
Revised: In this setting, the decoder pays the KL cost for the mutual information once, but is able to leverage the transmitted information multiple times to reduce its entropy about future predictions.

Original: We introduce an encoder structure motivated by this argument that explicitly restricts the latents at each timestep to encode information about the future.
Revised: We introduce an encoder structure motivated by this argument that explicitly restricts the latents at each timestep to encode information about the future.

Original: We refer to this structure as the anti-causal posterior structure as shown in Fig.
Revised: We refer to this structure as the anti-causal posterior structure as shown in Fig.

Original: Alternatively, one can consider the non-causal structure that allows latents be inferred from all observations were the inference and generative graphs are densely connected.
Revised: Alternatively, one can consider the non-causal structure that allows latents be inferred from all observations were the inference and generative graphs are densely connected.

Original: The non-causal setup thus becomes very similar to non-temporal latent variable models as there now is no temporal order in either the encoder or the decoder.
Revised: The non-causal setup thus becomes very similar to non-temporal latent variable models as there now is no temporal order in either the encoder or the decoder.

Original: The anti-causal structure is a subgraph of the non-causal structure, so it is possible that in the course of training the model may learn to ignore the rest of the non-causal inference network.
Revised: The anti-causal structure is a subgraph of the non-causal structure, so it is possible that in the course of training the model may learn to ignore the rest of the non-causal inference network.

Original: Introducing this additional restriction is thus giving the model an additional inductive bias which can be useful insofar as the underlying premises hold true for the dataset and the decoder model.
Revised: Introducing this additional restriction is thus giving the model an additional inductive bias which can be useful insofar as the underlying premises hold true for the dataset and the decoder model.

Original: We compare the performance of both approaches in different settings in Sect.
Revised: We compare the performance of both approaches in different settings in Appendix F.1.

Original: zx 1,: zzx 2,: x 3,: zx 1,: zzx 2,: x 3,:
Revised: zx 1,: zzx 2,: x 3,: zx 1,: zzx 2,: x 3,:

Original: [Figure]

Original: Model

Original: [Figure]
Revised: [Figure]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The main focus of our work is on representation learning and density modeling in latent variable models with powerful decoders.
Revised: The main focus of our work is on representation learning and density modeling in latent variable models with powerful decoders.

Original: Earlier work has focused on this kind of architecture, but has addressed the problem of posterior collapse in different ways.
Revised: Earlier work has focused on this kind of architecture, but has addressed the problem of posterior collapse in different ways.

Original: In terms of our architecture, the decoders for our image models build on advances in autoregressive modeling from van den Oord et al.
Revised: In terms of our architecture, the decoders for our image models build on advances in autoregressive modeling from van den Oord et al.

Original: (2016b)
Revised: (2016b)

Original: ; Salimans et al.
Revised: ; Salimans et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: ; Parmar et al.
Revised: ; Parmar et al.

Original: Unlike prior models, we use sequential latent variables to generate the image row by row.
Revised: Unlike prior models, we use sequential latent variables to generate the image row by row.

Original: This differs from Gregor et al.
Revised: This differs from Gregor et al.

Original: (2016), where the latents are sequential but the entire image is generated at each timestep.
Revised: (2016), where the latents are sequential but the entire image is generated at each timestep.

Original: Our sequential image generation model resembles latent variable models used for timeseries (Chung et al., 2015; Babaeizadeh et al., 2017; Denton & Fergus, 2018) but does not rely on KL annealing, and has an additional autoregressive dependence of the outputs over time (rows of the image).
Revised: Our sequential image generation model resembles latent variable models used for timeseries (Chung et al., 2015; Babaeizadeh et al., 2017; Denton & Fergus, 2018) but does not rely on KL annealing, and has an additional autoregressive dependence of the outputs over time (rows of the image).

Original: Another main difference between our work and previous sequential latent variable models is our proposed Anti-Causal structure for the inference network (see Sect. 2.2).
Revised: Another main difference between our work and previous sequential latent variable models is our proposed anti-causal structure for the inference network (see Sect. 2.2).

Original: We motivate this structure from a code-efﬁciency and representation learning standpoints and demonstrate its effectiveness empirically in Sect.
Revised: We motivate this structure from a code-efﬁciency and representation learning standpoints and demonstrate its effectiveness empirically in Sect.

Original: For textual data, we use the Transformer architecture from Vaswani et al.
Revised: For textual data, we use the Transformer architecture from Vaswani et al.

Original: (2017) as our main blueprint for the decoder.
Revised: (2017) as our main blueprint for the decoder.

Original: As shown in Sect.
Revised: As shown in Sect.

Original: 4, our method is able to preserve the performance of these models, in terms of likelihoods, whilst learning informative latent variables.
Revised: 4, our method is able to preserve the performance of these models, in terms of likelihoods, whilst learning informative latent variables.

Original: To prevent posterior collapse, most prior work has focused on modifying the training objective.
Revised: To prevent posterior collapse, most prior work has focused on modifying the training objective.

Original: Bowman et al.
Revised: Bowman et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: ; Kim et al.
Revised: ; Kim et al.

Original: (2018) and Gulrajani et al.
Revised: (2018) and Gulrajani et al.

Original: (2016) use an annealing strategy, where they anneal the weight on the rate from 0 to 1 over the course of training.
Revised: (2016) use an annealing strategy, where they anneal the weight on the rate from 0 to 1 over the course of training.

Original: This approach does not directly optimize a lower-bound on likelihood for most of training, and tuning the annealing schedule to prevent collapse can be challenging (see Sect. 4).
Revised: This approach does not directly optimize a lower-bound on likelihood for most of training, and tuning the annealing schedule to prevent collapse can be challenging (see Sect. 4).

Original: Similarly, Higgins et al.
Revised: Similarly, Higgins et al.

Original: (2017) proposes using a ﬁxed coefﬁcient > 1 on the rate term to learn disentangled representations.
Revised: (2017) proposes using a ﬁxed coefﬁcient > 1 on the rate term to learn disentangled representations.

Original: Zhao et al.
Revised: Zhao et al.

Original: (2017) adds a term to the objective to pick the model with maximal rate.
Revised: (2017) adds a term to the objective to pick the model with maximal rate.

Original: Chen et al.
Revised: Chen et al.

Original: ; Kingma et al.
Revised: ; Kingma et al.

Original: (2016) use free-bits to allow the model to hit a target minimum rate, but the objective is non-smooth which leads to optimization difﬁculties in our hands, and deviations from a lower bound on likelihood when the soft version is used with a coefﬁcient less than 1.
Revised: (2016) use free-bits to allow the model to hit a target minimum rate, but the objective is non-smooth which leads to optimization difﬁculties in our hands, and deviations from a lower bound on likelihood when the soft version is used with a coefﬁcient less than 1.

Original: Lucas & Verbeek (2017) add an auxiliary objective that reconstructs a low-resolution version of the input to prevent posterior collapse.
Revised: Lucas & Verbeek (2017) add an auxiliary objective that reconstructs a low-resolution version of the input to prevent posterior collapse.

Original: Alemi et al.
Revised: Alemi et al.

Original: (2017) argue that the ELBO is a defective objective function for representation learning as it does not distinguish between models with different rates, and advocate for model-selection based on downstream tasks.
Revised: (2017) argue that the ELBO is a defective objective function for representation learning as it does not distinguish between models with different rates, and advocate for model-selection based on downstream tasks.

Original: Their method for sweeping models was to use β -VAE with different coefﬁcients, which can be challenging as the mapping from β to rate is highly nonlinear, and model- and data-dependent.
Revised: Their method for sweeping models was to use β -VAE with different coefﬁcients, which can be challenging as the mapping from β to rate is highly nonlinear, and model- and data-dependent.

Original: While we adopt the same perspective as Alemi et al.
Revised: While we adopt the same perspective as Alemi et al.

Original: (2017), we present a new way of achieving a target rate while optimizing the vanilla ELBO objective.
Revised: (2017), we present a new way of achieving a target rate while optimizing the vanilla ELBO objective.

Original: Most similar to our approach is work on constraining the variational family to regularize the model.
Revised: Most similar to our approach is work on constraining the variational family to regularize the model.

Original: VQ-VAE (van den Oord et al., 2017) uses discrete latent variables obtained by vector quantization of the latent space. Using discrete latent variables with a uniform prior over the outcome yields a ﬁxed KL divergence equal to log K , where K is the size of the codebook.
Revised: VQ-VAE (van den Oord et al., 2017) uses discrete latent variables obtained by vector quantization of the latent space that, given a uniform prior over the outcome, yields a ﬁxed KL divergence equal to log K , where K is the size of the codebook.

Original: A number of recent papers have also used  von Mises-Fisher (vMF) distribution to obtain a ﬁxed KL divergence and mitigate the posterior collapse problem.
Revised: A number of recent papers have also used the von Mises-Fisher (vMF) distribution to obtain a ﬁxed KL divergence and mitigate the posterior collapse problem.

Original: In particular, Guu et al.
Revised: In particular, Guu et al.

Original: ; Xu & Durrett (2018)
Revised: ; Xu & Durrett (2018)

Original: ; Davidson et al.
Revised: ; Davidson et al.

Original: (2018) use vMF( µ, κ )distribution with a ﬁxed κ as their posterior, and the uniform distribution ( i.e. vMF( · , 0)) as the prior.
Revised: (2018) use vMF( µ, κ ) with a ﬁxed κ as their posterior, and the uniform distribution ( i.e. vMF( · , 0)) as the prior.

Original: The mismatching prior-posterior thus give a constant KL divergence.
Revised: The mismatching prior-posterior thus give a constant KL divergence.

Original: As such, this approach can be considered as the continuous analogue of VQ-VAE.
Revised: As such, this approach can be considered as the continuous analogue of VQ-VAE.

Original: Unlike the VQVAE and vMF approach which have a constant KL divergence for every datapoint, δ -VAE can allow higher KL for different datapoints.
Revised: Unlike the VQ-VAE and vMF approaches which have a constant KL divergence for every datapoint, δ -VAE can allow higher KL for different datapoints.

Original: This allows the model to allocate more bits for more complicated inputs, which has been shown to be useful for detecting outliers in datasets (Alemi et al., 2018).
Revised: This allows the model to allocate more bits for more complicated inputs, which has been shown to be useful for detecting outliers in datasets (Alemi et al., 2018).

Original: As such, δ -VAE may be considered a generalisation of ﬁxed-KL approaches.
Revised: As such, δ -VAE may be considered a generalisation of ﬁxed-KL approaches.

Revised: The Associative Compression Networks (ACN) (Graves & Menick, 2014) is a new method for learning latent variables with powerful decoders that exploits the associations between training examples in the dataset by amortizing the description length of the code among many similar training examples.

Revised: In so doing, ACN deviates from the i.i.d training regime of the classical methods in statistics and machine learning, and is considered a procedure for compressing whole datasets rather than individual training examples.

Revised: GECO (Jimenez Rezende & Viola, 2018) is a recently proposed method to stabilize the training of β -VAEs by ﬁnding an automatic annealing schedule for the KL that satisﬁes a tolerance constraint for maximum allowed distortion, and solving the resulting Langrange multiplier for the KL penalty.

Revised: The value of β , however, does not necessarily approach one, which means that the optimized objective may not be a lower-bound for the marginal likelihood.

Original:  E XPERIMENTS 4. N ATURAL I MAGES
Revised:  E XPERIMENTS 4. N ATURAL I MAGES

Original: We applied our method to generative modeling of images on the CIFAR-10 (Krizhevsky et al.) and downsampled Imagenet (Deng et al., 2009) ( 32 × 32 as prepared in van den Oord et al.
Revised: We applied our method to generative modeling of images on the CIFAR-10 (Krizhevsky et al.) and downsampled Imagenet (Deng et al., 2009) ( 32 × 32 as prepared in van den Oord et al.

Original: (2016a))
Revised: (2016a))

Original: datasets.
Revised: datasets.

Original: We describe the main components in the following.
Revised: We describe the main components in the following.

Original: The details of our hyperparameters can be found in Appendix E.
Revised: The details of our hyperparameters can be found in Appendix E.

Original: Decoder : Our decoder network is closest to PixelSNAIL (Chen et al., 2017) but also incorporates elements from the original GatedPixelCNN (van den Oord et al 2016b
Revised: Decoder : Our decoder network is closest to PixelSNAIL (Chen et al., 2017) but also incorporates elements from the original GatedPixelCNN (van den Oord et al., 2016b).

Original: In particular, as introduced by Salimans et al.
Revised: In particular, as introduced by Salimans et al.

Original: and used in Chen et al.
Revised: and used in Chen et al.

Original: (2017), we use a single channel network to output the components of discretized mixture of logistics distributions for each channel, and linear dependencies between the RGB colour channels.
Revised: (2017), we use a single channel network to output the components of discretized mixture of logistics distributions for each channel, and linear dependencies between the RGB colour channels.

Original: As in PixelSNAIL, we use attention layers interleaved with masked gated convolution layers.
Revised: As in PixelSNAIL, we use attention layers interleaved with masked gated convolution layers.

Original: We use the same architecture of gated convolution introduced in van den Oord et al.
Revised: We use the same architecture of gated convolution introduced in van den Oord et al.

Original: (2016b).
Revised: (2016b).

Original: We also use the multi-head attention module of Vaswani et al.
Revised: We also use the multi-head attention module of Vaswani et al.

Original: To condition the decoder, similar to Transformer and unlike PixelCNN variants that use 1x1 convolution, we use attention over the output of the encoder.
Revised: To condition the decoder, similar to Transformer and unlike PixelCNN variants that use 1x1 convolution, we use attention over the output of the encoder.

Original: The decoder-encoder attention is causally masked to realize the anti-causal inference structure, and is unmasked for the non-causal structure.
Revised: The decoder-encoder attention is causally masked to realize the anti-causal inference structure, and is unmasked for the non-causal structure.

Original: Encoder .
Revised: Encoder .

Original: Our encoder also uses the same blueprint as the decoder.
Revised: Our encoder also uses the same blueprint as the decoder.

Original: To introduce the anti-causal structure the input is reversed, shifted and cropped by one in order to obtain the desired future context.
Revised: To introduce the anti-causal structure the input is reversed, shifted and cropped by one in order to obtain the desired future context.

Original: Using one latent variable for each pixel is too inefﬁcient in terms of computation.
Revised: Using one latent variable for each pixel is too inefﬁcient in terms of computation.

Original: We, therefore, encode each row of the image with a multi-dimensional latent variable.
Revised: We, therefore, encode each row of the image with a multi-dimensional latent variable.

Original: Auxiliary Prior .
Revised: Auxiliary Prior .

Original: Tomczak & Welling (2017); Matthew D. Hoffman (2016) show that VAEs performance can suffer when there is a signiﬁcant mismatch between the prior and the aggregate posterior, q ( z ) =
Revised: Tomczak & Welling (2017); Matthew D. Hoffman (2016); Jimenez Rezende & Viola (2018) show that VAEs performance can suffer when there is a signiﬁcant mismatch between the prior and the aggregate posterior, q ( z ) =

Original: E x ∼D [ q ( z | x )] .
Revised: E x ∼D [ q ( z | x )] .

Original: When such a gap exists, the decoder is likely to have never seen samples from regions of the support of the prior distribution to which the marginal posterior assigns small probability mass.
Revised: When such a gap exists, the decoder is likely to have never seen samples from regions of the support of the prior distribution to which the aggregate posterior assigns small probability mass.

Original: This phenomenon can be exacerbated in δ -VAEs, where the systematic mismatch between the prior and the posterior might induce a large gap between the prior and aggregate posterior.
Revised: This phenomenon, also known as the ”posterior holes” problem (Jimenez Rezende & Viola, 2018), can be exacerbated in δ -VAEs, where the systematic mismatch between the prior and the posterior might induce a large gap between the prior and aggregate posterior.

Original: Increasing the complexity of the variational family can reduce this gap (Rezende & Mohamed, 2015), but require changes in the objective to control the rate and prevent posterior collapse (Kingma et al., 2016).
Revised: Increasing the complexity of the variational family can reduce this gap (Rezende & Mohamed, 2015), but require changes in the objective to control the rate and prevent posterior collapse (Kingma et al., 2016).

Original: To address this limitation, we adopt the approaches of van den Oord et al.
Revised: To address this limitation, we adopt the approaches of van den Oord et al.

Original: ; Roy et al.
Revised: ; Roy et al.

Original: and train a rich auxiliary prior over the course of learning to match the aggregate posterior, but that does not inﬂuence the encoder or decoder.
Revised: and train a rich auxiliary prior over the course of learning to match the aggregate posterior, but that does not inﬂuence the encoder or decoder.

Original: We used a simple autoregressive model for the auxiliary prior p
Revised: We used a simple autoregressive model for the auxiliary prior p

Original: aux : a single-layer LSTM network with conditional-Gaussian outputs.
Revised: aux : a single-layer LSTM network with conditional-Gaussian outputs.

Original: 4.1. D ENSITY E STIMATION R ESULTS
Revised: 4.1. D ENSITY E STIMATION R ESULTS

Original: We begin by comparing our approach to prior work on CIFAR-10 and downsampled Imagenet 32x in Table 1.
Revised: We begin by comparing our approach to prior work on CIFAR-10 and downsampled Imagenet 32x in Table 1.

Original: As expected, we found that the capacity of the employed autoregressive decoder had a large impact on the overall performance.
Revised: As expected, we found that the capacity of the employed autoregressive decoder had a large impact on the overall performance.

Original: Nevertheless, our models have a negligible gap compared to their powerful autoregressive counterparts, while also learning informative latent variables.
Revised: Nevertheless, our models have a negligible gap compared to their powerful autoregressive counterparts, while also learning informative latent variables.

Original: In comparison, (Chen et al., 2016) had a 0.03 bits per dimension gap with respect to their employed
Revised: In comparison, (Chen et al., 2016) had a 0.03 bits per dimension gap with respect to their employed

Original: PixelCNN++ architecture 1 .
Revised: PixelCNN++ architecture 1 .

Original: On Imagenet 32x32, our latent variable model achieves on par performance with purely autoregressive Image Transformer (Parmar et al., 2018).
Revised: On Imagenet 32x32, our latent variable model achieves on par performance with purely autoregressive Image Transformer (Parmar et al., 2018).

Original: On CIFAR-10 we achieve a new state of the art of 2.83 bits per dimension, again matching the performance of our autoregressive baseline.
Revised: On CIFAR-10 we achieve a new state of the art of 2.83 bits per dimension, again matching the performance of our autoregressive baseline.

Original: Note that the values for KL appear quite small as they are reported in bits per dimension (e.g. 0.02 bits/dim translates to 61 bits/image encoded in the latents).
Revised: Note that the values for KL appear quite small as they are reported in bits per dimension (e.g. 0.02 bits/dim translates to 61 bits/image encoded in the latents).

Original: [-CIFAR
 Test (KL) Imagenet 32x
-]
Revised: [-CIFAR
 Test (KL) Imagenet 32x
-]The results on CIFAR-10 also demonstrate the effect of the auxiliary prior on improving the efﬁciency of the latent code; it leads to more than 50% (on average 30 bits per image) reduction in the rate of the model to achieve the same performance.

Original: Valid (KL)
Revised: [Table]

Original: LatentVariable Models
Revised: Latent

Revised: [Table]

Original: ConvDraw (Gregor et al.
Revised: ConvDraw (Gregor et al.

Original: (2016)) ≤ 3.
Revised: (2016)) ≤ 3.

Original: DenseNet VLAE (Chen et al.
Revised: DenseNet VLAE (Chen et al.

Original: ≈ 2.
Revised: ≈ 2.

Original: - δ -VAE + PixelSNAIL + AR(1) Prior ≤ 2.85 (0.02) ≤ 3.78 (0.08) δ -VAE + PixelSNAIL + Auxiliary Prior ≤ 2.83 (0.01) ≤ 3.77 (0.07) Autoregressive Models Gated PixelCNN(van den Oord et al. (2016b))
Revised: - δ -VAE + PixelSNAIL + AR(1) Prior ≤ 2.85 (0.02) ≤ 3.78 (0.08) δ -VAE + PixelSNAIL + Auxiliary Prior ≤ 2.83 (0.

Original: [Table]

Original: [Table]

Original: PixelCNN++ (Salimans et al.

Original: [Table]

Original: PixelRNN (van den Oord et al.

Original: (2016a))

Original: [Table]

Original: ImageTransformer (Parmar et al.

Original: [Table]

Original: [Table]

Original: PixelSNAIL (Chen et al.

Original: [Table]

Original: [Table]

Original: Our Decoder baseline

Original: [Table]
Revised: [Table]

Original: 4. U TILIZATION OF L ATENT V ARIABLES
Revised: 4. U TILIZATION OF L ATENT V ARIABLES

Original: In this section, we aim to demonstrate that our models learn meaningful representations of the data in the latent variables.
Revised: In this section, we aim to demonstrate that our models learn meaningful representations of the data in the latent variables.

Original: We ﬁrst investigate the effect of z on the generated samples from the model.
Revised: We ﬁrst investigate the effect of z on the generated samples from the model.

Original: Fig.
Revised: Fig.

Original: 3 depicts samples from an ImageNet model (see Appendix for CIFAR-10), where we sample from the decoder network multiple times conditioned on a ﬁxed sample from the auxiliary prior.
Revised: 3 depicts samples from an ImageNet model (see Appendix for CIFAR-10), where we sample from the decoder network multiple times conditioned on a ﬁxed sample from the auxiliary prior.

Original: We see similar global structure (e.g. same color background, scale and structure of objects) but very different details.
Revised: We see similar global structure (e.g. same color background, scale and structure of objects) but very different details.

Original: This indicates that the model is using the latent variable to capture global structure, while the autoregressive decoder is ﬁlling in local statistics and patterns.
Revised: This indicates that the model is using the latent variable to capture global structure, while the autoregressive decoder is ﬁlling in local statistics and patterns.

Original: For a more quantitative assessment of how useful the learned representations are for downstream tasks, we performed linear classiﬁcation from the representation to the class labels on CIFAR-10.
Revised: For a more quantitative assessment of how useful the learned representations are for downstream tasks, we performed linear classiﬁcation from the representation to the class labels on CIFAR-10.

Original: We also study the effect of the chosen rate of the model on classiﬁcation accuracy as illustrated in  the right graph of Fig.
Revised: We also study the effect of the chosen rate of the model on classiﬁcation accuracy as illustrated in Fig. 4b, along with the performance of other methods.

Original: We ﬁnd that generally a model with higher rate gives better classiﬁcation accuracy, with our highest rate model, encoding 0.1 bits per dimension, giving the best accuracy of 68% .
Revised: We ﬁnd that generally a model with higher rate gives better classiﬁcation accuracy, with our highest rate model, encoding 92 bits per image, giving the best accuracy of 68% .

Original: However, contrasting classiﬁcation accuracy with negative log-likelihood of the models we ﬁnd that higher likelihood does not necessarily yield features for better linear classiﬁcation results.
Revised: However, contrasting classiﬁcation accuracy with negative loglikelihood of the models we ﬁnd that higher likelihood does not necessarily yield features for better linear classiﬁcation results.

Original: We caution that an obviously important requirement for this task is the  linear separability of the learned feature space, which does not necessarily align with the desire to learn highly compressed representations.
Revised: We caution that an obviously important requirement for this task is the [Figure] linear separability of the learned feature space, which does not necessarily align with the desire to learn highly compressed representations.

Original: 4. A BLATION STUDIES
Revised: 4. A BLATION STUDIES

Original: Table 2 reports the  results of our experimentscomparing δ -VAE with popular mitigation techniques for the posterior collapse problem.
Revised: We performed compare δ -VAE with other approaches to prevent posterior collapse on the CIFAR10 dataset. We employ the same medium sized encoder and decoder for evaluating all methods as detailed in Appendix E. Fig. 4a reports the rate-distortion results of our experiments for the CIFAR10 test set. To better highlight the difference between models and to put into perspective the amount of information that latent variables capture about images, the rate and distortion results in Fig.

Revised: 4a are reported in bits per images.

Original: We  report results for models that encode a non-negligible amount information in latent variables.
Revised: We only report results for models that encode a non-negligible amount information in latent variables.

Original: Unlike the committed information rate approach of δ -VAE, most alternative solutions required considerable amount of effort to get the training converge or prevent the KL from collapsing altogether.
Revised: Unlike the committed information rate approach of δ -VAE, most alternative solutions required considerable amount of effort to get the training converge or prevent the KL from collapsing altogether.

Original: For example, with linear annealing of KL (Bowman et al., 2015), despite trying a wide range of values for the end step of the annealing schedule, we were not able to train a model with a signiﬁcant usage of latent variables; the KL collapsed as soon as β approached 1.0.
Revised: For example, with linear annealing of KL (Bowman et al., 2015), despite trying a wide range of values for the end step of the annealing schedule, we were not able to train a model with a signiﬁcant usage of latent variables; the KL collapsed as soon as β approached 1.0.

Original: A practical advantage of our approach is its simple formula to choose the target minimum rate of the model.
Revised: A practical advantage of our approach is its simple formula to choose the target minimum rate of the model.

Original: Targeting a desired rate in β -VAE, on the other hand, proved to be difﬁcult, as many of[Figure] our attempts resulted in either collapsed KL, or very large KL values that led to inefﬁcient inference.
Revised: Targeting a desired rate in β -VAE, on the other hand, proved to be difﬁcult, as many of our attempts resulted in either collapsed KL, or very large KL values that led to inefﬁcient inference.

Original: As reported in Chen et al.
Revised: As reported in Chen et al.

Original: , we also observed that optimising models with the free-bits loss was challenging and sensitive to hyperparameter values.
Revised: , we also observed that optimising models with the free-bits loss was challenging and sensitive to hyperparameter values.

Revised: To assess each methods tendency to overﬁt across the range of rates, we also report the rate-distortion results for CIFAR-10 training sets in Appendix F.

Revised: While beta-VAEs do ﬁnd points along the ratedistortion optimal frontier on the training set, we found that they overﬁt more than delta-VAEs, with delta-VAEs dom inating the rate-distoertion frontier on heldout data.

Original: Next, we compare the performance of the anti-causal encoder structure with the non-causal structure  discussed in Sect.
Revised: Next, we compare the performance of the anti-causal encoder structure with the non-causal structure on the CIFAR-10 dataset discussed in Sect.

Original: The results for several conﬁgurations of our model are reported in the Appendix Table 6.
Revised: The results for several conﬁgurations of our model are reported in the Appendix Table 6.

Original: In models where the decoder is not powerful enough (such as our 6-layer PixelCNN that has no attention and consequently a receptive ﬁeld smaller than the causal context for most pixels), the anti-causal structure does not perform as well as the non-causal structure.
Revised: In models where the decoder is not powerful enough (such as our 6-layer PixelCNN that has no attention and consequently a receptive ﬁeld smaller than the causal context for most pixels), the anti-causal structure does not perform as well as the noncausal structure.

Original: The performance gap is however closed as the decoder becomes more powerful and its receptive ﬁeld grows by adding self-attention and more layers.
Revised: The performance gap is however closed as the decoder becomes more powerful and its receptive ﬁeld grows by adding self-attention and more layers.

Original: We observed that the anti-causal structure outperforms the non-causal encoder for very high capacity decoders, as well as for medium size models with a high
Revised: We observed that the anticausal structure outperforms the non-causal encoder for very high capacity decoders, as well as for [Figure] i s t o r t i

Revised: [Figure]

Revised: [Figure]

Original: 4. T EXT
Revised: i t s / I [Figure] a [Figure]

Revised: medium size models with a high rate.

Revised: We also repeated these experiments with both anti-causal and non-causal structures but without imposing a committed information rate or using other mitigation strategies, and found that neither structure by itself is able to mitigate the posterior collapse issue; in both cases the KL divergence drops to negligible values (¡ 10 − 8 bits per dimension) only after a few thousand training steps and never recovers.

Revised: 4. T EXT

Original: For our experiments on natural language, we used the 1 Billion Words or LM1B (Chelba et al., 2013) dataset in its processed form in the Tensor2Tensor Vaswani et al.
Revised: For our experiments on natural language, we used the 1 Billion Words or LM1B (Chelba et al., 2013) dataset in its processed form in the Tensor2Tensor Vaswani et al.

Original: (2018) codebase 2 .
Revised: (2018) codebase 2 .

Original: The dataset is[Table] All the models had two self-attention layers interspersed with 6 layers of gated convolution not tokenized in subword tokens with a vocabulary size of about 32000.
Revised: The dataset is tokenized in subword tokens with a vocabulary size of about 32000.

Original: We discarded sequences longer than 256 tokens.
Revised: We discarded sequences longer than 256 tokens.

Original: Our employed architecture for text closely follows the Transformer network of
Revised: Our employed architecture for text closely follows the Transformer network of

Original: Vaswani et al.
Revised: Vaswani et al.

Original: Our sequence of latent variables has the same number of elements as in the number of tokens in the input, each having two dimensions with α = 0 .
Revised: Our sequence of latent variables has the same number of elements as in the number of tokens in the input, each having two dimensions with α = 0 .

Original: 2 and 0 .
Revised: 2 and 0 .

Original: Our decoder uses causal self-attention as in Vaswani et al.
Revised: Our decoder uses causal self-attention as in Vaswani et al.

Original: For the anti-causal structure in the encoder, we use the inverted causality masks as in the decoder to only allow looking at the current timestep and the future.
Revised: For the anti-causal structure in the encoder, we use the inverted causality masks as in the decoder to only allow looking at the current timestep and the future.

Original: The quantitative results for our experiments is listed in Table 3, indicating that our model achieves slightly worse log-likelihood compared to its auto-regressive baseline, but makes considerable use of latent variables.
Revised: The quantitative results for our experiments is listed in Table 2, indicating that our model achieves slightly worse log-likelihood compared to its auto-regressive baseline, but makes considerable use of latent variables.

Original: Please refer to Appendix H for samples from our text model demonstrating this.
Revised: Please refer to Appendix H for samples from our text model demonstrating this.

Original: [Table]
Revised: [Table]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this work, we have demonstrated that our proposed δ -VAE model provides a simple, intuitive, and effective solution to posterior collapse in latent variable models, enabling them to be paired with powerful decoders.
Revised: In this work, we have demonstrated that our proposed δ -VAE model provides a simple, intuitive, and effective solution to posterior collapse in latent variable models, enabling them to be paired with powerful decoders.

Original: Unlike prior work, we do not require changes to the objective or weakening of the decoder, and we can learn useful representations as well as achieving state-of-the-art likelihoods.
Revised: Unlike prior work, we do not require changes to the objective or weakening of the decoder, and we can learn useful representations as well as achieving state-of-the-art likelihoods.

Original: While our work presents two simple posterior-prior pairs, there are a number of other possibilities that could be explored in future work.
Revised: While our work presents two simple posterior-prior pairs, there are a number of other possibilities that could be explored in future work.

Original: Our work also points to at least two interesting challenges for latent-variable models: can they exceed the performance of a strong autoregressive baseline?
Revised: Our work also points to at least two interesting challenges for latent-variable models: can they exceed the performance of a strong autoregressive baseline?

Original: and can the representations aid state-of-the-art models for downstream applications such as classiﬁcation?
Revised: and can the representations aid state-of-the-art models for downstream applications such as classiﬁcation?

Original: A D ERIVATION OF THE KL DIVERGENCE FOR SEQUENTIAL LATENT VARIABLES
Revised: A D ERIVATION OF THE KL DIVERGENCE FOR SEQUENTIAL LATENT VARIABLES

Original: [Equation]
Revised: [Equation]

Original: B D ERIVATION OF THE KL- DIVERGENCE BETWEEN AR(1) AND DIAGONAL G AUSSIAN , AND ITS LOWER - BOUND
Revised: B D ERIVATION OF THE KL- DIVERGENCE BETWEEN AR(1) AND DIAGONAL G AUSSIAN , AND ITS LOWER - BOUND

Original: [Equation]
Revised: [Equation]

Original: Noting the analytic form for the KL-divergence for two uni-variate Gaussian distributions:
Revised: Noting the analytic form for the KL-divergence for two uni-variate Gaussian distributions:

Original: [Equation]
Revised: [Equation]

Original: we now derive the lower-bound for KL-divergence.
Revised: we now derive the lower-bound for KL-divergence.

Original: Without loss of generality and to avoid clutter
Revised: Without loss of generality and to avoid clutter

Original: [Equation]
Revised: [Equation]

Original: we have assume the mean vector µ i has equal values in each dimension. ? .
Revised: we have assume the mean vector µ i has equal values in each dimension. ? .

Original: [Equation]
Revised: [Equation]

Original: Where f a ( x ) = ax − ln ( x ) − 1 .
Revised: Where f a ( x ) = ax − ln ( x ) − 1 .

Original: Using the fact that σ 2 i
Revised: Using the fact that σ 2 i

Original: =
Revised: =

Original: E
Revised: E

Original: [( z i − µ i ) 2 ]
Revised: [( z i − µ i ) 2 ]

Original: =
Revised: =

Original: E ( z 2 i ) − µ 2 i , the expectation inside the summation can be simpliﬁed as follows.
Revised: E ( z 2 i ) − µ 2 i , the expectation inside the summation can be simpliﬁed as follows.

Original: [Equation]
Revised: [Equation]

Original: Plugging this back gives us the following analytic form for the KL-divergence for the sequential latent variable z .
Revised: Plugging this back gives us the following analytic form for the KL-divergence for the sequential latent variable z .

Original: [Equation]
Revised: [Equation]

Original: C D ERIVATION OF THE LOWER - BOUND
Revised: C D ERIVATION OF THE LOWER - BOUND

Original: Removing non-negative quadratic terms involving µ i in equation 3 and expanding back f inside the summation yields
Revised: Removing non-negative quadratic terms involving µ i in equation 3 and expanding back f inside the summation yields

Original: [Equation]
Revised: [Equation]

Original: Consider f a ( x ) =
Revised: Consider f a ( x ) =

Original: ax − ln( x ) − 1 and its ﬁrst and second order derivatives, f (cid:48) a ( x ) =
Revised: ax − ln( x ) − 1 and its ﬁrst and second order derivatives, f (cid:48) a ( x ) =

Original: a − 1 x and f (cid:48)(cid:48) a ( x ) ≥ 0 .
Revised: a − 1 x and f (cid:48)(cid:48) a ( x ) ≥ 0 .

Original: Thus, f a is convex and obtains its minimum value of ln( a ) at x = a − 1 .
Revised: Thus, f a is convex and obtains its minimum value of ln( a ) at x = a − 1 .

Original: Substituting σ 21 = 1 − α 2 , σ 2 n = 1 − α 2 and σ 2 i = 1 − α 2 

Revised: Substituting σ 21 = 1 − α 2 , σ 2 n = 1 − α 2 and σ 2 i = 1 − α 2

Original: + α 2 yields the following lower-bound for the KL:
Revised: + α 2 yields the following lower-bound for the KL:

Original: [Equation]
Revised: [Equation]

Original: When using multi-dimensional z i at each timestep, the committed rate is the sum of the KL for each individual dimension:
Revised: When using multi-dimensional z i at each timestep, the committed rate is the sum of the KL for each individual dimension:

Original: [Equation]
Revised: [Equation]

Original: D I NDEPENDENT δ -VAE S
Revised: D I NDEPENDENT δ -VAE S

Original: The most common choice for variational families is to assume that the components of the posterior are independent, for example using a multivariate Gaussian with a diagonal covariance: q φ ( z | x ) =
Revised: The most common choice for variational families is to assume that the components of the posterior are independent, for example using a multivariate Gaussian with a diagonal covariance: q φ ( z | x ) =

Original: N ( z ; µ q ( x ) , σ q ( x )) .
Revised: N ( z ; µ q ( x ) , σ q ( x )) .

Original: When paired with a standard Gaussian prior, p ( z ) =
Revised: When paired with a standard Gaussian prior, p ( z ) =

Original: N ( z ; 0 , 1) , we can guarantee a committed information rate δ by constraining the mean and variance of the variational family (see Appendix C) :
Revised: N ( z ; 0 , 1) , we can guarantee a committed information rate δ by constraining the mean and variance of the variational family (see Appendix C) :

Original: [Equation]
Revised: [Equation]

Original: We can, thus, numerically solve
Revised: We can, thus, numerically solve

Original: [Equation]
Revised: [Equation]

Original: to obtain the feasible interval [ σ lq , σ uq ] where the above equation has a solution for µ q , and the committed rate δ .
Revised: to obtain the feasible interval [ σ lq , σ uq ] where the above equation has a solution for µ q , and the committed rate δ .

Original: Posterior parameters can thus be parameterised as:
Revised: Posterior parameters can thus be parameterised as:

Original: [Equation]
Revised: [Equation]

Original: Where φ parameterizes the data-dependent part of µ q ad σ q , which allow the rate to go above the designated lower-bound δ .
Revised: Where φ parameterizes the data-dependent part of µ q ad σ q , which allow the rate to go above the designated lower-bound δ .

Revised: We compare this model with the temporal version of δ -VAE discussed in the paper and report the results in Table 3.

Revised: While independent δ -VAE also prevents the posterior from collapsing to prior, its performance in density modeling lags behind temporal δ -VAE.

Revised: [Table]

Original: E A RCHITECTUER D ETAILS E. I MAGE MODELS
Revised: E A RCHITECTUER D ETAILS E. I MAGE MODELS

Original: In this section we provide the details of our architecture used in our experiments.
Revised: In this section we provide the details of our architecture used in our experiments.

Original: The overall architecture diagram is depicted in Fig.
Revised: The overall architecture diagram is depicted in Fig.

Original: To establish the anti-causal context for the inference network we ﬁrst reverse the input image and pad each spatial dimension by one before feeding it to the encoder.
Revised: To establish the anti-causal context for the inference network we ﬁrst reverse the input image and pad each spatial dimension by one before feeding it to the encoder.

Original: The output of the encoder is cropped and reversed again.
Revised: The output of the encoder is cropped and reversed again.

Original: As show in Fig.
Revised: As show in Fig.

Original: 5, this gives each pixel the anti-causal context (i.e., pooling information from its own value and future values).
Revised: 5, this gives each pixel the anti-causal context (i.e., pooling information from its own value and future values).

Original: We then apply average pooling to this representation to give us row-wise latent variables, on which the decoder network is conditioned. [Figure]
Revised: We then apply average pooling to this representation to give us row-wise latent variables, on which the decoder network is conditioned. avg. pool rows

Original: P i x e l
Revised: P i x e l

Original: CNN
Revised: CNN

Original: E n c o d e r            {} {4} {4,3,2} {4,3} {4,3} {4,3, 2,1} {4,3} {4,3, 2,1} {4} {4,3,2} z 
Revised: E n c o d e r            {} {4} {4,3,2} {4,3} {4,3} {4,3, 2,1} {4,3} {4,3, 2,1} {4} {4,3,2} z z2 P i x e l

Original: [Figure] P i
Revised: CNN

Original: [Figure] e l [Figure] e r 
Revised: D ec o d e r reverse pad c r o p a n d r e v e

Original: [Figure]
Revised: [Figure]

Original: The exact hyper-parameters of our network is detailed in Table 4.
Revised: The exact hyper-parameters of our network is detailed in Table 4.

Original: We used dropout only in our decoder and applied it the activations of the hidden units as well as the attention matrix.
Revised: We used dropout only in our decoder and applied it the activations of the hidden units as well as the attention matrix.

Original: As in (Vaswani et al., 2017), we used rectiﬁed linear units and layer normalization (Ba et al., 2016) after the multi-head attention layers.
Revised: As in (Vaswani et al., 2017), we used rectiﬁed linear units and layer normalization (Ba et al., 2016) after the multi-head attention layers.

Original: We found layer normalization to be essential for stabilizing training.
Revised: We found layer normalization to be essential for stabilizing training.

Original: For optimization we used the Adam optimizer (Kingma & Ba, 2014).
Revised: For optimization we used the Adam optimizer (Kingma & Ba, 2014).

Original: We used the learning rate schedule proposed in (Vaswani et al., 2017) with a few tweaks as in the formulae: 
Revised: We used the learning rate schedule proposed in (Vaswani et al., 2017) with a few tweaks as in the formulae: LR imagenet = [Equation]

Original: [Equation]
Revised: [Equation]

Revised: 5 d min( step num 0 .

Revised: 35 , step num × 4000 1 . 5 )

Original: We developed our code using Tensorﬂow (Abadi et al., 2016).
Revised: LR cifar= 0 . [Equation] h − 0 . 5 d min( step num 0 .

Revised: 35 , step num × 8000 1 . 5 )

Revised: LR ablation = 0 .

Revised: We use multi-dimensional latent variables per each timestep, with different slowness factors linearly spaced between a chosen interval.

Revised: For our ablation studies

Revised: [Table]

Revised: we chose corresponding hyperparameters of each method we compare against to target rates between 25-100 bits per image.

Revised: [Equation]

Revised: ah

Revised: [Equation]

Revised: do d z

Revised: [Table]

Revised: We developed our code using Tensorﬂow (Abadi et al., 2016).

Original: Our experiments on natural images were conducted on Google Cloud TPU accelerators.
Revised: Our experiments on natural images were conducted on Google Cloud TPU accelerators.

Original: For Imagenet, we used 128 TPU cores with batch size of 1024.
Revised: For Imagenet, we used 128 TPU cores with batch size of 1024.

Original: We used 8 TPU cores for CIFAR-10 with batch size of 64.
Revised: We used 8 TPU cores for CIFAR-10 with batch size of 64.

Original: E. T EXT M ODELS
Revised: E. T EXT M ODELS

Original: The architecture of our model for text experiment is closely based on the Transformer network of
Revised: The architecture of our model for text experiment is closely based on the Transformer network of

Original: Vaswani et al.
Revised: Vaswani et al.

Revised: We realize the encoder anti-causal structure by inverting the causal attention masks to upper triangular bias matrices.

Original: The exact hyper-parameters are summarized in Table 5.
Revised: The exact hyper-parameters are summarized in Table 5.

Original: l e /l
Revised: l h r ah d z α

Original: d h e /h
Revised: LM1B    0. F A BLATION S TUDIES For our ablation studies on CIFAR-10, we trained our model with the conﬁguration listed in Table 4. After training the model, we inferred the mean of the posterior distribution corresponding to each training example in the CIFAR-10 test set, and subsequently trained a multi-class logistic regression classiﬁer on top of it.

Original: d r e /r d a e /a d ah n dmol do d z α
Revised: For each model, the linear classiﬁer was optimized for 100 epochs using the Adam optimizer with the starting learning rate of 0 . 003 . The learning rate was decayed by a factor of 0 . 3 every 30 epochs.

Revised: We also report the rate-distortion curves for the CIFAR-10 training set in Fig.

Revised: In contrast to the graph of Fig.

Revised: 4a for the test set, δ -VAE achieves relatively higher negative log-likelihood compared to other methods on the training seen, especially for larger rates.

Revised: This suggests that δ -VAE is less prone to overﬁtting compared to β -VAE and free-bits.

Original: CIFAR20/ 128/ 1024/ 11/   0.
Revised: [Figure] F. E NCODER A BLATION In table Table 6, we report the details of evaluating our proposed anti-causal encoder architecture (discussed in Sect. 2.2) against the non-causal architecture in which there is no restriction in the connectivity of the encoder network. The reported experiments are conducted on the CIFAR-10 dataset.

Revised: We trained 4 different conﬁgurations of our model to provide comparison in different capacity and information rate regimes, using the temporal δ -VAE approach to prevent posterior collapse.

Original: Imagenet 32x 6/ 128/ 1024/ 2/   0.
Revised: We found that the anti-causal structure is beneﬁcial when the decoder has sufﬁciently large receptive ﬁeld, and also when encoding relatively high amount of information in latent variables. l =h =a =low-rate l =h =a =low-rate l =h =a =high-rate l e = 20 , h e =l d = 30 , h d =a =low-rate Non-Causal AR(1) 3.04 (0.02) 3.01 (0.03) 3.32 (0.22) 2.88 (0.05) Non-Causal Aux 3.03 (0.01) 2.98 (0.004) 3.11 (0.02) 2.85 (0.01)

Revised: Anti-Causal AR(1) 3.07 (0.02) 3.01 (0.03) 3.22 (0.22) 2.87 (0.05)

Revised: Anti-Causal Aux 3.

Original: [Table]
Revised: [Table]

Original: F V ISUALIZATION OF THE L ATENT S PACE
Revised: G V ISUALIZATION OF THE L ATENT S PACE

Original: It is generally expected that images from the same class are mapped to the same region of the latent space.
Revised: It is generally expected that images from the same class are mapped to the same region of the latent space.

Original: Fig.
Revised: Fig.

Original: 6 illustrates the t-SNE (van der Maaten & Hinton, 2008) plot of latent variables inferred from 3000 examples from the test set of CIFAR-10 colour coded based on class labels.
Revised: 7 illustrates the t-SNE (van der Maaten & Hinton, 2008) plot of latent variables inferred from 3000 examples from the test set of CIFAR-10 colour coded based on class labels.

Original: As can also be seen on the right hand plot classes that are closest are also mostly the one that have close semantic and often visual relationships (e.g., cat and dog, or deer and horse).
Revised: As can also be seen on the right hand plot classes that are closest are also mostly the one that have close semantic and often visual relationships (e.g., cat and dog, or deer and horse).

Original: G E NCODER A BLATIONS H A DDITIONAL S AMPLES
Revised: H A DDITIONAL S AMPLES

Original: =
Revised: =

Original: ===
Revised: ===

Original: Interpolating dimension 

Revised: Interpolating dimension

Original: == ==
Revised: == ==

Original: The company’s stock price is also up for a year-on-year rally,
Revised: The company’s stock price is also up for a year-on-year rally,

Original: when the
Revised: when the

Original: The company’s shares are trading at a record high for the year, when they were trading at The company’s shares were trading at $3.00, down from their 52-week low
Revised: The company’s shares are trading at a record high for the year, when they were trading at The company’s shares were trading at $3.00, down from their 52-week low

Original: The company’s shares fell $1.14, or 5.7 percent, to $ UNK
Revised: The company’s shares fell $1.14, or 5.7 percent, to $ UNK

Original: The company, which is based in New York, said it would cut 1,000 jobs in the
Revised: The company, which is based in New York, said it would cut 1,000 jobs in the

Original: The two-day meeting, held at the White House, was a rare opportunity for the United States The company, which is based in New York, said it was looking to cut costs, but added
Revised: The two-day meeting, held at the White House, was a rare opportunity for the United States The company, which is based in New York, said it was looking to cut costs, but added

Original: The company is the only company to have a signiﬁcant presence in China.
Revised: The company is the only company to have a signiﬁcant presence in China.

Original: The company is the only company to have a signiﬁcant presence in the North American market.
Revised: The company is the only company to have a signiﬁcant presence in the North American market.

Original: The two men, who were arrested, have been released.
Revised: The two men, who were arrested, have been released.

Original: =
Revised: =

Original: ===
Revised: ===

Original: Interpolating dimension {+
+}
Revised: Interpolating dimension {+
+}

Original: [Equation]

Original: = =
Revised: ==

Original: = =
Revised: ==

Original: In the meantime, however, the company is taking the necessary steps to keep the company in the UNK
Revised: In the meantime, however, the company is taking the necessary steps to keep the company in the UNK

Original: In the meantime, however, the company is expected to take some of the most aggressive steps in the In the meantime, the company is expected to report earnings of $2.15 to $4.
Revised: In the meantime, however, the company is expected to take some of the most aggressive steps in the In the meantime, the company is expected to report earnings of $2.15 to $4.

Original: The two men, who were both in their 20s , were arrested on suspicion of causing death by dangerous The company said it was ”disappointed” by a decision by the U.S. Food and Drug
Revised: The two men, who were both in their 20s , were arrested on suspicion of causing death by dangerous The company said it was ”disappointed” by a decision by the U.S. Food and Drug

Original: The company said it would continue to provide ﬁnancial support to its business and ﬁnancial services clients.
Revised: The company said it would continue to provide ﬁnancial support to its business and ﬁnancial services clients.

Original: Thenew plan would also provide a new national security dimension to U.S.- led efforts to ”I’ve always been a great customer and there´s always there´s a good chance ”It’s a great personal decision...
Revised: Thenew plan would also provide a new national security dimension to U.S.- led efforts to ”I’ve always been a great customer and there´s always there´s a good chance ”It’s a great personal decision...

Original: = ===
Revised: [Table]

Original: [-Interpolating dimension 
-]
Revised: [-Interpolating dimension 
-]

Original: == == ”I’ll
Revised: We are not [Table]

Original: be in the process of making the case he
Revised: to get any more [Table] on the situation,

Original: said

Original: ”I’ve
Revised: [Table]

Original: got to take the best possible shot at the top
Revised: the U. N. mission in Afghanistan, which is to be formally

Original: ” he said ”I’m not going to take any chances,” he said
Revised: We are not going to get the [Table] back,”

Original: ”I’m
Revised: [Table]

Original:  notgoing
Revised: We are not

Original: to take any chances,” he said. ”I’m not going to take any chances,” he said. We are not going to get any more information on the situation,” said a spokesman for the U. N. mission in Afghanistan, which is to be formally
Revised: [Table] [Table]

Original: We are not going to get the money back,” he said.
Revised: ”We [Table]

Original: We are not going to get the money back,” said one of the co - workers. ”We are notgoing to get the money back,” said the man. ” We are not going to get a lot of money back,” said the man.
Revised: not

Original: = ===
Revised: [Table]

Original: [-Interpolating dimension 
-]
Revised: [-Interpolating dimension 
-]

Original: === =
Revised: [Table]

Original: The company said the company, which employs more than 400 people, did not respond to requests for comment, but did not respond to an email seeking comment, which The new rules, which are expected to take effect in the coming weeks, will allow the government to take steps to ensure that the current system does not take too ”The only thing that could be so important is the fact that the government is not going to be able to get the money back, so the people are taking ”I’m not sure if the government will be able to do that,” he said.
Revised: The company said the company, which employs more than 400 people, did not [Table] to requests for comment, but did not respond to an email seeking comment, which The new rules, which are expected to take effect in the coming weeks, will allow the government to take steps to ensure that the current system does not take too ”The only thing that could be so important is the fact that the government is not going to be able to get the money back, so the people are taking ”I’m not sure if the government will be able to do that,” he said.

Original: ”We are not going to get any more information about the situation,” said Mr. O’Brien.
Revised: ”We are not [Table] [Table]

Original: It’s a very important thing to have a president who has a strong and strong relationship with our country,” said Mr. Obama, who has been the ”It’s a very important thing tohave a president who has a great chance to make a great president,” said
Revised: It’s a very [Table] thing to

Original: Mr. Obama, a former senator from ”It’s a very important decision,” said Mr. Obama.
Revised: [Table] a [Table]

Original: ”It’s a very difﬁcult decision to make,” said Mr. McCain.
Revised: relationship [Table] our country,” [Table]

Revised: [Table]

Revised: the ”It’s

Revised: [Table]

Revised: ,” said

Revised: [Table]

Revised: [Table]

Revised: ”It’s

Revised: [Table]

Revised: ”It’s a very

Revised: [Table]

Revised: to make,”

Original: [Table]
Revised: [Table]

Revised: sampled from the

Revised: auxliray

Revised: prior.

Revised: The

Revised: company

Original: The company is now the world’s cheapest for consumers .
Revised: is now the world’s cheapest for consumers .

Revised: The

Revised: company

Revised: is now

Revised: the

Original: The company is now the world’s biggest producer of oil andgas, with an estimated annual revenue of $2.2 billion.
Revised: world’s biggest producer of oil and

Revised: gas,

Revised: with

Revised: an

Revised: estimated annual revenue of

Revised: $

Revised: billion

Revised: The

Revised: company

Revised: is now

Original: The company is now the world’s third-largest producer of the drug, after Pﬁzer and AstraZeneca, whichis based in the UK.
Revised: the world’s third-largest producer of the drug, after Pﬁzer and AstraZeneca, which

Revised: is based in the UK

Revised: The

Original: The company is now the world’s biggest producer of the popular games console, with salesof more than $1bn (312m) in the US and about $3bn in the UK.
Revised: company is now the world’s biggest producer of the popular games console with sales

Revised: of more than $

Revised: 1bn

Revised: (312m

Revised: ) in

Original:  The company is now the world’s largest company, with over $7.5 billion in annual revenue in 2008, and has been in the past for more than two decades.
Revised: the US and about $ 3bn in the UK. The company is now the world’s largest company, with over $7.5 billion in annual revenue in 2008, and has been in the past for more than two decades.

Original: The company is now the world’s second-largest, after the cellphone company, which is dominated by the iPhone, which has the iPhone and the ability to store in - store, rather than having to buy, the product, said, because of the Apple-based device.
Revised: The company is now the world’s second-largest, after the cellphone company, which is dominated by the iPhone, which has the iPhone and the ability to store in - store, rather than having to buy, the product, said, because of the Apple-based device.

Original: The company is now the world’s biggest manufacturer of the door-to-door design for cars and the auto industry.
Revised: The company is now the world’s biggest manufacturer of the door-to-door design for cars and the auto industry.

Original: The company is now the world’s third-largest maker of commercial aircraft, behind Boeing and Airbus.
Revised: The company is now the world’s third-largest maker of commercial aircraft, behind Boeing and Airbus.

Original: The company is now the world’s largest producer of silicon, and one of the biggest producers of silicon in the world.
Revised: The company is now the world’s largest producer of silicon, and one of the biggest producers of silicon in the world.

Original: The company is now the world’s largest maker of computer -based software, with a market value of $4.2 billion (2.6 billion) and an annual turnover of $400 million (343 million).
Revised: The company is now the world’s largest maker of computer -based software, with a market value of $4.2 billion (2.6 billion) and an annual turnover of $400 million (343 million).