Original: Learning to Drop Out: An Adversarial Approach to Training Sequence VAEs
Revised: Learning to Drop Out: An Adversarial Approach to Training Sequence VAEs

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: In principle, applying variational autoencoders (VAEs) to sequential data offers amethod for controlled sequence generation, manipulation, and structured representation learning.
Revised: In principle, applying variational autoencoders (VAEs) to sequential data offers amethod for controlled sequence generation, manipulation, and structured representation learning.

Original: However, training sequence VAEs is challenging: autoregressivedecoders can often explain the data without utilizing the latent space, known asposterior collapse .
Revised: However, training sequence VAEs is challenging: autoregressivedecoders can often explain the data without utilizing the latent space, known asposterior collapse .

Original: To mitigate this, state-of-the-art models ‘weaken’ the ‘powerfuldecoder’ by applying uniformly random dropout to the decoder input.
Revised: To mitigate this, state-of-the-art models ‘weaken’ the ‘powerfuldecoder’ by applying uniformly random dropout to the decoder input.

Original: We show theoretically that this removes pointwise mutual information provided by the decoderinput, which is compensated for by utilizing the latent space.
Revised: We show theoretically that this removes pointwise mutual information provided by the decoderinput, which is compensated for by utilizing the latent space.

Original: We then proposean adversarial training strategy to achieve information-based stochastic dropout .
Revised: We then proposean adversarial training strategy to achieve information-based stochastic dropout .

Original: Compared to uniform dropout on standard text benchmark datasets, our targetedapproach increases both sequence modeling performance and the informationcaptured in the latent space.
Revised: Compared to uniform dropout on standard text benchmark datasets, our targetedapproach increases both sequence modeling performance and the informationcaptured in the latent space.

Original:  Introduction
Revised:  Introduction

Original: Training autoregressive models via maximum likelihood estimation (MLE) is a common strategyfor representing sequential data.
Revised: Training autoregressive models via maximum likelihood estimation (MLE) is a common strategyfor representing sequential data.

Original: Such autoregressive models obtain state-of-the-art in modeling text[Brown et al., 2020, Chowdhery et al., 2022, Zhang et al., 2022], speech [Oord et al., 2018, Conneauet al., 2020], and video sequences [Babaeizadeh et al., 2018].
Revised: Such autoregressive models obtain state-of-the-art in modeling text[Brown et al., 2020, Chowdhery et al., 2022, Zhang et al., 2022], speech [Oord et al., 2018, Conneauet al., 2021], and video sequences [Babaeizadeh et al., 2018].

Original: In this simple and intuitive approach,the joint distribution of a sequence is factorized into a product of conditional distributions, with eachsequence element conditioned on its history.
Revised: In this simple and intuitive approach,the joint distribution of a sequence is factorized into a product of conditional distributions, with eachsequence element conditioned on its history.

Original: However, in their basic form, autoregressive modelsdo not necessarily learn latent variables that reflect semantic content of the data, as often desired.
Revised: However, in their basic form, autoregressive modelsdo not necessarily learn latent variables that reflect semantic content of the data, as often desired.

Original: As such, supplementing autoregressive models with latent variables is a promising way to enhancecontrol in sequence generation, and enable structured representation learning.
Revised: As such, supplementing autoregressive models with latent variables is a promising way to enhancecontrol in sequence generation, and enable structured representation learning.

Original: A recent approach to latent variable modeling of sequences is to integrate autoregressive componentsinto the variational autoencoder (VAE) framework [Kingma and Welling, 2013, Rezende et al., 2014].
Revised: A recent approach to latent variable modeling of sequences is to integrate autoregressive componentsinto the variational autoencoder (VAE) framework [Kingma and Welling, 2014, Rezende et al., 2014].

Original: Sequence VAEs offer a theoretically principled solution, but have not been widely adopted.
Revised: Sequence VAEs offer a theoretically principled solution, but have not been widely adopted.

Original: Arguably,this is largely due to the phenomenon of posterior collapse [Bowman et al., 2016, Chen et al., 2017,
Revised: Arguably,this is largely due to the phenomenon of posterior collapse [Bowman et al., 2016, Chen et al., 2017,

Original: Van Den Oord et al., 2017, Li et al., 2020].
Revised: Van Den Oord et al., 2017, Li et al., 2020].

Original: Posterior collapse describes when a VAE’s posteriorprobability over latent variables ‘collapses’ to the latent prior, rendering the latent space completelyuninformative, or independent , of the data.
Revised: Posterior collapse describes when a VAE’s posteriorprobability over latent variables ‘collapses’ to the latent prior, rendering the latent space completelyuninformative, or independent , of the data.

Original: Recent years have seen an abundance of research intoposterior collapse
Revised: Recent years have seen an abundance of research intoposterior collapse

Original: [Bowman et al., 2015, Chen et al., 2017, Lucas et al., 2019, Dai et al., 2020, Fanget al., 2020, Pang et al., 2021], identifying the ‘power’ of autoregressive decoding as a main cause.
Revised: [Bowman et al., 2015, Chen et al., 2017, Lucas et al., 2019, Dai et al., 2020, Fanget al., 2021, Pang et al., 2021], identifying the ‘power’ of autoregressive decoding as a main cause.

Original: Inparticular, autoregressive decoders are shown to obtain satisfactory sequence modeling performancewithout utilizing the latent space [Bowman et al., 2016, Chen et al., 2017].
Revised: Inparticular, autoregressive decoders are shown to obtain satisfactory sequence modeling performancewithout utilizing the latent space [Bowman et al., 2016, Chen et al., 2017].

Original: This suggests that forsequence VAEs to progress, improved techniques are needed to understand and alleviate posteriorcollapse, and thereby learn informative latent structure.
Revised: This suggests that forsequence VAEs to progress, improved techniques are needed to understand and alleviate posteriorcollapse, and thereby learn informative latent structure.

Original: In this work, we propose a theoretically principled way to mitigate posterior collapse when training autoregressive sequence VAEs.
Revised: In this work, we propose a theoretically principled way to mitigate posterior collapse when training autoregressive sequence VAEs.

Original: The intuition that autoregressive VAE decoders are too ‘powerful’ has led to the notion of ‘weakening’them, e.g. by regularization.
Revised: The intuition that autoregressive VAE decoders are too ‘powerful’ has led to the notion of ‘weakening’them, e.g. by regularization.

Original: However, weakening an autoregressive model arbitrarily might of courseharm its performance, leading to a trade off between informativeness of the latent space and thequality of sequence modeling (or density estimation performance).
Revised: However, weakening an autoregressive model arbitrarily might of courseharm its performance, leading to a trade off between informativeness of the latent space and thequality of sequence modeling (or density estimation performance).

Original: State-of-the-art VAEs applyvariants of dropout [Srivastava et al., 2014] to the sequence input into the decoder: either to individualdimensions of sequence elements [Kim et al., 2018], or to mask entire sequence elements [Iyyeret al., 2015, Bowman et al., 2016].
Revised: State-of-the-art VAEs applyvariants of dropout [Srivastava et al., 2014] to the sequence input into the decoder: either to individualdimensions of sequence elements [Kim et al., 2018], or to mask entire sequence elements [Iyyeret al., 2015, Bowman et al., 2016].

Original: In either case, dropout is generally applied uniformly at random,meaning that each sequence element has an equal probability of being dropped.
Revised: In either case, dropout is generally applied uniformly at random,meaning that each sequence element has an equal probability of being dropped.

Original: From the intuitionthat different words carry different information about the next word, this work investigates whethera non-uniform dropout policy can achieve a better trade-off between capturing latent information andsequence modeling performance.
Revised: From the intuitionthat different words carry different information about the next word, this work investigates whethera non-uniform dropout policy can achieve a better trade-off between capturing latent information andsequence modeling performance.

Original: Specifically, by stochastically dropping each sequence elementaccording to its importance to the autoregressive decoder, our method takes a targeted approach todropout relative to uniform sampling, maintaining sequence modeling performance while achievingthe benefits of dropout on the latent space.
Revised: Specifically, by stochastically dropping each sequence elementaccording to its importance to the autoregressive decoder, our method takes a targeted approach todropout relative to uniform sampling, maintaining sequence modeling performance while achievingthe benefits of dropout on the latent space.

Original: To apply a non-uniform, data-dependent dropout scheme,our approach introduces an ‘adversary’ that learns which elements to drop .
Revised: To apply a non-uniform, data-dependent dropout scheme,our approach introduces an ‘adversary’ that learns which elements to drop .

Original: The proposed frameworkis represented in Figure 1, theoretically developed in Section 4.1 and implemented in Section 4.2.
Revised: The proposed frameworkis represented in Figure 1, theoretically developed in Section 4.1 and implemented in Section 4.2.

Original: The contributions of this paper can be summarized as follows: (i)
Revised: The contributions of this paper can be summarized as follows: (i)

Original: We propose an adversarial approachto learning a stochastic dropout policy that mitigates posterior collapse in sequence VAEs.
Revised: We propose an adversarial approachto learning a stochastic dropout policy that mitigates posterior collapse in sequence VAEs.

Original: (ii) Weshow theoretically that dropping out sequence elements deducts mutual information not alreadylearned by the model , between one sequence element and the next, and that our approach maximisesthis quantity, leading to a constrained minimax ELBO objective that explains an adversarial approach.(iii)
Revised: (ii) Weshow theoretically that dropping out sequence elements deducts mutual information not alreadylearned by the model , between one sequence element and the next, and that our approach maximisesthis quantity, leading to a constrained minimax ELBO objective that explains an adversarial approach.(iii)

Original: We evaluate the proposed scheme on standard text benchmarks, showing that our approachrenders an informative latent space, without trading-off but rather improving sentence modeling.
Revised: We evaluate the proposed scheme on standard text benchmarks, showing that our approachrenders an informative latent space, without trading-off but rather improving sentence modeling.

Original: (iv)
Revised: (iv)

Original: We examine properties of the adversarial network and find that the adversary typically selects wordsthat carry sentence semantics that fit our theoretical analysis.
Revised: We examine properties of the adversarial network and find that the adversary typically selects wordsthat carry sentence semantics that fit our theoretical analysis.

Original:  Background
Revised:  Background

Original: Here, we formally describe the use of autoregressive decoding within VAEs for sequential data andthe problem of posterior collapse that often arises.
Revised: Here, we formally describe the use of autoregressive decoding within VAEs for sequential data andthe problem of posterior collapse that often arises.

Original: For notation: X denotes a random variable,(boldface) x denotes a sequence that is a realization of X , (subscript) x i denotes the i -th sequenceelement of x and x <i collectively denotes all sequence elements prior x i .
Revised: For notation: X denotes a random variable, (boldface) x denotes a sequence that is a realization of X , (subscript) x i denotes the i -th sequence element of x and x <i collectively denotes all sequence elements prior x i .

Original: z denotes a global latentvector and (superscript) x
Revised: z denotes a global latent vector and (superscript) x

Original: ( i ) denotes the i -th sample from a dataset.
Revised: ( i ) denotes the i -th sample froma dataset.

Original: The task is to approximate the true probability density function p ∗ ( x ) of a random sequence X ∼p ∗ ( x ) using a model p θ ( x ) with parameters θ .
Revised: The task is to approximate the true probability density function p ∗ ( x ) of a random sequence X ∼p ∗ ( x ) using a model p θ ( x ) with parameters θ .

Original: Given a set of observations { x ( i ) } Ni =1 , θ can beestimated by maximum likelihood estimation (MLE):
Revised: Given a set of observations { x ( i ) } Ni =1 , θ can beestimated by maximum likelihood estimation (MLE):

Original: [Equation]
Revised: [Equation]

Original: If the data has latent generative factors z , modelled as p θ ( x )=
Revised: If the data has latent generative factors z , modelled as p θ ( x )=

Original: (cid:82) z p θ ( x | z ) p θ ( z ) , Eq 1 is intractableand one can instead maximize the evidence lower bound (ELBO), which for any observation x is:
Revised: (cid:82) z p θ ( x | z ) p θ ( z ) , Eq 1 is intractableand one can instead maximize the evidence lower bound (ELBO), which for any observation x is:

Original: [Equation]
Revised: [Equation]

Original: where, q ϕ ( z | x ) approximates the posterior, parameterized by ϕ , and p θ ( z ) is a prior distributionover latent variables.
Revised: where, q ϕ ( z | x ) approximates the posterior, parameterized by ϕ , and p θ ( z ) is a prior distributionover latent variables.

Original: In this work, we consider the framework of variational autoencoders (VAEs)[Kingma
Revised: In this work, we consider the framework of variational autoencoders (VAEs)[Kingma

Original: and Welling, 2013, Rezende et al., 2014], where the encoder q ϕ ( z | x ) and decoder p θ ( x | z )are parameterised by neural networks and p θ ( z ) is a standard Gaussian, denoted p 0 ( z ) .
Revised: and Welling, 2014, Rezende et al., 2014], where the encoder q ϕ ( z | x ) and decoder p θ ( x | z )are parameterised by neural networks and p θ ( z ) is a standard Gaussian, denoted p 0 ( z ) .

Original: Further, wefocus specifically on VAEs where the decoder p θ ( x | z ) is autoregressive [Bowman et al., 2016, Liet al., 2020,
Revised: Further, wefocus specifically on VAEs where the decoder p θ ( x | z ) is autoregressive [Bowman et al., 2016, Liet al., 2020,

Original: He et al., 2019], meaning log p θ ( x | z )
Revised: He et al., 2019], meaning log p θ ( x | z )

Original: = (cid:80) i log p θ ( x i | x <i , z ) , and the ELBO becomes:ELBO AR
Revised: = (cid:80) i log p θ ( x i | x <i , z ) , and the ELBO becomes:

Original: [Equation]
Revised: [Equation]

Original: [Equation] = [Equation] Posterior collapse: Autoregressive models improve density estimation [Oord et al., 2016] byexplicitly modeling statistical dependencies between sequence elements.
Revised: Posterior collapse: Autoregressive models improve density estimation [Oord et al., 2016] byexplicitly modeling statistical dependencies between sequence elements.

Original: However, when used as a
Revised: However, when used as a

Original: VAE decoder p θ ( x | z ) , autoregressive models are often found to ignore z , i.e. p ( x | z ) ≈ p ( x ) , and theposterior is said to ‘collapse’ to the prior, i.e. q θ ( z | x ) ≈ p 0 ( z ) , rendering z de facto independent of x[Bowman et al., 2016, He et al., 2019].
Revised: VAE decoder p θ ( x | z ) , autoregressive models are often found to ignore z , i.e. p ( x | z ) ≈ p ( x ) , and theposterior is said to ‘collapse’ to the prior, i.e. q θ ( z | x ) ≈ p 0 ( z ) , rendering z de facto independent of x[Bowman et al., 2016, He et al., 2019].

Original: In terms of Eq 3, this implies p θ ( x i | x <i , z )
Revised: In terms of Eq 3, this implies p θ ( x i | x <i , z )

Original: ≈
Revised: ≈

Original: p θ ( x i | x <i ) .
Revised: p θ ( x i | x <i ) .

Original: From this, posterior collapse can be considered due to the information about x i provided by x <ibeing such that each x i is (approximately) conditionally independent of z given x <i , in other wordsz is essentially redundant .
Revised: From this, posterior collapse can be considered due to the information about x i provided by x <ibeing such that each x i is (approximately) conditionally independent of z given x <i , in other wordsz is essentially redundant .

Original: Based on this, we interpret the notion that autoregressive models are toopowerful for use as a VAE decoder to mean that they allow too much information flow .
Revised: Based on this, we interpret the notion that autoregressive models are toopowerful for use as a VAE decoder to mean that they allow too much information flow .

Original:  Related Work
Revised:  Related Work

Original: Posterior collapse The phenomenon of posterior collapse has been observed in the context of text[Bowman et al., 2016, Yang et al., 2017], images [Chen et al., 2017, Razavi et al., 2018], videos[Babaeizadeh et al., 2018], speech [Chorowski et al., 2019] and graphs [Kipf et al., 2018].
Revised: Posterior collapse The phenomenon of posterior collapse has been observed in the context of text[Bowman et al., 2016, Yang et al., 2017], images [Chen et al., 2017, Razavi et al., 2018], videos[Babaeizadeh et al., 2018], speech [Chorowski et al., 2019] and graphs [Kipf et al., 2018].

Original: In broadterms, previous solutions can be divided into two complementary categories: (i) latent-variableoriented methods that focus mainly on relaxing the KL penalty; and (ii) decoder-oriented methodsthat regularise or ‘weaken’ the autoregressive decoder.
Revised: In broadterms, previous solutions can be divided into two complementary categories: (i) latent-variableoriented methods that focus mainly on relaxing the KL penalty; and (ii) decoder-oriented methodsthat regularise or ‘weaken’ the autoregressive decoder.

Original: Typically, both types are required in order tolearn an informative latent space [Bowman et al., 2016, Chen et al., 2017, Kim et al., 2018].
Revised: Typically, both types are required in orderto learn an informative latent space [Bowman et al., 2016, Chen et al., 2017, Kim et al., 2018].

Original: Latent-variable-oriented solutions Bowman et al.
Revised: Latent-variable-oriented solutions Bowman et al.

Original: [2016] suggest KL annealing , a technique thatintroduces the KL term gradually into training according to a predefined schedule.
Revised: [2016] suggest KL annealing , a technique thatintroduces the KL term gradually into training according to a predefined schedule.

Original: Free bits [Kingmaet al., 2016] prevent penalizing of the KL term if its magnitude is below a predefined threshold.
Revised: Free bits [Kingmaet al., 2016] prevent penalizing of the KL term if its magnitude is below a predefined threshold.

Original: Van
Revised: Van

Original: Den Oord et al.
Revised: Den Oord et al.

Original: [2017] use a discrete VAE, avoiding posterior collapse by design.
Revised: [2017] use a discrete VAE, avoiding posterior collapse by design.

Original: Fu et al.
Revised: Fu et al.

Original: [2019]apply a cyclical form of KL annealing.
Revised: [2019]apply a cyclical form of KL annealing.

Original: Razavi et al.
Revised: Razavi et al.

Original: [2018] constrain the variational family of theposterior distribution (encoder), preventing it from closely approximating the prior and so holding the
Revised: [2018] constrain the variational family of theposterior distribution (encoder), preventing it from closely approximating the prior and so holding the

Original: KL term away from zero.
Revised: KL term away from zero.

Original: Lagging inference networks [He et al., 2019] ‘aggressively’ optimize theencoder before each decoder update.
Revised: Lagging inference networks [He et al., 2019] ‘aggressively’ optimize theencoder before each decoder update.

Original: A similar procedure is followed by Semi-Amortized VAEs [Kimet al., 2018].
Revised: A similar procedure is followed by Semi-Amortized VAEs [Kimet al., 2018].

Original: Generative skip models [Dieng et al., 2019] introduce skip connections to create a moreexplicit link between the latent variables and the likelihood function.
Revised: Generative skip models [Dieng et al., 2019] introduce skip connections to create a moreexplicit link between the latent variables and the likelihood function.

Original: Sinha and Dieng [2021] proposea consistency regularizer by minimizing the KL divergence between the posterior approximations ofan observation and a random transformation of it.
Revised: Sinha and Dieng [2021] proposea consistency regularizer by minimizing the KL divergence between the posterior approximations ofan observation and a random transformation of it.

Original: Decoder-oriented solutions Bowman et al.
Revised: Decoder-oriented solutions Bowman et al.

Original: [2016] apply word dropout [Iyyer et al., 2015] touniformly drop words during autoregressive decoding.
Revised: [2016] apply word dropout [Iyyer et al., 2015] to uniformly drop words during autoregressive decoding.

Original: Other methods [Kim et al., 2018, He et al. ,2019] apply parameter dropout
Revised: Other methods [Kim et al., 2018, He et al., 2019] apply parameter dropout

Original: [Srivastava et al., 2014] to word embeddings.
Revised: [Srivastava et al., 2014]to word embeddings.

Original: Chen et al.
Revised: Chen et al.

Original:  Semeniuta et al.
Revised: [2017], Semeniuta et al.

Original: [2017], Yang et al.
Revised: [2017], Yang et al.

Original: [2017] constrain the receptive field of the decoder, limiting thewindow of autoregression, however, this is not readily applicable to RNN architectures given theirunbounded receptive field.
Revised: [2017] constrain thereceptive field of the decoder, limiting the window of autoregression, however, this is not readilyapplicable to RNN architectures given their unbounded receptive field.

Original: Our proposed adversarial method falls into this category as an extensionof word dropout, which is then subsumed as a special case of adversarial word dropout (Section 4.2).
Revised: Our proposed adversarialmethod falls into this category as an extension of word dropout, which is then subsumed as a specialcase of adversarial word dropout (Section 4.2).

Original: Non-uniform dropout Previous work has recognized the benefit of adapting the dropout rate acrossdifferent architectural components, though in entirely different contexts [Kingma et al., 2015, Galet al., 2017, Achille and Soatto, 2018].
Revised: Non-uniform dropout Previous work has recognized the benefit of adapting the dropout rate acrossdifferent architectural components, though in entirely different contexts [Kingma et al., 2015, Galet al., 2017, Achille and Soatto, 2018].

Original: These works introduce different approaches to regularizingthe magnitude of weights or activations (groups of weights).
Revised: These works introduce different approaches to regularizingthe magnitude of weights or activations (groups of weights).

Original: Elements that are deemed irrelevantduring training are then dropped.
Revised: Elements that are deemed irrelevantduring training are then dropped.

Original: For instance, variational dropout [Kingma et al., 2015] has beenused to prune weights in deep neural networks [Molchanov et al., 2017].
Revised: For instance, variational dropout [Kingma et al., 2015] has beenused to prune weights in deep neural networks [Molchanov et al., 2017].

Original: A key difference to ourapproach is that instead of a regularization term, we employ an adversary that selects elements fordropout during training based on their information content not yet learned by the model.
Revised: A key difference to ourapproach is that instead of a regularization term, we employ an adversary that selects elements fordropout during training based on their information content not yet learned by the model.

Original: Exposure bias The method by which the autoregressive decoder is trained, known as teacherforcing , bears a connection to exposure bias , which refers to the gap between training and inference[Ranzato et al., 2015].
Revised: Exposure bias The method by which the autoregressive decoder is trained, known as teacherforcing , bears a connection to exposure bias , which refers to the gap between training and inference[Ranzato et al., 2015].

Original: Namely, in language generation, a trained model generates sequences at testtime without access to the ground truth history that was accessible during training.
Revised: Namely, in language generation, a trained model generates sequences at testtime without access to the ground truth history that was accessible during training.

Original: Since the modelwas not trained to continue its predictions, this can lead to error accumulation [Ranzato et al., 2015].
Revised: Since the modelwas not trained to continue its predictions, this can lead to error accumulation [Ranzato et al., 2015].

Original: The most common approach to tackle exposure bias is to switch between conditioning on groundtruth and model predictions, with the latter being preferred towards the end of training [Daumé et al.
Revised: The most common approach to tackle exposure bias is to switch between conditioning on groundtruth and model predictions, with the latter being preferred towards the end of training [Daumé et al.

Original: ,2009, Bengio et al., 2015, Ranzato et al., 2015].
Revised: ,2009, Bengio et al., 2015, Ranzato et al., 2015].

Original: In preliminary studies, we implemented scheduledsampling [Bengio et al., 2015] but did not find it to outperform even uniform dropout.
Revised: In preliminary studies, we implemented scheduledsampling [Bengio et al., 2015] but did not find it to outperform even uniform dropout.

Original:  Adversarial Word Dropout 
Revised:  Adversarial Word Dropout (AWD)

Original: (AWD)

Original: 4.1 Theoretical Basis
Revised: 4.1 Theoretical Basis

Original: As a precursor to our adversarial approach, we first derive the effect of stochastic dropout on the
Revised: As a precursor to our adversarial approach, we first derive the effect of stochastic dropout on the

Original: ELBO under the autoregressive assumption (Equation 3).
Revised: ELBO under the autoregressive assumption (Equation 3).

Original: Word dropout: Our interpretation of posterior collapse at the end of Section 2 suggests that to‘weaken the decoder’ one should restrict the available information , such that z is no longer redundantand must capture some of the restricted information.
Revised: Word dropout: Our interpretation of posterior collapse at the end of Section 2 suggests that to‘weaken the decoder’ one should restrict the available information , such that z is no longer redundantand must capture some of the restricted information.

Original: Word dropout [Bowman et al., 2016] can be seento do this by stochastically masking the previous word x i − 1 with probability
Revised: Word dropout [Bowman et al., 2016] can be seento do this by stochastically masking the previous word x i − 1 with probability

Original: d i , which substitutesp θ ( x i | x <i , z ) with p θ ( x i | x <i − 1 , z ) in Equation 3, i.e.:
Revised: d i , which substitutesp θ ( x i | x <i , z ) with p θ ( x i | x <i − 1 , z ) in Equation 3, i.e.:

Original: [Equation]
Revised: [Equation]

Original: Equation 5 shows the effect (in expectation) of applying word dropout when maximising the ELBOwith an autoregressive decoder in Eq 3 (of which ‘KL’ denotes the last term).
Revised: Equation 5 shows the effect (in expectation) of applying word dropout when maximising the ELBOwith an autoregressive decoder in Eq 3 (of which ‘KL’ denotes the last term).

Original: In effect, a weighted sumis introduced with weights given by dropout probabilities d i and components that define conditionalpoint-wise mutual information (PMI), where classic point-wise mutual information between twovariables is defined as PMI ( x 1 , x 2 ) =
Revised: In effect, a weighted sumis introduced with weights given by dropout probabilities d i and components that define conditionalpoint-wise mutual information (PMI), where classic point-wise mutual information between twovariables is defined as PMI ( x 1 , x 2 ) =

Original: p ( x 1 | x 2 ) p ( x 1 ) .
Revised: p ( x 1 | x 2 ) p ( x 1 ) .

Original: Each PMI term in Eq 5 captures the informationthat one sequence element x i − 1 has regarding the next x i , over and above any information
Revised: Each PMI term in Eq 5 captures the informationthat one sequence element x i − 1 has regarding the next x i , over and above any information

Original: fromearlier elements x <i − 1 and the latent state z .
Revised: fromearlier elements x <i − 1 and the latent state z .

Original: Since the dropout-adapted ELBO (Eq 5) is maximisedw.r.t.
Revised: Since the dropout-adapted ELBO (Eq 5) is maximisedw.r.t.

Original: θ and ϕ , the weighted PMI term is minimised, which is only achievable by increasing theinformation learned by the model : extracted from x <i − 1 or captured in z .
Revised: θ and ϕ , the weighted PMI term is minimised, which is only achievable by increasing theinformation learned by the model : extracted from x <i − 1 or captured in z .

Original: Interestingly this shows word  quite  the  intuition  decoder  restrictinginformation . 
Revised: Interestingly this showsthat word dropout quite literally follows the earlier intuition and ‘weakens’ the decoder by restrictinginformation . Thus, word dropout leads to a looser variational objective, i.e. ELBO WD θ,ϕ ( x ) ≤ ELBO AR θ,ϕ ( x ) . Rainforth et al.

Revised: [2018] has argued that tighter bounds are not necessarily better.

Revised: Hence,our analysis provides a justification for using word dropout.

Original: The minimax objective: Under uniform dropout, all d i are equal over a sequence, and usingdropout to ‘push information into z ’ is applied evenly.
Revised: The minimax objective: Under uniform dropout, all d i are equal over a sequence, and usingdropout to ‘push information into z ’ is applied evenly.

Original: However, some sequence elements may holdmore information than others, e.g. in language, given syntactic rules and a summary of the previous text, some words may be highly predictable without specifically knowing which word preceded them,whereas others may depend heavily on their predecessor despite the other information.
Revised: However, some sequence elements may holdmore information than others, e.g. in language, given syntactic rules and a summary of the previoustext, some words may be highly predictable without specifically knowing which word preceded them,whereas others may depend heavily on their predecessor despite the other information.

Original: This suggeststhat information content may be non-uniform and so applying dropout non-uniformly may be moreappropriate.
Revised: This suggeststhat information content may be non-uniform and so applying dropout non-uniformly may be moreappropriate.

Original: We, therefore, propose targeted dropout of elements according to their ‘incremental’information content defined by the PMI terms, i.e. that not yet learned by the model.
Revised: We, therefore, propose targeted dropout of elements according to their ‘incremental’information content defined by the PMI terms, i.e. that not yet learned by the model.

Original: Further, sincesuch incremental information content is not explicitly computed and will vary over training as themodel learns, we train an adaptive dropout schedule to continually maximise the information dropped,and so minimize Eq 5, with respect to d i .
Revised: Further, sincesuch incremental information content is not explicitly computed and will vary over training as themodel learns, we train an adaptive dropout schedule to continually maximise the information dropped,and so minimize Eq 5, with respect to d i .

Original: Since Eq 5 is maximised with respect to all other parameters,this leads to a minimax objective.
Revised: Since Eq 5 is maximised with respect to all other parameters,this leads to a minimax objective.

Original: We introduce the two ‘players’ in the proposed minimax setting: VAE ϕ,θ , trained to maximizethe ELBO, and the adversary A ψ trained to do the opposite by determining each probability d i ofdropping out the previous sequence element x i − 1 , preventing it from helping to predict its successorx i .
Revised: We introduce the two ‘players’ in the proposed minimax setting: VAE ϕ,θ , trained to maximizethe ELBO, and the adversary A ψ trained to do the opposite by determining each probability d i ofdropping out the previous sequence element x i − 1 , preventing it from helping to predict its successorx i .

Original: The full objective for an observed sequence is given by
Revised: The full objective for an observed sequence is given by

Original: [Equation]
Revised: [Equation]

Original: ) where R ψ ( x ) is a regularization term explained below (Section 4.2) and
Revised: where R ψ ( x ) is a regularization term explained below (Section 4.2) and

Original: L ψ,ϕ,θ =

Original: E q ϕ ( z | x )

Original: [Equation]
Revised: [Equation]

Original: L ψ,ϕ,θ ( x ) is a modified version of the ELBO in Equation 5 with the additional parameters ψ of theadversary A ψ .
Revised: L ψ,ϕ,θ ( x ) is a modified version of the ELBO in Equation 5 with the additional parameters ψ of theadversary A ψ .

Original: The adversary manifests via the masking operator mask K,ψ ( · ; x ) , conditioned on theentire input sequence x .
Revised: The adversary manifests via the masking operator mask K,ψ ( · ; x ) , conditioned on theentire input sequence x .

Original: Since unconstrained minimisation of Eq 7 w.r.t ψ would cause all elementsto be dropped out, a constraint K is introduced such that exactly K elements of x are dropped duringautoregressive decoding.
Revised: Since unconstrained minimisation of Eq 7 w.r.t ψ would cause all elementsto be dropped out, a constraint K is introduced such that exactly K elements of x are dropped duringautoregressive decoding.

Original: In practice, K is sequence length-dependent, treated as a hyperparameterthat controls the level of adversarial regularization.
Revised: In practice, K is sequence length-dependent, treated as a hyperparameterthat controls the level of adversarial regularization.

Original: The adversary plays no role at test time.
Revised: The adversary plays no role at test time.

Original: 4.2 Implementation
Revised: 4.2 Implementation

Original: A high-level overview of our framework is shown in Figure 1.
Revised: A high-level overview of our framework is shown in Figure 1.

Original: An encoder maps the input sequenceinto a hidden representation to produce the parameters of q ϕ ( z | x ) , from which z is sampled usingthe ‘reparametrization trick’ [Kingma and Welling, 2013, Rezende et al., 2014].
Revised: An encoder maps the input sequenceinto a hidden representation to produce the parameters of q ϕ ( z | x ) , from which z is sampled usingthe ‘reparametrization trick’ [Kingma and Welling, 2014, Rezende et al., 2014].

Original: We implement thedecoder using a unidirectional LSTM [Hochreiter and Schmidhuber, 1997] for consistency with the RNN-VAE network of Bowman et al.
Revised: We implement thedecoder using a unidirectional LSTM [Hochreiter and Schmidhuber, 1997] for consistency withthe RNN-VAE network of Bowman et al.

Original: [2016], but other distributional approximators can be used thatneed not be autoregressive, e.g. attention-based transformer networks [Vaswani et al., 2017] are apromising choice to explore in future work.The autoregressive decoder is also implemented using aunidirectional recurrent network.
Revised: [2016], because LSTMs are the most popular backbonefor sequence VAEs and to facilitate comparison with prior work.

Original: Inspired by the recent work of [Dieng et al., 2019], we choose a Double-LSTM recurrent unit that elucidates the skip-connections to promote higher latent informationcontent flow (see Appendix ?? for details).
Revised: Inspired by the recent work of[Dieng et al., 2019], we choose a Double-LSTM recurrent unit that elucidates the skip-connections topromote higher latent information content flow (see Appendix A for details).

Original: As described in Section 4.1, the main innovation of our proposed method is the adversarial trainingprocedure: instead of dropping out decoder inputs uniformly at random, we introduce a trainableadversary to drop words most important for the VAE to reconstruct the original sequence.
Revised: As described in Section 4.1, the main innovation of our proposed method is the adversarial trainingprocedure: instead of dropping out decoder inputs uniformly at random, we introduce a trainableadversary to drop words most important for the VAE to reconstruct the original sequence.

Original: Belowwe describe implementation details of the adversary, specifying the relevant components in Figure 1.
Revised: Below wedescribe implementation details of the adversary, specifying the relevant components in Figure 1.

Original: A more comprehensive figure specific to the implemented architecture is provided in Appendix ?? . Note that the theoretical framework in Section 4.1 permits many alternative approaches for generatingper word dropout probabilities
Revised: Amore comprehensive figure specific to the implemented architecture is provided in Appendix B. Notethat the theoretical framework in Section 4.1 permits many alternative approaches for generatingper word dropout probabilities

Original: d i , such as attention mechanisms mentioned above.
Revised: d i , such as attention mechanisms mentioned above.

Original: We leave furtherrefinement of the adversarial architecture to future work.
Revised: We leave furtherrefinement of the adversarial architecture to future work.

Original: Producing dropout scores To determine which elements to drop out of a sequence, the adversary
Revised: Producing dropout scores To determine which elements to drop out of a sequence, the adversary

Original: A ψ samples a score s i ∈ R for each sequence element x i .
Revised: A ψ samples a score s i ∈ R for each sequence element x i .

Original: Sequence elements with the K smallestscores are dropped.
Revised: Sequence elements with the K smallestscores are dropped.

Original: The scores are sampled from distributions p ψ ( s i | x ) conditioned on the inputsequence x , modelled as a series of Gaussians with mean ( µ ) and variance ( σ ) parameterised by theoutputs of a unidirectional LSTM.
Revised: The scores are sampled from distributions p ψ ( s i | x ) conditioned on the inputsequence x , modelled as a series of Gaussians with mean ( µ ) and variance ( σ ) parameterised by theoutputs of a unidirectional LSTM.

Original: Scores are generated using the reparameterization trick, reducinggradient variance [Kingma and Welling, 2013].
Revised: Scores are generated using the reparameterization trick, reducinggradient variance [Kingma and Welling, 2014].

Original: Thus, p ψ ( s i | x ) is described ass i | x ∼ N ( µ i , σ i ; x , ψ ) where, [ µ i , σ i ] =
Revised: Thus, p ψ ( s i | x ) is described ass i | x ∼ N ( µ i , σ i ; x , ψ ) where, [ µ i , σ i ] =

Original: Linear ψ ( LSTM i,ψ ( x ))
Revised: Linear ψ ( LSTM i,ψ ( x ))

Original: The stochastic generation of scores gives the adversary an ‘exploratory’ capability during training,e.g. preventing it becoming ‘stuck’ on a set of sequence elements with high information content that the model cannot learn, meaning that their PMI terms (in Eq 5) remain large and the adversaryrepeatedly selects them to minimise the ELBO.
Revised: The stochastic generation of scores gives the adversary an ‘exploratory’ capability during training,e.g. preventing it becoming ‘stuck’ on a set of sequence elements with high information contentthat the model cannot learn, meaning that their PMI terms (in Eq 5) remain large and the adversaryrepeatedly selects them to minimise the ELBO.

Original: Top-K word selection Based on the sampled scores s = { s i } Ti =1 , the subset of K words with thesmallest scores are masked during decoding.
Revised: Top-K word selection Based on the sampled scores s = { s i } Ti =1 , the subset of K words with thesmallest scores are masked during decoding.

Original: To estimate gradients of the objective with respectto the parameters ψ , we use a stochastic softmax trick from Paulus et al.
Revised: To estimate gradients of the objective with respectto the parameters ψ , we use a stochastic softmax trick from Paulus et al.

Original: [2020]: during training,the stochastic subset selected in the forward pass is relaxed to admit a (biased) reparameterizationgradient in the backward pass.
Revised: [2020]: during training,the stochastic subset selected in the forward pass is relaxed to admit a (biased) reparameterizationgradient in the backward pass.

Original: In our experiments, we use a straight-through variant of the trick forour method to resemble word dropout, i.e. to produce discrete values ∈{ 0 , 1 } .
Revised: In our experiments, we use a straight-through variant of the trick forour method to resemble word dropout, i.e. to produce discrete values ∈{ 0 , 1 } .

Original: The relaxation is onlyused in the backward pass to compute the gradient estimator.
Revised: The relaxation is onlyused in the backward pass to compute the gradient estimator.

Original: Gradient reversal
Revised: Gradient reversal

Original: The final component of the adversarial network is the gradient reversal layer[Ganin et al., 2016].
Revised: The final component of the adversarial network is the gradient reversal layer[Ganin et al., 2016].

Original: In the forward pass, the layer performs no transformation to the input; in thebackward pass, the gradients are negated.
Revised: In the forward pass, the layer performs no transformation to the input; in thebackward pass, the gradients are negated.

Original: Gradient reversal offers a computationally simple methodto ensure that the parameters ψ of A ψ are updated such that ELBO is minimized.
Revised: Gradient reversal offers a computationally simple methodto ensure that the parameters ψ of A ψ are updated such that ELBO is minimized.

Original: If I is an identitymatrix, the ‘pseudo-function’ of gradient reversal can be described as
Revised: If I is an identitymatrix, the ‘pseudo-function’ of gradient reversal can be described as

Original: [Equation]
Revised: [Equation]

Original: 4.3 Optimization challenges
Revised: 4.3 Optimization challenges

Original: Since our adversarial dropout network is fully differentiable, it is readily optimized by gradientmethods, such as backpropagation.
Revised: Since our adversarial dropout network is fully differentiable, it is readily optimized by gradientmethods, such as backpropagation.

Original: However, the magnitudes of the word dropout scores s sampledfrom distributions parameterized by an LSTM are unconstrained.
Revised: However, the magnitudes of the word dropout scores s sampledfrom distributions parameterized by an LSTM are unconstrained.

Original: Controlling the magnitude of thescores was found to be important for the adversary to maintain its exploratory capability (discussedabove).
Revised: Controlling the magnitude of thescores was found to be important for the adversary to maintain its exploratory capability (discussedabove).

Original: We therefore add a KL-divergence between the distribution of each score p ( s i | x i ) and astandard Gaussian p 

Revised: We therefore add a KL-divergence between the distribution of each score p ( s i | x i ) and astandard Gaussian p

Original: ( s i )=
Revised: ( s i )=

Original: N ( s i ; 0 , 1) to ‘regularize’ scores in Eq (6), subject to a scalar 
Revised: N ( s i ; 0 , 1) to ‘regularize’ scores in Eq (6), subject to a scalar λ> 0 :

Original: [Equation]
Revised: [Equation]

Original:  Adversarial vs. random word dropout Standard word dropout
Revised: 224 Adversarial vs. random word dropout Standard word dropout

Original: [Iyyer et al., 2015, Bowman et al.,2016] can be seen as a special case of adversarial word dropout.
Revised: [Iyyer et al., 2015, Bowman et al.,2016] can be seen as a special case of adversarial word dropout.

Original: By setting λ sufficiently high, theregularization term in Eq 8 can be made to dominate such that p ψ ( s i | x ) ≈ p 

Revised: By setting λ sufficiently high, theregularization term in Eq 8 can be made to dominate such that p ψ ( s i | x ) ≈ p

Original: ( s i ) and all scores s i areeffectively sampled from a standard Gaussian p 

Revised: ( s i ) and all scores s i areeffectively sampled from a standard Gaussian p

Original: ( s i ) and words are dropped approximately uniformly.
Revised: ( s i ) and words are dropped approximately uniformly.

Original: On the other hand, for small values of λ , A ψ will learn which words VAE ϕ,θ relies upon most toaccurately decode the sequence and target those for drop out.
Revised: On the other hand, for small values of λ , A ψ will learn which words VAE ϕ,θ relies upon most toaccurately decode the sequence and target those for drop out.

Original: As a result, whilst adding anotherhyperparameter to calibrate, λ offers a simple way to moderate the difference between adversarialword dropout and standard uniform word dropout, which we consider in our experiments.
Revised: As a result, whilst adding anotherhyperparameter to calibrate, λ offers a simple way to moderate the difference between adversarialword dropout and standard uniform word dropout, which we consider in our experiments.

Original:  Experiments
Revised:  Experiments

Original: Here we present the results of our experiments that: (i) demonstrate that a sequence VAE trainedwith adversarial word dropout (AWD) outperforms similar models of the same class, achievingeither competitive or improved sentence modeling performance and/or  informativeness of the latentspace; (ii) examine the contributions and behaviour of the adversarial network’s componentsandhyperparameters, and (iii) qualitatively study the trained adversary and VAE.
Revised: Here we present the results of our experiments that: (i) demonstrate that a sequence VAE trainedwith adversarial word dropout (AWD) outperforms other sequence VAEs; it achieves improvedsentence modeling performance and/or improved informativeness of the latent space; (ii) examine thecontributions and behaviour of the adversarial network’s components and hyperparameters, and (iii)qualitatively study the trained adversary and VAE.

Original: Datasets We conducted experiments on 3 different datasets: Yahoo questions and answers [Yanget al., 2017], Yelp reviews [Yang et al., 2017] and downsampled Stanford Natural Language Inference( SNLI ) corpus
Revised: Datasets We conducted experiments on 4 different datasets: Yahoo questions and answers [Yanget al., 2017], Yelp reviews [Yang et al., 2017],

Original:  [Bowman et al., 2015, Li et al., 2019].
Revised: Penn Tree Bank (PTB) Marcus et al. [1993] anddownsampled Stanford Natural Language Inference ( SNLI ) corpus [Bowman et al., 2015, Li et al.

Original: Yahoo and Yelp  datasets were used in manyprevious works [Yang et al., 2017, Kim et al., 2018, He et al., 2019, Fu et al., 2019, Dieng et al. ,2019] hence were used to benchmark our proposed method against comparable related works andalso against standard word dropout
Revised: Yahoo, Yelp and PTB datasets were used in many previous works [Yang et al., 2017, Kimet al., 2018, He et al., 2019, Fu et al., 2019, Dieng et al., 2019] hence were used to benchmarkour proposed method against comparable related works and also against standard word dropout.

Original: Yahoo and Yelp  contain sentences with average lengths of 78and 96  words respectively SNLI sentences are much shorter with an average length of 9 wordsand are more suitable for qualitative studies three datasets contain 100K sentences in the trainingset, 10K in the validation set,  10K set.
Revised: Yahoo, Yelp and PTB contain sentences with average lengths of 78, 96 and 22 words respectively,while SNLI sentences are much shorter with an average length of 9 words and are more suitable forqualitative studies. Yahoo, Yelp and SNLI datasets contain 100K sentences in the training set, 10K inthe validation set, and 10K in the test set, while PTB is much smaller with a total of 42K sentences.

Original: [Table]
Revised: [Table]

Original: Experimental setup We (re-)implement the standard VAE, a VAE with  uniform dropout and a VAEwith our proposed dropout, each with the same hyperparameter configuration of previous comparableworks [Bowman et al., 2015, Kim et al., 2018, He et al., 2019], except for the differences specifiedbelow applied fairly to each implementation.
Revised: Experimental setup We (re-)implement the standard VAE, a VAE with standard uniform worddropout and a VAE with our adversarial dropout method. On each dataset, we performed thesame grid search over both learning rate (from { 0 . 0001 , 0 .

Original: We use SGD with a learning rate of 1  and exponentialdecay of 0.96, which was found to improve training stability compared to decay based on validationperformance [Li and Arora, 2019].
Revised: 001 , 0 . 1 , 1 } ) and dropout rate R (from{ 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } ) for both the word dropout baseline and our method.

Original: We increase the hidden state size of the decoder LSTM from1024 to 2048 to improve overall performance, except for the SNLI dataset, where this offered noimprovement. We apply Polyak averaging [Polyak and Juditsky, 1992] with a coefficient of 0.9995,slightly improving validation performance. We found KL annealing [Bowman et al., 2016] necessary,but apply no other techniques to improve performance proposed in previous works (see Section 3).
Revised: This gives 16 different hyperparameter configurations for each method on each dataset. For training, we also use an exponentiallearning decay of 0.96 as in [Li and Arora, 2019], increased the hidden state sze of the decoder LSTMfrom 1024 to 2048 (except on SNLI), applied Polyak averaging [Polyak and Juditsky, 1992] with acoefficient of 0.9995 and used KL annealing [Bowman et al., 2016].

Original: We apply early stopping based onwhen the ELBO of the validation set reaches a maximum. Standarddeviations are computed over 5 random seeds.
Revised: We apply early stopping based on validation ELBO and repeat each experiment for five differentrandom seeds to report standard deviations.

Original: All experiments are performed on a 12GB Nvidia
Revised: All experiments are performed on a 12GB Nvidia

Original: TitanXP GPU with an average run time of 4 hours for Yelp and Yahoo and 1 hour for SNLI.
Revised: TitanXP GPU with an average run time of 4 hours for Yelp and Yahoo and 1 hour for SNLI.

Original: Metrics Overall model performance is assessed based on two main aspects: (i) sentence modeling– measured with respect to ELBO and perplexity (PPL).
Revised: Metrics Overall model performance is assessed based on two main aspects: (i) sentence modeling– measured with respect to ELBO and perplexity (PPL).

Original: ELBO and PPL quantify the ability ofthe trained decoder to recognise or generate natural language, reflecting the quality of densityestimation; (ii) latent space informativeness – measured with respect to the KL term in Eq (7) andmutual information
Revised: ELBO and PPL quantify the ability ofthe trained decoder to recognise or generate natural language, reflecting the quality of densityestimation; (ii) latent space informativeness – measured with respect to the KL term in Eq (7) andmutual information

Original: I ( X ; Z ) between observed and latent variables.
Revised: I ( X ; Z ) between observed and latent variables.

Original: I ( X ; Z ) is computed using theprocedure of Hoffman and Johnson [2016], as implemented by Dieng et al.
Revised: I ( X ; Z ) is computed using theprocedure of Hoffman and Johnson [2016], as implemented by Dieng et al.

Original: As in comparableworks [Yang et al., 2017, Kim et al., 2018,
Revised: As in comparableworks [Yang et al., 2017, Kim et al., 2018,

Original: He et al., 2019, Dieng et al., 2019, Li et al., 2019],hyperparameters are set based on a balanced assessment of these metrics.
Revised: He et al., 2019, Dieng et al., 2019, Li et al., 2019],hyperparameters are set based on a balanced assessment of these metrics.

Original: Adversary hyperparameters The number K of elements  dropped out from a particular sequenceis calculated from a fixed global dropout rate R and the sequence length T , as round ( R × T ) whereround computes the closest integer. 
Revised: Adversary hyperparameters To apply the global dropout rate R to the number K of elements tobe dropped out we use K = round ( R × T ) where round computes the closest integer. The additionalhyperparameter λ in Eq 8 was set globally for all datasets to λ = 1 .

Original: We compare validation results for R ∈ { 0 . 2 , 0 . 3 , 0 . 4 ,0 . 5} , anduse R = 0 . 3 in our final model; meaning that ≈ 30% of each sentence is dropped. To set λ in Eq 8, we compare λ∈ { 0 .
Revised: This was based on an initialexploratory analysis on the Yahoo dataset, where we grid searched λ in { 0 . 001 , 0 . 01 , 0 . 1 , 1 , 5 , 10 }for various combinations of dropout and learning rates to find that λ

Original: 1 , 1 , 5 , 10 } and find λ = 1 to work consistentlywell across all datasets.
Revised: = 1 consistently achieved bestvalidation performance.

Original: Below we provide a study on the effects of λ .
Revised: Please refer to the detailed analysis of λ in the appendix.

Original: The adversary is trained withthe Adam optimizer [Kingma and Ba, 2014] with the learning rate selected from a hyperparametersearch over { 0 . 0001 , 0 . 001 , 0 . 1 , 1 } : 0 .

Original: [-01 for SNLI and 0 . 
-]
Revised: [-01 for SNLI and 0 . 
-]

Original: for Yahoo and Yelp.

Original: 5.1 Quantitative analysis
Revised: 5.1 Quantitative analysis

Original: Table 1 compares (i) a VAE trained with no dropout; (ii) a VAE trained with uniform word dropout[Bowman et al., 2016]with dropout rate R =0 .
Revised: Table 1 compares (i) the VAE trained with no dropout; (ii) the VAE trained with uniform word dropout[Bowman et al., 2016]

Original:  4 ; (iii) a VAE trained with adversarial word dropout( R =0 . 3 ); and (iv) previously reported results for comparable models (Section 3).
Revised: (dropout rate R =0 . 4 found to be best); (iii) a VAE trained with adversarialword dropout ( R = 0 . 3 found best); and (iv) previously reported results for comparable models(Section 3).

Original: The main observation from these results is that our proposed adversarial approach achieves a betterbalance between learning an  informative latent space (high MI) and sequence modeling(PPL)compared to uniform word dropout. Indeed, training with adversarial dropout achieves eithercomparable or improved performance  on all metrics, suggesting that the model not only bettercaptures thedata distribution p ( x ) , but does so by better reflecting its latent structure.
Revised: Our adversarial dropout method trains models that achieve better sentence modeling(lower ELBO, PPL) with an equally informative latent space (Yahoo, PTB) or more informative latentspace (higher KL, MI) while maintaining sentence modeling performance (Yelp) and improves bothmetrics on SNLI (Appendix Table 5) . Thus, it allows users to more effectively trade-off sentencemodelling and informativenes of the latent space than standard word dropout. The gains are modestin size, but larger on SNLI and PTB, and comparable to those improvements reported in previousword, e.g., [He et al., 2019] (Appendix Table 4, 5).

Original: Our method also compares favourably to previous models, consistently achieving an improved balance betweensentence modeling performance (e.g. PPL) and an informative latent space (e.g. MI).
Revised: Our method also compares favourably to previousmodels, consistently achieving an improved balance between sentence modeling performance (e.g. PPL) and an informative latent space (e.g. MI).

Original: We note also that the vanilla VAE does not obtain a lower perplexity than either VAE with worddropout, as might be anticipated if dropout were an arbitrary ‘regularisation’ method that may improvethe latent space but at a cost to sequence modeling performance.
Revised: We note also that the vanilla VAE does not obtain a lower perplexity than either VAE with word dropout, as might be anticipated if dropout were anarbitrary ‘regularisation’ method that may improve the latent space but at a cost to sequence modelingperformance.

Original: This supports our theoretical analysisindicating that word dropout is not a typical ‘regularizer’ (Section 4.1), rather that it disrupts the flowof available information during training of the autoregressive decoder, forcing it to compensate bystoring information in the latent space.
Revised: This supports our theoretical analysis indicating that word dropout is not a typical‘regularizer’ (Section 4.1), rather that it disrupts the flow of available information during trainingof the autoregressive decoder, forcing it to compensate by storing information in the latent space.

Original: Figure 2 compares adversarial and uniform word dropout, varying the dropout rate on the SNLIdataset.
Revised: Figure 2 compares adversarial and uniform word dropout, varying the dropout rate on the SNLIdataset..

Original: For any given dropout rate, adversarial dropout learns a more informative latent space underboth metrics.
Revised: For any given dropout rate, adversarial dropout learns a more informative latent space underboth metrics.

Original: The figure also ablates the benefit of the Double-LSTM.
Revised: The figure also ablates the benefit of the Double-LSTM.

Original: The role of λ
Revised: The role of λ

Original: The hyperparameter λ can be seen to determine the exploration-exploitation trade-offof the adversary.
Revised: The hyperparameter λ can be seen to determine the exploration-exploitation trade-offof the adversary.

Original: As shown in Figure 3, for small values of λ , the magnitudes and standard deviationsof word dropout scores grow large, causing the distribution of dropped words to become concentrated.
Revised: As shown in Figure 3, for small values of λ , the magnitudes and standard deviationsof word dropout scores grow large, causing the distribution of dropped words to become concentrated.

Revised: For large values of λ , scores become very small with low variance and the adversary convergestowards uniform word dropout.

Original:  Figure 3 aligns with our findings that λ
Revised: The right part of Figure 3 depicts why λ

Original: =1 serves as a useful choice.
Revised: =1 is a good choice.

Original: For large values of λ , scoresbecome very small with low variance and the adversary converges towards uniform word dropout.

Original: Qualitative analysis We obtain further insight into what the adversary learns by analysing theword dropout scores for different sentences.
Revised: Qualitative analysis We obtain further insight into what the adversary learns by analysing theword dropout scores for different sentences.

Original: Table 2 shows that the adversary applies lower dropoutprobabilities to less informative words such as ‘unknown’ tokens that replace all out-of-dictionarywords, and so offer little information about the next word.
Revised: Table 2 shows that the adversary applies lower dropoutprobabilities to less informative words such as ‘unknown’ tokens that replace all out-of-dictionarywords, and so offer little information about the next word.

Original: Depending on the data semantics, theadversary selects different types of words: for SNLI, verbs tend to be picked that explain the activity,e.g. working and participating ; for Yahoo, words are identified that carry question semantics, e.g.what , how , if , and when .
Revised: Depending on the data semantics, theadversary selects different types of words: for SNLI, verbs tend to be picked that explain the activity,e.g. working and participating ; for Yahoo, words are identified that carry question semantics, e.g.what , how , if , and when .

Original: We also show that adversarial training learns a useful generative model withmeaningful latent space by interpolating between sentences (Table 3).
Revised: We also show that adversarial training learns a useful generative model withmeaningful latent space by interpolating between sentences (Table 3).

Revised:  Conclusions

Revised: In this work, we address the phenomenon of posterior collapse that occurs in autoregressive variationalautoencoders (VAEs) used for sequential data.

Revised: To mitigate posterior collapse, we propose a noveladversarial approach that learns to drop words based on their information content , leading toan improvement in both the learned latent structure and sequence modeling performance.

Revised: Wetheoretically derive the effect of word dropout to show that, during training, it removes incremental

Original: <sos> the person giving the demonstration is a man <eos> <sos> a dog participating in a race while wearing the number 6 .
Revised: <sos> the person giving the demonstration is a man <eos> <sos> a dog participating in a race while wearing the number 6 .

Original: <eos> <sos> three women are talking about their matching wigs and silver leggings .
Revised: <eos> <sos> three women are talking about their matching wigs and silver leggings .

Original: <eos> <sos> is there any truth to the rumor that the host of the _unk on _unk , greg _unk is dead ?
Revised: <eos> <sos> is there any truth to the rumor that the host of the _unk on _unk , greg _unk is dead ?

Original: _unk has been down and there is no new information on the web about the future of the site or the _unk , greg _unk .
Revised: _unk has been down and there is no new information on the web about the future of the site or the _unk , greg _unk .

Original: what ’s going on ?
Revised: what ’s going on ?

Original: how could _unk not comment on the past us election ( nov. 7th ) nor the iranian nuclear program ?
Revised: how could _unk not comment on the past us election ( nov. 7th ) nor the iranian nuclear program ?

Original: something must be wrong , where is this guy ?
Revised: something must be wrong , where is this guy ?

Original: internet chat rooms have noted that _unk was injured in a fly fishing accident earlier this month .
Revised: internet chat rooms have noted that _unk was injured in a fly fishing accident earlier this month .

Original: details are unknown .
Revised: details are unknown .

Original: can anyone clear this up ? <
Revised: can anyone clear this up ? <

Original: eos> <sos> is it okay if i mind _unk with my goldfish ?
Revised: eos> <sos> is it okay if i mind _unk with my goldfish ?

Original: yes , but be careful , you may be overwhelmed when you connect to such a clearly superior intellect ... <eos>
Revised: yes , but be careful , you may be overwhelmed when you connect to such a clearly superior intellect ... <eos>

Original: <sos> = start of the sequence token; <eos>
Revised: <sos> = start of the sequence token; <eos>

Original: = end of the sequence token; _unk = unknown token.
Revised: = end of the sequence token; _unk = unknown token.

Original: I ’m not sure what all the hype is about .
Revised: I ’m not sure what all the hype is about .

Original: i ’ve been here a few times and it ’s just ok .
Revised: i ’ve been here a few times and it ’s just ok .

Original: nothing special .
Revised: nothing special .

Original: I would n’t go out of my way to come here .
Revised: I would n’t go out of my way to come here .

Original: I ’ve been here a few times and it ’s always been good .
Revised: I ’ve been here a few times and it ’s always been good .

Original: the food is good , but the service is not so great .
Revised: the food is good , but the service is not so great .

Original: I ’ve been here a few times and it ’s always been good .
Revised: I ’ve been here a few times and it ’s always been good .

Original: i ’ve had the chicken and waffles and the service was good .
Revised: i ’ve had the chicken and waffles and the service was good .

Original: Great place to go for a quick bite to eat .
Revised: Great place to go for a quick bite to eat .

Original: the food is great and the service is great .
Revised: the food is great and the service is great .

Original: i have been here a few times and have never been disappointed .
Revised: i have been here a few times and have never been disappointed .

Original: Great food , great service , and great service .
Revised: Great food , great service , and great service .

Original: i ’ve been here a few times and have never been disappointed Conclusions [Table] In this work, we address the phenomenon of posterior collapse that occurs in autoregressive variationalautoencoders (VAEs) used for sequential data.
Revised: i ’ve been here a few times and have never been disappointed information that each word provides about the next above that available from earlier words or thelatent variable z .

Original: To mitigate posterior collapse, we propose a noveladversarial approach that learns to drop words based on their information content , leading toan improvement in both the learned latent structure and sequence modeling performance.

Original: Morefundamentally, we theoretically derive the effect of word dropout to show that, during training, itremoves incremental information that each word provides about the next above that available fromearlier words or the latent variable z .

Original: The only way for the model to compensate and minimise theincremental information lost is to learn more information in the latent space .
Revised: The only way for the model to compensate and minimise the incremental informationlost is to learn more information in the latent space .

Original: We believe this findingprovides novel and interesting insight as a means of manipulating information within a hierarchicallatent variable model since dropout is used to ‘push’ specific information into the latent variable.
Revised: We believe this finding provides novel andinteresting insight as a means of manipulating information within a hierarchical latent variable modelsince dropout is used to ‘push’ specific information into the latent variable.

Revised: For future work, it maybe interesting to extend our method to transformer-based architectures [Vaswani et al., 2017] thatcan exacerbate posterior collapse when used for decoding and self-supervised learning, where input(words in NLP, image patches in computer vision) are often masked randomly [Devlin et al., 2018,

Revised: He et al., 2022], but a learnt policy might improve performance or convergence.

Original: Sequence VAEs are a promising framework that, beyond purely autoregressive models [Brown et al.
Revised: Sequence VAEs are a promising framework that, beyond purely autoregressive models [Brown et al.

Original: ,2020], hold the prospect of controlled sequence generation.
Revised: ,2020], hold the prospect of controlled sequence generation.

Original: Improvements to such general methodsmay inevitably be used for good or ill, from the generation of targeted fake news on the one hand tothe possibility of personalised human-computer interactions in, say, the medical domain on the other.
Revised: Improvements to such general methodsmay inevitably be used for good or ill, from the generation of targeted fake news on the one hand tothe possibility of personalised human-computer interactions in, say, the medical domain on the other.

Original: In future work, we plan to explore alternative implementation options, particularly of the adversary,and extend adversarial dropout to other domains, such as images or speech.
Revised: In future work, we plan to explore alternative implementation options, particularly of the adversary,and extend adversarial dropout to other domains, such as images or speech.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes] (Section 4.2)
Revised: [Yes] (Section 4.2)

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] (Section 6)
Revised: [Yes] (Section 6)

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] (Section6)(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [Yes] (Section (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] (All Sections)
Revised: [Yes] (All Sections)

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (Section 4.1)
Revised: [Yes] (Section 4.1)

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] (Section 4.1)
Revised: [Yes] (Section 4.1)

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] (Supplementary)
Revised: [Yes] (Supplementary)

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how they were chosen)?
Revised: g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] (Section 5)
Revised: [Yes] (Section 5)

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-
Revised: g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] (Table 1)
Revised: [Yes] (Table 1)

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] (Section 5)
Revised: [Yes] (Section 5)

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (Section 5)
Revised: [Yes] (Section 5)

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Revised: [Table]

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Revised: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Revised: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Revised: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original:  (a) Did you include the full text of instructions given to participants and screenshots, if 
Revised: [Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional 
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Review
Revised: [Table]

Original: Board ( approvals, if include the estimated hourly
Revised: Board (IRB) approvals, if applicable? [N/A]

Original: [Table]
Revised: [Table]

Original:  paid to participants and the total 
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A
Revised: [N/A]

Revised: [Table]