Original: Learning to Drop Out: An Adversarial Approach to Training Sequence VAEs
Revised: Learning to Drop Out: An Adversarial Approach to Training Sequence VAEs

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: In principle, applying variational autoencoders (VAEs) to sequential data offers amethod for controlled sequence generation, manipulation, and structured representation learning.
Revised: In principle, applying variational autoencoders (VAEs) to sequential data offers a method for controlled sequence generation, manipulation, and structured representation learning.

Original: However, training sequence VAEs is challenging: autoregressivedecoders can often explain the data without utilizing the latent space, known asposterior collapse .
Revised: However, training sequence VAEs is challenging: autoregressive decoders can often explain the data without utilizing the latent space, known as posterior collapse .

Original: To mitigate this, state-of-the-art models ‘weaken’ the ‘powerfuldecoder’ by applying uniformly random dropout to the decoder input.
Revised: To mitigate this, state-of-the-art models ‘weaken’ the ‘powerful decoder’ by applying uniformly random dropout to the decoder input.

Original: We show theoretically that this removes pointwise mutual information provided by the decoderinput, which is compensated for by utilizing the latent space.
Revised: We show theoretically that this removes pointwise mutual information provided by the decoder input, which is compensated for by utilizing the latent space.

Original: We then proposean adversarial training strategy to achieve information-based stochastic dropout .
Revised: We then propose an adversarial training strategy to achieve information-based stochastic dropout .

Original: Compared to uniform dropout on standard text benchmark datasets, our targetedapproach increases both sequence modeling performance and the informationcaptured in the latent space.
Revised: Compared to uniform dropout on standard text benchmark datasets, our targeted approach increases both sequence modeling performance and the information captured in the latent space.

Original:  Introduction
Revised:  Introduction

Original: Training autoregressive models via maximum likelihood estimation (MLE) is a common strategyfor representing sequential data.
Revised: Training autoregressive models via maximum likelihood estimation (MLE) is a common strategy for representing sequential data.

Original: Such autoregressive models obtain state-of-the-art in modeling text[Brown et al., 2020, Chowdhery et al., 2022, Zhang et al., 2022], speech [Oord et al., 2018, Conneauet al., 2021], and video sequences [Babaeizadeh et al., 2018].
Revised: Such autoregressive models obtain state-of-the-art in modeling text [Brown et al., 2020, Chowdhery et al., 2022, Zhang et al., 2022], speech [Oord et al., 2018, Conneau et al., 2021] and video sequences [Babaeizadeh et al., 2018].

Original: In this simple and intuitive approach,the joint distribution of a sequence is factorized into a product of conditional distributions, with eachsequence element conditioned on its history.
Revised: In this simple and intuitive approach, the joint distribution of a sequence is factorized into a product of conditional distributions, with each sequence element conditioned on its history.

Original: However, in their basic form, autoregressive modelsdo not necessarily learn latent variables that reflect semantic content of the data, as often desired.
Revised: However, in their basic form, autoregressive models do not necessarily learn latent variables that encode informative content, as often desired.

Original: As such, supplementing autoregressive models with latent variables is a promising way to enhancecontrol in sequence generation, and enable structured representation learning.
Revised: As such, supplementing autoregressive models with latent variables is a promising way to enhance control in sequence generation, and enable structured representation learning.

Original: A recent approach to latent variable modeling of sequences is to integrate autoregressive componentsinto the variational autoencoder (VAE) framework [Kingma and Welling, 2014, Rezende et al., 2014].
Revised: A recent approach to latent variable modeling of sequences is to integrate autoregressive components into the variational autoencoder (VAE) framework [Kingma and Welling, 2014, Rezende et al., 2014].

Original: Sequence VAEs offer a theoretically principled solution, but have not been widely adopted.
Revised: Sequence VAEs offer a theoretically principled solution, but have not been widely adopted.

Original: Arguably,this is largely due to the phenomenon of posterior collapse [Bowman et al., 2016, Chen et al., 2017,
Revised: Arguably, this is largely due to the phenomenon of posterior collapse [Bowman et al., 2016, Chen et al., 2017,

Original: Van Den Oord et al., 2017, Li et al., 2020].
Revised: Van Den Oord et al., 2017, Chen et al., 2020].

Original: Posterior collapse describes when a VAE’s posteriorprobability over latent variables ‘collapses’ to the latent prior, rendering the latent space completelyuninformative, or independent , of the data.
Revised: Posterior collapse describes when a VAE’s posterior probability over latent variables ‘collapses’ to the latent prior, rendering the latent space completely uninformative, or independent , of the data.

Original: Recent years have seen an abundance of research intoposterior collapse [Bowman et al., 2015, Chen et al., 2017, Lucas et al., 2019, Dai et al., 2020, Fanget al., 2021, Pang et al., 2021], identifying the ‘power’ of autoregressive decoding as a main cause.
Revised: Recent years have seen an abundance of research into posterior collapse [Bowman et al., 2015, Chen et al., 2017, Lucas et al., 2019, Dai et al., 2020, Fang et al., 2021, Pang et al., 2021], identifying the ‘power’ of autoregressive decoding as a main cause.

Original: Inparticular, autoregressive decoders are shown to obtain satisfactory sequence modeling performancewithout utilizing the latent space [Bowman et al., 2016, Chen et al., 2017].
Revised: In particular, autoregressive decoders are shown to obtain satisfactory sequence modeling performance without utilizing the latent space [Bowman et al., 2016, Chen et al., 2017].

Original: This suggests that forsequence VAEs to progress, improved techniques are needed to understand and alleviate posteriorcollapse, and thereby learn informative latent structure.
Revised: This suggests that for sequence VAEs to progress, improved techniques are needed to understand and alleviate posterior collapse, and thereby learn informative latent structure.

Original: In this work, we propose a theoretically principled way to mitigate posterior collapse when training autoregressive sequence VAEs.
Revised: In this work, we propose a theoretically principled way to mitigate posterior collapse when training autoregressive sequence VAEs.

Original: The intuition that autoregressive VAE decoders are too ‘powerful’ has led to the notion of ‘weakening’them, e.g. by regularization.
Revised: The intuition that autoregressive VAE decoders are too ‘powerful’ has led to the notion of ‘weakening’ them, e.g. by regularization.

Original: However, weakening an autoregressive model arbitrarily might of courseharm its performance, leading to a trade off between informativeness of the latent space and thequality of sequence modeling (or density estimation performance).
Revised: However, weakening an autoregressive model arbitrarily might of course harm its performance, leading to a trade off between informativeness of the latent space and the quality of sequence modeling (or density estimation performance).

Original: State-of-the-art VAEs applyvariants of dropout [Srivastava et al., 2014] to the sequence input into the decoder: either to individualdimensions of sequence elements [Kim et al., 2018], or to mask entire sequence elements [Iyyeret al., 2015, Bowman et al., 2016].
Revised: State-of-the-art VAEs apply variants of dropout [Srivastava et al., 2014] to the sequence input into the decoder: either to individual dimensions of sequence elements [Kim et al., 2018], or to mask entire sequence elements [Iyyer et al., 2015, Bowman et al., 2016].

Original: In either case, dropout is generally applied uniformly at random,meaning that each sequence element has an equal probability of being dropped.
Revised: In either case, dropout is generally applied uniformly at random, meaning that each sequence element has an equal probability of being dropped.

Original: From the intuitionthat different words carry different information about the next word, this work investigates whethera non-uniform dropout policy can achieve a better trade-off between capturing latent information andsequence modeling performance.
Revised: From the intuition that different words carry different information about the next word, this work investigates whether a non-uniform dropout policy can achieve a better trade-off between capturing latent information and sequence modeling performance.

Original: Specifically, by stochastically dropping each sequence elementaccording to its importance to the autoregressive decoder, our method takes a targeted approach todropout relative to uniform sampling, maintaining sequence modeling performance while achievingthe benefits of dropout on the latent space.
Revised: Specifically, by stochastically dropping each sequence element according to its importance to the autoregressive decoder, our method takes a targeted approach to dropout relative to uniform sampling, maintaining sequence modeling performance while achieving the benefits of dropout on the latent space.

Original: To apply a non-uniform, data-dependent dropout scheme,our approach introduces an ‘adversary’ that learns which elements to drop .
Revised: To apply a non-uniform, data-dependent dropout scheme, our approach introduces an ‘adversary’ that learns which elements to drop .

Original: The proposed frameworkis represented in Figure 1, theoretically developed in Section 4.1 and implemented in Section 4.2.
Revised: The proposed framework is represented in Figure 1, theoretically developed in Section 4.1 and implemented in Section 4.2.

Original: The contributions of this paper can be summarized as follows: (i)
Revised: The contributions of this paper can be summarized as follows: (i)

Original: We propose an adversarial approachto learning a stochastic dropout policy that mitigates posterior collapse in sequence VAEs.
Revised: We propose an adversarial approach to learning a stochastic dropout policy that mitigates posterior collapse in sequence VAEs.

Original: (ii) Weshow theoretically that dropping out sequence elements deducts mutual information not alreadylearned by the model , between one sequence element and the next, and that our approach maximisesthis quantity, leading to a constrained minimax ELBO objective that explains an adversarial approach.(iii)
Revised: (ii) We show theoretically that dropping out sequence elements deducts mutual information not already learned by the model , between one sequence element and the next, and that our approach maximises this quantity, leading to a constrained minimax ELBO objective that explains an adversarial approach. (iii)

Original: We evaluate the proposed scheme on standard text benchmarks, showing that our approachrenders an informative latent space, without trading-off but rather improving sentence modeling.
Revised: We evaluate the proposed scheme on standard text benchmarks, showing that our approach renders an informative latent space, without trading-off but rather improving sentence modeling.

Original: (iv)
Revised: (iv)

Original: We examine properties of the adversarial network and find that the adversary typically selects wordsthat carry sentence semantics that fit our theoretical analysis.
Revised: We examine properties of the adversarial network and find that the adversary typically selects words that carry sentence semantics that fit our theoretical analysis.

Original:  Background
Revised:  Background

Original: Here, we formally describe the use of autoregressive decoding within VAEs for sequential data andthe problem of posterior collapse that often arises.
Revised: Here, we formally describe the use of autoregressive decoding within VAEs for sequential data and the problem of posterior collapse that often arises (refer to Appendix D for some clarification around the term posterior collapse ).

Original: For notation: X denotes a random variable, (boldface) x denotes a sequence that is a realization of
Revised: For notation: X denotes a random variable, (boldface) x denotes a sequence that is a realization of

Original: X , (subscript) x i denotes the i -th sequence element of x and x <i collectively denotes all sequence elements prior x i .
Revised: X , (subscript) x i denotes the i -th sequence element of x and x <i collectively denotes all sequence elements prior x i .

Original: z denotes a global latent vector and (superscript) x
Revised: z denotes a global latent vector and (superscript) x

Original: ( i ) denotes the i -th sample froma dataset.
Revised: ( i ) denotes the i -th sample from a dataset.

Original: The task is to approximate the true probability density function p ∗ ( x ) of a random sequence X ∼p ∗ ( x ) using a model p θ ( x ) with parameters θ .
Revised: The task is to approximate the true probability density function p ∗ ( x ) of a random sequence X ∼ p ∗ ( x ) using a model p θ ( x ) with parameters θ .

Original: Given a set of observations { x ( i ) } Ni =1 , θ can beestimated by maximum likelihood estimation (MLE):
Revised: Given a set of observations { x ( i ) } Ni =1 , θ can be estimated by maximum likelihood estimation (MLE):

Original: [Equation]
Revised: [Equation]

Original: If the data has latent generative factors z , modelled as p θ ( x )=
Revised: If the data has latent generative factors z , modelled as p θ ( x )=

Original: (cid:82) z p θ ( x | z ) p θ ( z ) , Eq 1 is intractableand one can instead maximize the evidence lower bound (ELBO), which for any observation x is:
Revised: (cid:82) z p θ ( x | z ) p θ ( z ) , Eq 1 is intractable and one can instead maximize the evidence lower bound (ELBO), which for any observation x is:

Original: [Equation]
Revised: [Equation]

Original: where, q ϕ ( z | x ) approximates the posterior, parameterized by ϕ , and p θ ( z ) is a prior distributionover latent variables.
Revised: where, q ϕ ( z | x ) approximates the posterior, parameterized by ϕ , and p θ ( z ) is a prior distribution over latent variables.

Original: In this work, we consider the framework of variational autoencoders (VAEs)[Kingma
Revised: In this work, we consider the framework of variational autoencoders (VAEs)

Original:  and Welling, 2014, Rezende et al., 2014], where the encoder q ϕ ( z | x ) and decoder p θ ( x | z )are parameterised by neural networks and p θ ( z ) is a standard Gaussian, denoted p 0 ( z ) .
Revised: [Kingma and Welling, 2014, Rezende et al., 2014], where the encoder q ϕ ( z | x ) and decoder p θ ( x | z ) are parameterised by neural networks and p θ ( z ) is a standard Gaussian, denoted p 0 ( z ) .

Original: Further, wefocus specifically on VAEs where the decoder p θ ( x | z ) is autoregressive [Bowman et al., 2016, Liet al., 2020,
Revised: Further, we focus specifically on VAEs where the decoder p θ ( x | z ) is autoregressive [Bowman et al., 2016, Chen et al., 2020,

Original: He et al., 2019], meaning log p θ ( x | z )
Revised: He et al., 2019], meaning log p θ ( x | z )

Original: = (cid:80) i log p θ ( x i | x <i , z ) , and the ELBO becomes:
Revised: = (cid:80) i log p θ ( x i | x <i , z ) , and the ELBO becomes:

Original: [Equation]
Revised: [Equation]

Original: Posterior collapse: Autoregressive models improve density estimation [Oord et al., 2016] byexplicitly modeling statistical dependencies between sequence elements.
Revised: Posterior collapse: Autoregressive models improve density estimation [Oord et al., 2016] by explicitly modeling statistical dependencies between sequence elements.

Original: However, when used as a
Revised: However, when used as a

Original: VAE decoder p θ ( x | z ) , autoregressive models are often found to ignore z , i.e. p ( x | z ) ≈ p ( x ) , and theposterior is said to ‘collapse’ to the prior, i.e. q θ ( z | x ) ≈ p 0 ( z ) , rendering z de facto independent of x[Bowman et al., 2016, He et al., 2019].
Revised: VAE decoder p θ ( x | z ) , autoregressive models are often found to ignore z , i.e. p ( x | z ) ≈ p ( x ) , and the posterior is said to ‘collapse’ to the prior, i.e. q θ ( z | x ) ≈ p 0 ( z ) , rendering z de facto independent of x [Bowman et al., 2016, He et al., 2019].

Original: In terms of Eq 3, this implies p θ ( x i | x <i , z )
Revised: In terms of Eq 3, this implies p θ ( x i | x <i , z )

Original: ≈
Revised: ≈

Original: p θ ( x i | x <i ) .
Revised: p θ ( x i | x <i ) .

Original: From this, posterior collapse can be considered due to the information about x i provided by x <ibeing such that each x i is (approximately) conditionally independent of z given x <i , in other wordsz is essentially redundant .
Revised: From this, posterior collapse can be considered due to the information about x i provided by x <i being such that each x i is (approximately) conditionally independent of z given x <i , in other words z is essentially redundant .

Original: Based on this, we interpret the notion that autoregressive models are toopowerful for use as a VAE decoder to mean that they allow too much information flow .
Revised: Based on this, we interpret the notion that autoregressive models are too powerful for use as a VAE decoder to mean that they allow too much information flow .

Original:  Related Work
Revised:  Related Work

Original: Posterior collapse The phenomenon of posterior collapse has been observed in the context of text[Bowman et al., 2016, Yang et al., 2017], images [Chen et al., 2017, Razavi et al., 2018], videos[Babaeizadeh et al., 2018], speech [Chorowski et al., 2019] and graphs [Kipf et al., 2018].
Revised: Posterior collapse The phenomenon of posterior collapse has been observed in the context of text [Bowman et al., 2016, Yang et al., 2017], images [Chen et al., 2017, Razavi et al., 2018, Miladinovi´c et al., 2021], videos [Babaeizadeh et al., 2018, Miladinovi´c et al., 2019a], speech [Chorowski et al., 2019] and graphs [Kipf et al., 2018].

Original: In broadterms, previous solutions can be divided into two complementary categories: (i) latent-variableoriented methods that focus mainly on relaxing the KL penalty; and (ii) decoder-oriented methodsthat regularise or ‘weaken’ the autoregressive decoder.
Revised: In broad terms, previous solutions can be divided into two complementary categories: (i) latent-variable-oriented methods that focus mainly on relaxing the KL penalty; and (ii) decoder-oriented methods that regularise or ‘weaken’ the autoregressive decoder.

Original: Typically, both types are required in orderto learn an informative latent space 
Revised: Typically, both types are required in order to learn an informative latent space [Bowman et al., 2016,

Original: [Bowman et al., 2016, Chen et al., 2017, Kim et al., 2018].
Revised: Chen et al., 2017, Kim et al., 2018].

Original: Latent-variable-oriented solutions Bowman et al.
Revised: Latent-variable-oriented solutions Bowman et al.

Original: [2016] suggest KL annealing , a technique thatintroduces the KL term gradually into training according to a predefined schedule.
Revised: [2016] suggest KL annealing , a technique that introduces the KL term gradually into training according to a predefined schedule.

Original: Free bits [Kingmaet al., 2016] prevent penalizing of the KL term if its magnitude is below a predefined threshold.
Revised: Free bits [Kingma et al., 2016] prevent penalizing of the KL term if its magnitude is below a predefined threshold.

Original: Van
Revised: Van

Original: Den Oord et al.
Revised: Den Oord et al.

Original: [2017] use a discrete VAE, avoiding posterior collapse by design.
Revised: [2017] use a discrete VAE, avoiding posterior collapse by design.

Original: Fu et al.
Revised: Fu et al.

Original: [2019]apply a cyclical form of KL annealing.
Revised: [2019] apply a cyclical form of KL annealing.

Original: Razavi et al.
Revised: Razavi et al.

Original: [2018] constrain the variational family of theposterior distribution (encoder), preventing it from closely approximating the prior and so holding the
Revised: [2018] constrain the variational family of the posterior distribution (encoder), preventing it from closely approximating the prior and so holding the

Original: KL term away from zero.
Revised: KL term away from zero.

Original: Lagging inference networks [He et al., 2019] ‘aggressively’ optimize theencoder before each decoder update.
Revised: Lagging inference networks [He et al., 2019] ‘aggressively’ optimize the encoder before each decoder update.

Original: A similar procedure is followed by Semi-Amortized VAEs [Kimet al., 2018].
Revised: A similar procedure is followed by Semi-Amortized VAEs [Kim et al., 2018].

Original: Generative skip models [Dieng et al., 2019] introduce skip connections to create a moreexplicit link between the latent variables and the likelihood function.
Revised: Generative skip models [Dieng et al., 2019] introduce skip connections to create a more explicit link between the latent variables and the likelihood function.

Original: Sinha and Dieng [2021] proposea consistency regularizer by minimizing the KL divergence between the posterior approximations ofan observation and a random transformation of it.
Revised: Sinha and Dieng [2021] propose a consistency regularizer by minimizing the KL divergence between the posterior approximations of an observation and a random transformation of it.

Original: Decoder-oriented solutions Bowman et al.
Revised: Decoder-oriented solutions Bowman et al.

Original: [2016] apply word dropout [Iyyer et al., 2015] to uniformly drop words during autoregressive decoding.
Revised: [2016] apply word dropout [Iyyer et al., 2015] to uniformly drop words during autoregressive decoding.

Original: Other methods [Kim et al., 2018, He et al., 2019] apply parameter dropout
Revised: Other methods [Kim et al., 2018, He et al., 2019] apply parameter dropout

Original: [Srivastava et al., 2014]to word embeddings.
Revised: [Srivastava et al., 2014] to word embeddings.

Original: Chen et al.
Revised: Chen et al.

Original: [2017], Semeniuta et al.
Revised: Semeniuta et al.

Original: [2017], Yang et al.
Revised: [2017], Yang et al.

Original: [2017] constrain thereceptive field of the decoder, limiting the window of autoregression, however, this is not readilyapplicable to RNN architectures given their unbounded receptive field.
Revised: [2017] constrain the receptive field of the decoder, limiting the window of autoregression, however, this is not readily applicable to RNN architectures given their unbounded receptive field.

Original: Our proposed adversarialmethod falls into this category as an extension of word dropout, which is then subsumed as a specialcase of adversarial word dropout (Section 4.2).
Revised: Our proposed adversarial method falls into this category as an extension of word dropout, which is then subsumed as a special case of adversarial word dropout (Section 4.2).

Original: Non-uniform dropout Previous work has recognized the benefit of adapting the dropout rate acrossdifferent architectural components, though in entirely different contexts [Kingma et al., 2015, Galet al., 2017, Achille and Soatto, 2018].
Revised: Non-uniform dropout Previous work has recognized the benefit of adapting the dropout rate across different architectural components, though in entirely different contexts [Kingma et al., 2015, Gal et al., 2017, Achille and Soatto, 2018].

Original: These works introduce different approaches to regularizingthe magnitude of weights or activations (groups of weights).
Revised: These works introduce different approaches to regularizing the magnitude of weights or activations (groups of weights).

Original: Elements that are deemed irrelevantduring training are then dropped.
Revised: Elements that are deemed irrelevant during training are then dropped.

Original: For instance, variational dropout [Kingma et al., 2015] has beenused to prune weights in deep neural networks [Molchanov et al., 2017].
Revised: For instance, variational dropout [Kingma et al., 2015] has been used to prune weights in deep neural networks [Molchanov et al., 2017].

Original: A key difference to ourapproach is that instead of a regularization term, we employ an adversary that selects elements fordropout during training based on their information content not yet learned by the model.
Revised: A key difference to our approach is that instead of a regularization term, we employ an adversary that selects elements for dropout during training based on their information content not yet learned by the model.

Original: Exposure bias The method by which the autoregressive decoder is trained, known as teacherforcing , bears a connection to exposure bias , which refers to the gap between training and inference[Ranzato et al., 2015].
Revised: Exposure bias The method by which the autoregressive decoder is trained, known as teacher forcing , bears a connection to exposure bias , which refers to the gap between training and inference [Ranzato et al., 2015].

Original: Namely, in language generation, a trained model generates sequences at testtime without access to the ground truth history that was accessible during training.
Revised: Namely, in language generation, a trained model generates sequences at test time without access to the ground truth history that was accessible during training.

Original: Since the modelwas not trained to continue its predictions, this can lead to error accumulation [Ranzato et al., 2015].
Revised: Since the model was not trained to continue its predictions, this can lead to error accumulation [Ranzato et al., 2015].

Original: The most common approach to tackle exposure bias is to switch between conditioning on groundtruth and model predictions, with the latter being preferred towards the end of training [Daumé et al. ,2009, Bengio et al., 2015, Ranzato et al., 2015].
Revised: The most common approach to tackle exposure bias is to switch between conditioning on ground truth and model predictions, with the latter being preferred towards the end of training [Daumé et al., 2009, Bengio et al., 2015, Ranzato et al., 2015].

Original: In preliminary studies, we implemented scheduledsampling [Bengio et al., 2015] but did not find it to outperform even uniform dropout.
Revised: In preliminary studies, we implemented scheduled sampling [Bengio et al., 2015] but did not find it to outperform even uniform dropout.

Original:  Adversarial Word Dropout (AWD) 
Revised:  Adversarial Word Dropout (AWD) 4.1 Theoretical Basis

Original: 4.1 Theoretical Basis

Original: As a precursor to our adversarial approach, we first derive the effect of stochastic dropout on the
Revised: As a precursor to our adversarial approach, we first derive the effect of stochastic dropout on the

Original: ELBO under the autoregressive assumption (Equation 3).
Revised: ELBO under the autoregressive assumption (Equation 3).

Original: Word dropout: Our interpretation of posterior collapse at the end of Section 2 suggests that to‘weaken the decoder’ one should restrict the available information , such that z is no longer redundantand must capture some of the restricted information.
Revised: Word dropout: Our interpretation of posterior collapse at the end of Section 2 suggests that to ‘weaken the decoder’ one should restrict the available information , such that z is no longer redundant and must capture some of the restricted information.

Original: Word dropout [Bowman et al., 2016] can be seento do this by stochastically masking the previous word x i − 1 with probability d i , which substitutesp θ ( x i | x <i , z ) with p θ ( x i | x <i − 1 , z ) in Equation 3, i.e.:
Revised: Word dropout [Bowman et al., 2016] can be seen to do this by stochastically masking the previous word x i − 1 with probability d i , which substitutes p θ ( x i | x <i , z ) with p θ ( x i | x <i − 1 , z ) in Equation 3, i.e.:

Original: [Equation]
Revised: [Equation]

Original: Equation 5 shows the effect (in expectation) of applying word dropout when maximising the ELBOwith an autoregressive decoder in Eq 3 (of which ‘KL’ denotes the last term).
Revised: Equation 5 shows the effect (in expectation) of applying word dropout when maximising the ELBO with an autoregressive decoder in Eq 3 (of which ‘KL’ denotes the last term).

Original: In effect, a weighted sumis introduced with weights given by dropout probabilities d i and components that define conditionalpoint-wise mutual information (PMI), where classic point-wise mutual information between twovariables is defined as PMI ( x 1 , x 2 ) =
Revised: In effect, a weighted sum is introduced with weights given by dropout probabilities d i and components that define conditional point-wise mutual information (PMI), where classic point-wise mutual information between two variables is defined as PMI ( x 1 , x 2 ) =

Original: p ( x 1 | x 2 ) p ( x 1 ) .
Revised: p ( x 1 | x 2 ) p ( x 1 ) .

Original: Each PMI term in Eq 5 captures the informationthat one sequence element x i − 1 has regarding the next x i , over and above any information fromearlier elements x <i − 1 and the latent state z .
Revised: Each PMI term in Eq 5 captures the information that one sequence element x i − 1 has regarding the next x i , over and above any information from earlier elements x <i − 1 and the latent state z .

Original: Since the dropout-adapted ELBO (Eq 5) is maximisedw.r.t.
Revised: Since the dropout-adapted ELBO (Eq 5) is maximised w.r.t.

Original: θ and ϕ , the weighted PMI term is minimised, which is only achievable by increasing theinformation learned by the model : extracted from x <i − 1 or captured in z .
Revised: θ and ϕ , the weighted PMI term is minimised, which is only achievable by increasing the information learned by the model : extracted from x < i − 1 or captured in z .

Original: Interestingly this showsthat word dropout quite literally follows the earlier intuition and ‘weakens’ the decoder by restrictinginformation .
Revised: Interestingly this shows that word dropout quite literally follows the earlier intuition and ‘weakens’ the decoder by restricting information .

Original: Thus, word dropout leads to a looser variational objective, i.e. ELBO WD θ,ϕ ( x ) ≤
Revised: Thus, word dropout leads to a looser variational objective, i.e. ELBO WD θ,ϕ ( x ) ≤

Original: ELBO AR θ,ϕ ( x ) .
Revised: ELBO AR θ,ϕ ( x ) .

Original: Rainforth et al.
Revised: Rainforth et al.

Original: [2018] has argued that tighter bounds are not necessarily better.
Revised: [2018] has argued that tighter bounds are not necessarily better.

Original: Hence,our analysis provides a justification for using word dropout.
Revised: Hence, our analysis provides a justification for using word dropout.

Original: The minimax objective: Under uniform dropout, all d i are equal over a sequence, and usingdropout to ‘push information into z ’ is applied evenly.
Revised: The minimax objective: Under uniform dropout, all d i are equal over a sequence, and using dropout to ‘push information into z ’ is applied evenly.

Original: However, some sequence elements may holdmore information than others, e.g. in language, given syntactic rules and a summary of the previoustext, some words may be highly predictable without specifically knowing which word preceded them,whereas others may depend heavily on their predecessor despite the other information.
Revised: However, some sequence elements may hold more information than others, e.g. in language, given syntactic rules and a summary of the previous text, some words may be highly predictable without specifically knowing which word preceded them, whereas others may depend heavily on their predecessor despite the other information.

Original: This suggeststhat information content may be non-uniform and so applying dropout non-uniformly may be moreappropriate.
Revised: This suggests that information content may be non-uniform and so applying dropout non-uniformly may be more appropriate.

Original: We, therefore, propose targeted dropout of elements according to their ‘incremental’information content defined by the PMI terms, i.e. that not yet learned by the model.
Revised: We, therefore, propose targeted dropout of elements according to their ‘incremental’ information content defined by the PMI terms, i.e. that not yet learned by the model.

Original: Further, sincesuch incremental information content is not explicitly computed and will vary over training as themodel learns, we train an adaptive dropout schedule to continually maximise the information dropped,and so minimize Eq 5, with respect to d i .
Revised: Further, since such incremental information content is not explicitly computed and will vary over training as the model learns, we train an adaptive dropout schedule to continually maximise the information dropped, and so minimize Eq 5, with respect to d i .

Original: Since Eq 5 is maximised with respect to all other parameters,this leads to a minimax objective.
Revised: Since Eq 5 is maximised with respect to all other parameters, this leads to a minimax objective.

Original: We introduce the two ‘players’ in the proposed minimax setting: VAE ϕ,θ , trained to maximizethe ELBO, and the adversary A ψ trained to do the opposite by determining each probability d i ofdropping out the previous sequence element x i − 1 , preventing it from helping to predict its successorx i .
Revised: We introduce the two ‘players’ in the proposed minimax setting: VAE ϕ,θ , trained to maximize the ELBO, and the adversary A ψ trained to do the opposite by determining each probability d i of dropping out the previous sequence element x i − 1 , preventing it from helping to predict its successor x i .

Original: The full objective for an observed sequence is given by
Revised: The full objective for an observed sequence is given by

Original: [Equation]
Revised: [Equation]

Original:  where R ψ ( x ) is a regularization term explained below (Section 4.2) and 
Revised: (6) where R ψ ( x ) is a regularization term explained below (Section 4.2) and L ψ,ϕ,θ = E

Original: [Equation]
Revised: [Equation]

Original: L ψ,ϕ,θ ( x ) is a modified version of the ELBO in Equation 5 with the additional parameters ψ of theadversary A ψ .
Revised: L ψ,ϕ,θ ( x ) is a modified version of the ELBO in Equation 5 with the additional parameters ψ of the adversary A ψ .

Original: The adversary manifests via the masking operator mask K,ψ ( · ; x ) , conditioned on theentire input sequence x .
Revised: The adversary manifests via the masking operator mask K,ψ ( · ; x ) , conditioned on the entire input sequence x .

Original: Since unconstrained minimisation of Eq 7 w.r.t ψ would cause all elementsto be dropped out, a constraint K is introduced such that exactly K elements of x are dropped duringautoregressive decoding.
Revised: Since unconstrained minimisation of Eq 7 w.r.t ψ would cause all elements to be dropped out, a constraint K is introduced such that exactly K elements of x are dropped during autoregressive decoding.

Original: In practice, K is sequence length-dependent, treated as a hyperparameterthat controls the level of adversarial regularization.
Revised: In practice, K is sequence length-dependent, treated as a hyperparameter that controls the level of adversarial regularization.

Original: The adversary plays no role at test time.
Revised: The adversary plays no role at test time.

Original: 4.2 Implementation
Revised: 4.2 Implementation

Original: A high-level overview of our framework is shown in Figure 1.
Revised: A high-level overview of our framework is shown in Figure 1.

Original: An encoder maps the input sequenceinto a hidden representation to produce the parameters of q ϕ ( z | x ) , from which z is sampled usingthe ‘reparametrization trick’ [Kingma and Welling, 2014, Rezende et al., 2014].
Revised: An encoder maps the input sequence into a hidden representation to produce the parameters of q ϕ ( z | x ) , from which z is sampled using the ‘reparametrization trick’ [Kingma and Welling, 2014, Rezende et al., 2014].

Original: We implement thedecoder using a unidirectional LSTM [Hochreiter and Schmidhuber, 1997] for consistency withthe RNN-VAE network of Bowman et al.
Revised: We implement the decoder using a unidirectional LSTM [Hochreiter and Schmidhuber] for consistency with the RNN-VAE network of Bowman et al.

Original: [2016], because LSTMs are the most popular backbonefor sequence VAEs and to facilitate comparison with prior work.
Revised: [2016], because LSTMs are the most popular backbone for sequence VAEs and to facilitate comparison with prior work.

Original: Inspired by the recent work of[Dieng et al., 2019], we choose a Double-LSTM recurrent unit that elucidates the skip-connections topromote higher latent information content flow (see Appendix A for details).
Revised: Inspired by the recent work of [Dieng et al., 2019], we choose a Double-LSTM recurrent unit that elucidates the skip-connections to promote higher latent information content flow (see Appendix A for details).

Original: As described in Section 4.1, the main innovation of our proposed method is the adversarial trainingprocedure: instead of dropping out decoder inputs uniformly at random, we introduce a trainableadversary to drop words most important for the VAE to reconstruct the original sequence.
Revised: As described in Section 4.1, the main innovation of our proposed method is the adversarial training procedure: instead of dropping out decoder inputs uniformly at random, we introduce a trainable adversary to drop words most important for the VAE to reconstruct the original sequence.

Original: Below wedescribe implementation details of the adversary, specifying the relevant components in Figure 1.
Revised: Below we describe implementation details of the adversary, specifying the relevant components in Figure 1.

Original: Amore comprehensive figure specific to the implemented architecture is provided in Appendix B. Notethat the theoretical framework in Section 4.1 permits many alternative approaches for generatingper word dropout probabilities
Revised: A more comprehensive figure specific to the implemented architecture is provided in Appendix B. Note that the theoretical framework in Section 4.1 permits many alternative approaches for generating per word dropout probabilities

Original: d i , such as attention mechanisms mentioned above.
Revised: d i , such as attention mechanisms mentioned above.

Original: We leave furtherrefinement of the adversarial architecture to future work.
Revised: We leave further refinement of the adversarial architecture to future work.

Original: Producing dropout scores To determine which elements to drop out of a sequence, the adversary
Revised: Producing dropout scores To determine which elements to drop out of a sequence, the adversary

Original: A ψ samples a score s i ∈ R for each sequence element x i .
Revised: A ψ samples a score s i ∈ R for each sequence element x i .

Original: Sequence elements with the K smallestscores are dropped.
Revised: Sequence elements with the K smallest [Table] al

Original:  The scores are sampled from distributions p ψ ( s i | x ) conditioned on the inputsequence x , modelled as a series of Gaussians with mean ( µ ) and variance ( σ ) parameterised by theoutputs of a unidirectional LSTM.
Revised: [Table] scores are dropped. The scores are sampled from distributions p ψ ( s i | x ) conditioned on the input sequence x , modelled as a series of Gaussians with mean ( µ ) and variance ( σ ) parameterised by the outputs of a unidirectional LSTM.

Original: Scores are generated using the reparameterization trick, reducinggradient variance [Kingma and Welling, 2014].
Revised: Scores are generated using the reparameterization trick, reducing gradient variance [Kingma and Welling, 2014].

Original: Thus, p ψ ( s i | x ) is described ass i | x ∼ N ( µ i , σ i ; x , ψ ) where, [ µ i , σ i ] =
Revised: Thus, p ψ ( s i | x ) is described as s i | x ∼ N ( µ i , σ i ; x , ψ ) where, [ µ i , σ i ] =

Original: Linear ψ ( LSTM i,ψ ( x ))
Revised: Linear ψ ( LSTM i,ψ ( x ))

Original: The stochastic generation of scores gives the adversary an ‘exploratory’ capability during training,e.g. preventing it becoming ‘stuck’ on a set of sequence elements with high information contentthat the model cannot learn, meaning that their PMI terms (in Eq 5) remain large and the adversaryrepeatedly selects them to minimise the ELBO.
Revised: The stochastic generation of scores gives the adversary an ‘exploratory’ capability during training, e.g. preventing it becoming ‘stuck’ on a set of sequence elements with high information content that the model cannot learn, meaning that their PMI terms (in Eq 5) remain large and the adversary repeatedly selects them to minimise the ELBO.

Original: Top-K word selection Based on the sampled scores s = { s i } Ti =1 , the subset of K words with thesmallest scores are masked during decoding.
Revised: Top-K word selection Based on the sampled scores s = { s i } Ti =1 , the subset of K words with the smallest scores are masked during decoding.

Original: To estimate gradients of the objective with respectto the parameters ψ , we use a stochastic softmax trick from Paulus et al.
Revised: To estimate gradients of the objective with respect to the parameters ψ , we use a stochastic softmax trick from Paulus et al.

Original: [2020]: during training,the stochastic subset selected in the forward pass is relaxed to admit a (biased) reparameterizationgradient in the backward pass.
Revised: [2020a]: during training, the stochastic subset selected in the forward pass is relaxed to admit a (biased) reparameterization gradient in the backward pass.

Original: In our experiments, we use a straight-through variant  of the trick forour method to resemble word dropout, i.e. to produce discrete values ∈{ 0 , 1 } .
Revised: In our experiments, we use a straight-through variant [Bengio et al., 2013, Paulus et al., 2020b] of the trick for our method to resemble word dropout, i.e. to produce discrete values ∈{ 0 , 1 } .

Original: The relaxation is onlyused in the backward pass to compute the gradient estimator.
Revised: The relaxation is only used in the backward pass to compute the gradient estimator.

Original:  The final component of the adversarial network is the gradient reversal layer[Ganin et al., 2016].
Revised: Gradient reversal The final component of the adversarial network is the gradient reversal layer [Ganin et al., 2016].

Original: In the forward pass, the layer performs no transformation to the input; in thebackward pass, the gradients are negated.
Revised: In the forward pass, the layer performs no transformation to the input; in the backward pass, the gradients are negated.

Original: Gradient reversal offers a computationally simple methodto ensure that the parameters ψ of A ψ are updated such that ELBO is minimized.
Revised: Gradient reversal offers a computationally simple method to ensure that the parameters ψ of A ψ are updated such that ELBO is minimized.

Original: If I is an identitymatrix, the ‘pseudo-function’ of gradient reversal can be described as
Revised: If I is an identity matrix, the ‘pseudo-function’ of gradient reversal can be described as

Original: [Equation]
Revised: [Equation]

Original: 4.3 Optimization challenges
Revised: Optimization challenges

Original: Since our adversarial dropout network is fully differentiable, it is readily optimized by gradientmethods, such as backpropagation.
Revised: Since our adversarial dropout network is fully differentiable, it is readily optimized by gradient methods, such as backpropagation.

Original: However, the magnitudes of the word dropout scores s sampledfrom distributions parameterized by an LSTM are unconstrained.
Revised: However, the magnitudes of the word dropout scores s sampled from distributions parameterized by an LSTM are unconstrained.

Original: Controlling the magnitude of thescores was found to be important for the adversary to maintain its exploratory capability (discussedabove).
Revised: Controlling the magnitude of the scores was found to be important for the adversary to maintain its exploratory capability (discussed above).

Original: We therefore add a KL-divergence between the distribution of each score p ( s i | x i ) and astandard Gaussian p 

Revised: We therefore add a KL-divergence between the distribution of each score p ( s i | x i ) and a standard Gaussian p

Original: ( s i )=
Revised: ( s i )=

Original: N ( s i ; 0 , 1) to ‘regularize’ scores in Eq (6), subject to a scalar λ> 0 :
Revised: N ( s i ; 0 , 1) to ‘regularize’ scores in Eq (6), subject to a scalar λ> 0 :

Original: [Equation]
Revised: [Equation]

Original: 224 Adversarial vs. random word dropout Standard word dropout
Revised: Adversarial vs. random word dropout Standard word dropout

Original: [Iyyer et al., 2015, Bowman et al.,2016] can be seen as a special case of adversarial word dropout.
Revised: [Iyyer et al., 2015, Bowman et al., 2016] can be seen as a special case of adversarial word dropout.

Original: By setting λ sufficiently high, theregularization term in Eq 8 can be made to dominate such that p ψ ( s i | x ) ≈ p 

Revised: By setting λ sufficiently high, the regularization term in Eq 8 can be made to dominate such that p ψ ( s i | x ) ≈ p

Original: ( s i ) and all scores s i areeffectively sampled from a standard Gaussian p 

Revised: ( s i ) and all scores s i are effectively sampled from a standard Gaussian p

Original: ( s i ) and words are dropped approximately uniformly.
Revised: ( s i ) and words are dropped approximately uniformly.

Original: On the other hand, for small values of λ , A ψ will learn which words VAE ϕ,θ relies upon most toaccurately decode the sequence and target those for drop out.
Revised: On the other hand, for small values of λ , A ψ will learn which words VAE ϕ,θ relies upon most to accurately decode the sequence and target those for drop out.

Original: As a result, whilst adding anotherhyperparameter to calibrate, λ offers a simple way to moderate the difference between adversarialword dropout and standard uniform word dropout, which we consider in our experiments.
Revised: As a result, whilst adding another hyperparameter to calibrate, λ offers a simple way to moderate the difference between adversarial word dropout and standard uniform word dropout, which we consider in our experiments.

Original:  Experiments
Revised:  Experiments

Original: Here we present the results of our experiments that: (i) demonstrate that a sequence VAE trainedwith adversarial word dropout (AWD) outperforms other sequence VAEs; it achieves improvedsentence modeling performance and/or improved informativeness of the latent space; (ii) examine thecontributions and behaviour of the adversarial network’s components and hyperparameters, and (iii)qualitatively study the trained adversary and VAE.
Revised: Here we present the results of our experiments that: (i) demonstrate that a sequence VAE trained with adversarial word dropout (AWD) outperforms other sequence VAEs; it achieves improved sentence modeling performance and/or improved informativeness of the latent space; (ii) examine the contributions and behaviour of the adversarial network’s components and hyperparameters, and (iii) qualitatively study the trained adversary and VAE.

Original: Datasets We conducted experiments on 4 different datasets: Yahoo questions and answers [Yanget al., 2017], Yelp reviews [Yang et al., 2017],
Revised: Datasets We conducted experiments on 4 different datasets: Yahoo questions and answers [Yang et al., 2017], Yelp reviews [Yang et al., 2017],

Original: Penn Tree Bank (PTB) Marcus et al.
Revised: Penn Tree Bank (PTB) Marcus et al.

Original: [1993] anddownsampled Stanford Natural Language Inference ( SNLI ) corpus [Bowman et al., 2015, Li et al.
Revised: [1993] and downsampled Stanford Natural Language Inference ( SNLI ) corpus [Bowman et al., 2015, Li et al., 2019].

Original: Yahoo, Yelp and PTB datasets were used in many previous works [Yang et al., 2017, Kimet al., 2018, He et al., 2019, Fu et al., 2019, Dieng et al., 2019] hence were used to benchmarkour proposed method against comparable related works and also against standard word dropout.
Revised: Yahoo, Yelp and PTB datasets were used in many previous works [Yang et al., 2017, Kim et al., 2018, He et al., 2019, Fu et al., 2019, Dieng et al., 2019] hence were used to benchmark our proposed method against comparable related works and also against standard word dropout.

Original: Yahoo, Yelp and PTB contain sentences with average lengths of 78, 96 and 22 words respectively,while SNLI sentences are much shorter with an average length of 9 words and are more suitable forqualitative studies.
Revised: Yahoo, Yelp and PTB contain sentences with average lengths of 78, 96 and 22 words respectively, while SNLI sentences are much shorter with an average length of 9 words and are more suitable for qualitative studies.

Original: Yahoo, Yelp and SNLI datasets contain 100K sentences in the training set, 10K inthe validation set, and 10K in the test set, while PTB is much smaller with a total of 42K sentences.[Table]
Revised: Yahoo, Yelp and SNLI datasets contain 100K sentences in the training set, 10K in the validation set, and 10K in the test set, while PTB is much smaller with a total of 42K sentences.

Original: Experimental setup We (re-)implement the standard VAE, a VAE with standard uniform worddropout and a VAE with our adversarial dropout method.
Revised: Experimental setup We (re-)implement the standard VAE, a VAE with standard uniform word dropout and a VAE with our adversarial dropout method.

Original: On each dataset, we performed thesame grid search over both learning rate (from { 0 . 0001 , 0 .
Revised: On each dataset, we performed the same grid search over both learning rate (from { 0 . 0001 , 0 .

Original: 001 , 0 . 1 , 1 } ) and dropout rate R (from{ 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } )
Revised: 001 , 0 . 1 , 1 } ) and dropout rate R (from { 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } )

Original: for both the word dropout baseline and our method.
Revised: for both the word dropout baseline and our method.

Original: This gives 16 different hyperparameter configurations for each method on each dataset.
Revised: This gives 16 different hyperparameter configurations for each method on each dataset.

Original: For training, we also use an exponentiallearning decay of 0.96 as in [Li and Arora, 2019], increased the hidden state sze of the decoder LSTMfrom 1024 to 2048 (except on SNLI), applied Polyak averaging [Polyak and Juditsky, 1992] with acoefficient of 0.9995 and used KL annealing [Bowman et al., 2016].
Revised: For training, we also use an exponential learning decay of 0.96 as in [Li and Arora, 2019], increased the hidden state sze of the decoder LSTM from 1024 to 2048 (except on SNLI), applied Polyak averaging [Polyak and Juditsky, 1992] with a coefficient of 0.9995 and used KL annealing [Bowman et al., 2016].

Original: We apply early stopping based on validation ELBO and repeat each experiment for five differentrandom seeds to report standard deviations.
Revised: We apply early stopping based on validation ELBO and repeat each experiment for five different random seeds to report standard deviations.

Original: All experiments are performed on a 12GB Nvidia
Revised: All experiments are performed on a 12GB Nvidia

Original: TitanXP GPU with an average run time of 4 hours for Yelp and Yahoo and 1 hour for SNLI.
Revised: TitanXP GPU with an average run time of 4 hours for Yelp and Yahoo and 1 hour for SNLI.

Original: Metrics Overall model performance is assessed based on two main aspects: (i) sentence modeling– measured with respect to ELBO and perplexity (PPL).
Revised: Metrics Overall model performance is assessed based on two main aspects: (i) sentence modeling – measured with respect to ELBO and perplexity (PPL).

Original: ELBO and PPL quantify the ability ofthe trained decoder to recognise or generate natural language, reflecting the quality of densityestimation; (ii) latent space informativeness – measured with respect to the KL term in Eq (7) andmutual information
Revised: ELBO and PPL quantify the ability of the trained decoder to recognise or generate natural language, reflecting the quality of density estimation; (ii) latent space informativeness – measured with respect to the KL term in Eq (7) and mutual information

Original: I ( X ; Z ) between observed and latent variables.
Revised: I ( X ; Z ) between observed and latent variables.

Original: I ( X ; Z ) is computed using theprocedure of Hoffman and Johnson [2016], as implemented by Dieng et al.
Revised: I ( X ; Z ) is computed using the procedure of Hoffman and Johnson [2016], as implemented by Dieng et al.

Original: As in comparableworks [Yang et al., 2017, Kim et al., 2018, He et al., 2019, Dieng et al., 2019, Li et al., 2019],hyperparameters are set based on a balanced assessment of these metrics.
Revised: As in comparable works [Yang et al., 2017, Kim et al., 2018, He et al., 2019, Dieng et al., 2019, Li et al., 2019], hyperparameters are set based on a balanced assessment of these metrics.

Original: Adversary hyperparameters To apply the global dropout rate R to the number K of elements tobe dropped out we use K = round ( R × T ) where round computes the closest integer.
Revised: Adversary hyperparameters To apply the global dropout rate R to the number K of elements to be dropped out we use K = round ( R × T ) where round computes the closest integer.

Original: The additionalhyperparameter λ in Eq 8 was set globally for all datasets to λ = 1 .
Revised: The additional hyperparameter λ in Eq 8 was set globally for all datasets to λ = 1 .

Original: This was based on an initialexploratory analysis on the Yahoo dataset, where we grid searched λ in { 0 . 001 , 0 . 01 , 0 . 1 , 1 , 5 , 10 }for various combinations of dropout and learning rates to find that λ
Revised: This was based on an initial exploratory analysis on the Yahoo dataset, where we grid searched λ in { 0 . 001 , 0 . 01 , 0 . 1 , 1 , 5 , 10 } for various combinations of dropout and learning rates to find that λ

Original: = 1 consistently achieved bestvalidation performance.
Revised: = 1 consistently achieved best validation performance.

Original: Please refer to the detailed analysis of λ in the appendix.
Revised: Please refer to the detailed analysis of λ in the appendix.

Original: 5.1 Quantitative analysis
Revised: 5.1 Quantitative analysis

Original: Table 1 compares (i) the VAE trained with no dropout; (ii) the VAE trained with uniform word dropout[Bowman et al., 2016]
Revised: Table 1 compares (i) the VAE trained with no dropout; (ii) the VAE trained with uniform word dropout [Bowman et al., 2016]

Original: (dropout rate R =0 . 4 found to be best); (iii) a VAE trained with adversarialword dropout ( R = 0 .
Revised: (dropout rate R =0 . 4 found to be best); (iii) a VAE trained with adversarial word dropout ( R = 0 .

Original: 3 found best); and (iv) previously reported results for comparable models(Section 3).
Revised: 3 found best); and (iv) previously reported results for comparable models (Section 3).

Original: Our adversarial dropout method trains models that achieve better sentence modeling(lower ELBO, PPL) with an equally informative latent space (Yahoo, PTB) or more informative latentspace (higher KL, MI) while maintaining sentence modeling performance (Yelp) and improves bothmetrics on SNLI (Appendix Table 5) .
Revised: Our adversarial dropout method trains models that achieve better sentence modeling (lower ELBO, PPL) with an equally informative latent space (Yahoo, PTB) or more informative latent space (higher KL, MI) while maintaining sentence modeling performance (Yelp) and improves both metrics on SNLI (Appendix Table 5) .

Original: Thus, it allows users to more effectively trade-off sentencemodelling and informativenes of the latent space than standard word dropout.
Revised: Thus, it allows users to more effectively trade-off sentence modelling and informativenes of the latent space than standard word dropout.

Original: The gains are modestin size, but larger on SNLI and PTB, and comparable to those improvements reported in previousword, e.g., [He et al., 2019] (Appendix Table 4, 5).
Revised: The gains are modest in size, but larger on SNLI and PTB, and comparable to those improvements reported in previous word, e.g., [He et al., 2019] (Appendix Table 4, 5).

Original: Our method also compares favourably to previousmodels, consistently achieving an improved balance between sentence modeling performance (e.g.
Revised: Our method also compares favourably to previous models, consistently achieving an improved balance between sentence modeling performance (e.g.

Original: PPL) and an informative latent space (e.g. MI).
Revised: PPL) and an informative latent space (e.g. MI).

Original: We note also that the vanilla VAE does not obtain a lower perplexity than either VAE with word dropout, as might be anticipated if dropout were anarbitrary ‘regularisation’ method that may improve the latent space but at a cost to sequence modelingperformance.
Revised: We note also that the vanilla VAE does not obtain a lower perplexity than either VAE with word dropout, as might be anticipated if dropout were an arbitrary ‘regularisation’ method that may improve the latent space but at a cost to sequence modeling performance.

Original: This supports our theoretical analysis indicating that word dropout is not a typical‘regularizer’ (Section 4.1), rather that it disrupts the flow of available information during trainingof the autoregressive decoder, forcing it to compensate by storing information in the latent space.
Revised: This supports our theoretical analysis indicating that word dropout is not a typical ‘regularizer’ (Section 4.1), rather that it disrupts the flow of available information during training of the autoregressive decoder, forcing it to compensate by storing information in the latent space.

Original: Figure 2 compares adversarial and uniform word dropout, varying the dropout rate on the SNLIdataset..
Revised: Figure 2 compares adversarial and uniform word dropout, varying the dropout rate on the SNLI dataset.

Original: For any given dropout rate, adversarial dropout learns a more informative latent space underboth metrics. The figure also ablates the benefit of the Double-LSTM.
Revised: For any given dropout rate, adversarial dropout learns a more informative latent space (with higher KL and MI) metrics with comparable to lower negative ELBO.

Original: The role of λ
Revised: The role of λ

Original: The hyperparameter λ can be seen to determine the exploration-exploitation trade-offof the adversary.
Revised: The hyperparameter λ can be seen to determine the exploration-exploitation trade-off of the adversary.

Original: As shown in Figure 3, for small values of λ , the magnitudes and standard deviationsof word dropout scores grow large, causing the distribution of dropped words to become concentrated.
Revised: As shown in Figure 3, for small values of λ , the magnitudes and standard deviations of word dropout scores grow large, causing the distribution of dropped words to become concentrated.

Original: For large values of λ , scores become very small with low variance and the adversary convergestowards uniform word dropout.
Revised: For large values of λ , scores become very small with low variance and the adversary converges towards uniform word dropout.

Original: The right part of Figure 3 depicts why λ
Revised: The right part of Figure 3 depicts why λ

Original: =1 is a good choice.
Revised: =1 is a good choice.

Original: Qualitative analysis We obtain further insight into what the adversary learns by analysing theword dropout scores for different sentences.

Original: Table 2 shows that the adversary applies lower dropoutprobabilities to less informative words such as ‘unknown’ tokens that replace all out-of-dictionarywords, and so offer little information about the next word.

Original: Depending on the data semantics, theadversary selects different types of words: for SNLI, verbs tend to be picked that explain the activity,e.g. working and participating ; for Yahoo, words are identified that carry question semantics, e.g.what , how , if , and when .

Original: We also show that adversarial training learns a useful generative model withmeaningful latent space by interpolating between sentences (Table 3).

Original: Conclusions

Original: In this work, we address the phenomenon of posterior collapse that occurs in autoregressive variationalautoencoders (VAEs) used for sequential data.

Original: To mitigate posterior collapse, we propose a noveladversarial approach that learns to drop words based on their information content , leading toan improvement in both the learned latent structure and sequence modeling performance.

Original: Wetheoretically derive the effect of word dropout to show that, during training, it removes incremental

Original: <sos> the person giving the demonstration is a man <eos> <sos> a dog participating in a race while wearing the number 6 .
Revised: <sos> the person giving the demonstration is a man <eos> <sos> a dog participating in a race while wearing the number 6 .

Original: <eos> <sos> three women are talking about their matching wigs and silver leggings .
Revised: <eos> <sos> three women are talking about their matching wigs and silver leggings .

Original: <eos> <sos> is there any truth to the rumor that the host of the _unk on _unk , greg _unk is dead ?
Revised: <eos> <sos> is there any truth to the rumor that the host of the _unk on _unk , greg _unk is dead ?

Original: _unk has been down and there is no new information on the web about the future of the site or the _unk , greg _unk .
Revised: _unk has been down and there is no new information on the web about the future of the site or the _unk , greg _unk .

Original: what ’s going on ?
Revised: what ’s going on ?

Original: how could _unk not comment on the past us election ( nov. 7th ) nor the iranian nuclear program ?
Revised: how could _unk not comment on the past us election ( nov. 7th ) nor the iranian nuclear program ?

Original: something must be wrong , where is this guy ?
Revised: something must be wrong , where is this guy ?

Original: internet chat rooms have noted that _unk was injured in a fly fishing accident earlier this month .
Revised: internet chat rooms have noted that _unk was injured in a fly fishing accident earlier this month .

Original: details are unknown .
Revised: details are unknown .

Original: can anyone clear this up ? <
Revised: can anyone clear this up ? <

Original: eos> <sos> is it okay if i mind _unk with my goldfish ?
Revised: eos> <sos> is it okay if i mind _unk with my goldfish ?

Original: yes , but be careful , you may be overwhelmed when you connect to such a clearly superior intellect ... <eos>
Revised: yes , but be careful , you may be overwhelmed when you connect to such a clearly superior intellect ... <eos>

Original: <sos> = start of the sequence token; <eos>
Revised: <sos> = start of the sequence token; <eos>

Original: = end of the sequence token; _unk = unknown token.
Revised: = end of the sequence token; _unk = unknown token.

Revised: Qualitative analysis We obtain further insight into what the adversary learns by analyzing the word dropout scores for different sentences.

Original: I ’m not sure what all the hype is about  . 
Revised: Table 2 shows that the adversary applies lower dropout probabilities to less informative words such as ‘unknown’ tokens that replace all out-of-dictionary words, and so offer little information about the next word. Depending on the data semantics, the adversary selects different types of words: for SNLI, verbs tend to be picked that explain the activity, e.g. working and participating ; for Yahoo, words are identified that carry question semantics, e.g. what , how , if , and when . Figure 4 (a) shows a quantitative analysis of dropout saliency map across different part-of-speech (POS) tags. Verb (verb), interjections (intj), and nouns (noun) have higher saliency scores (higher chances of being dropped) compared to punctuation (punc), determiners (det), and the start (sos) and end tokens (eos) which are relatively easier to predict given previous words.

Revised: We also show that adversarial training learns a useful generative model with meaningful latent space by interpolating between sentences (Table 3).

Revised: Computing BERT F 1 score Zhang et al.

Revised: [2020] between the interpolated sentences with the source and target sentence shows the increasing trend toward the target sentence for each interpolation (as each interpolated sentence is getting away from the source and closer to the target) and decreasing for the source sentence (Figure 4 (b)).

Revised: [Figure]

Revised:  Conclusions

Revised: In this work, we address the phenomenon of posterior collapse that occurs in autoregressive variational autoencoders (VAEs) used for sequential data.

Revised: To mitigate posterior collapse, we propose a novel adversarial approach that learns to drop words based on their information content , leading to an improvement in both the learned latent structure and sequence modeling performance.

Revised: We theoretically derive the effect of word dropout to show that, during training, it removes incremental information that each word provides about the next above that available from earlier words or the latent variable z .

Revised: The only way for the model to compensate and minimise the incremental information lost is to learn more information in the latent space .

Revised: We believe this finding provides novel and interesting insight as a means of manipulating information within a hierarchical latent variable model since dropout is used to ‘push’ specific information into the latent variable.

Revised: For future work, it may be interesting to extend our method to transformer-based architectures [Vaswani et al., 2017] that can exacerbate posterior collapse when used for decoding and self-supervised learning, where input (words in NLP, image patches in computer vision) are often masked randomly [Devlin et al., 2019,

Revised: He et al., 2022], but a learnt policy might improve performance or convergence.

Revised: Sequence VAEs are a promising framework that, beyond purely autoregressive models [Brown et al., 2020], hold the prospect of controlled sequence generation.

Revised: Improvements to such general methods may inevitably be used for good or ill, from the generation of targeted fake news on the one hand to the possibility of personalised human-computer interactions in, say, the medical domain on the other (e.g. for modeling sleep [Miladinovi ´ c et al., 2019b, Nowak et al., 2021]).

Revised: In future work, we plan to explore alternative implementation options, particularly of the adversary, and extend adversarial dropout to other domains, such as images, speech, or dynamical systems [Bauer et al., 2017].

Revised: I ’m not sure what all the hype is about .

Original: i ’ve been here a few times and it ’s just ok .
Revised: i ’ve been here a few times and it ’s just ok .

Original: nothing special .
Revised: nothing special .

Original: I would n’t go out of my way to come here .
Revised: I would n’t go out of my way to come here .

Original: I ’ve been here a few times and it’s always been good .
Revised: I ’ve been here a few times and it

Original:  the food is good , but the service is not so great.
Revised: ’s always been good . the food is good , but the service is not so great

Original: I ’ve been here a few times and it ’s always been good .
Revised: I ’ve been here a few times and it ’s always been good .

Original: i ’ve had the chicken and waffles and the service was good.
Revised: i ’ve had the chicken and waffles and the service was good

Original: Great place to go for a quick bite to eat .
Revised: Great place to go for a quick bite to eat .

Original: the food is great and the service is great .
Revised: the food is great and the service is great .

Original: i have been here a few times and have never been disappointed .
Revised: i have been here a few times and have never been disappointed .

Original: Great food , great service , and great service .
Revised: Great food , great service , and great service .

Original: i ’ve been here a few times and have never been disappointedinformation that each word provides about the next above that available from earlier words or thelatent variable z .
Revised: i ’ve been here a few times and have never been disappointed

Original:  The only way for the model to compensate and minimise the incremental informationlost is to learn more information in the latent space . We believe this finding provides novel andinteresting insight as a means of manipulating information within a hierarchical latent variable modelsince dropout is used to ‘push’ specific information into the latent variable. For future work, it maybe interesting to extend our method to transformer-based architectures [Vaswani et al., 2017] thatcan exacerbate posterior collapse when used for decoding and self-supervised learning, where input(words in NLP, image patches in computer vision) are often masked randomly [Devlin et al., 2018,
Revised:  Acknowledgments

Original: He et al., 2022], but a learnt policy might improve performance or convergence.
Revised: We thank Taylor Berg-Kirkpatrick for his thoughtful insights and valuable feedback.

Original: Sequence VAEs are a promising framework that, beyond purely autoregressive models [Brown et al. ,2020], hold the prospect of controlled sequence generation.
Revised: Carl is gratefully supported by an ETH AI Centre Postdoctoral Fellowship, a responsible AI grant by the Haslerstiftung;

Original: Improvements to such general methodsmay inevitably be used for good or ill, from the generation of targeted fake news on the one hand tothe possibility of personalised human-computer interactions in, say, the medical domain on the other. In future work, we plan to explore alternative implementation options, particularly of the adversary,and extend adversarial dropout to other domains, such as images or speech.
Revised: Swiss National Science Foundation (project # 201009), and an ETH Grant (ETH-19 21-1).

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (Section 4.2) (b) Did you describe the limitations of your work?
Revised: [Yes] (Section 4.2) (b) Did you describe the limitations of your work?

Original: [Yes] (Section 6) (c) Did you discuss any potential negative societal impacts of your work?[Yes] (Section
Revised: [Yes] (Section 6) (c) Did you discuss any potential negative societal impacts of your work?

Original:  (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] (Section 6) (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] (All Sections) 
Revised: [Yes] (All Sections) 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (Section 4.1) (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (Section 4.1) (b) Did you include complete proofs of all theoretical results?

Original: [Yes] (Section 4.1) 
Revised: [Yes] (Section 4.1) 3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (Supplementary) (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
Revised: [Yes] (Supplementary) (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (Section 5) (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: [Yes] (Section 5) (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (Table 1) (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (Table 1) (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] (Section 5) 
Revised: [Yes] (Section 5) 4.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (Section 5) (b) Did you mention the license of the assets?
Revised: [Yes] (Section 5) (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL? [N/A] 
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL? [N/A] (d) Did you discuss whether

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re[Table] using/curating?[N/A]
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable[Table] information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] [Table]
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects...[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: applicable? [N/A][Table] (b) Did you describe any potential participant risks, with links to Institutional Review 
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [Table]

Original: Board (IRB) approvals, if applicable?

Original: [N/A][Table] (c) Did you include the estimated hourly wage paid to participants and the total amount 
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [Table]
Revised: [N/A]

Original: spent on participant compensation?
Revised: Inspired by the recent work [Dieng et al., 2019] that elucidates how skip-connections promote higher latent information content, we introduce a simple-to-implement modification of a standard LSTM [Hochreiter and Schmidhuber]. DoubleLSTM aims to promote the utilization of the latent variable z , whilst increasing the expressive power of the autoregressive decoder.

Original: [N/A]
Revised: Double-LSTM consists of two LSTM units [Hochreiter and Schmidhuber] as depicted in Figure 5. The first LSTM unit is updated based on the latent variable z and the previous hidden state h .

Original: [Table]
Revised: The second LSTM unit is updated based on z , h and the input word embedding w , which is subject to teacher forcing and dropout. The benefit of Double-LSTM is that it provides a two-branched skip connection to link z with the output word. The state update performed by the first LSTM is guaranteed to extract information from latent states and not from ground truth input. In practice, Double-LSTM leads to performance improvements with little cost in terms of memory and computation time.

Revised: B Implementation Architecture

Revised: LSTM

Revised: LSTM

Revised: LSTM

Revised: VAEs are awesome

Revised: [Figure]

Revised: <sos>

Revised: Encoder

Revised: [Figure]

Revised: VAEs are awesome <eos>

Revised: Z

Revised: Adversary

Revised: Autoregressive Decoder z z z z embedding embedding embedding embedding embedding embedding embedding

Revised: VAEs are awesome embedding embedding embedding

Revised: LSTM

Revised: LSTM

Revised: LSTM

Revised: [Figure]

Revised: - based autoregressive sequence decoder (depicted in green) .

Revised: During MLE training, the ground truth sequence history is teacher-forced to the autoregressive decoder which reduces the utilization of z .

Revised: To regularize autoregressive decoding, the adversary A ψ (depicted in orange) obstracts VAE by learning to drop out words (sequence elements) that VAE requires.

Revised: Concretely, A ψ selectively masks different words at decoder’s input, effectively performing input-conditioned word dropout.

Revised: In the first stage, A ψ stochastically produces unnormalized word dropout scores for each word in a sequence, using another LSTM that architecturally mirrors the one in the encoder.

Revised: In the second stage, A ψ selects a subset of K ’dropouts’ using the differentiable top- K word selection module.

Revised: Finally, to ensure that A ψ optimizes an objective that is an inverted version of the VAE objective, gradient reversal [Ganin et al., 2016] is applied to negate the gradients in the backward pass.

Revised: C Additional Quantitative Analysis

Revised: [Table]

Revised: + Word Dropout

Revised: [Table]

Revised: + AWD [

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: + Word Dropout

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: + AWD [0.3] 32.66 (0.2)

Revised: [Table]

Revised: [Table]

Revised: are provided in the brackets.

Revised: Squared bracket contains the dropout rate DR .

Revised: ELBO – evidence lower bound; KL - KL divergence; MI – mutual information.

Revised: D Definition of "posterior collapse"

Revised: We note that that the phenomena we address is often referred to as posterior collapse , which could be misinterpreted as meaning that the posterior in fact collapses to a single point (as in an MLE or

Revised: MAP estimate), particularly since for VAEs, latent posteriors are typically more concentrated than the prior.

Revised: As such, it might be less ambiguous to refer to the phenomenon as posterior dispersion or posterior ignorance or similar to better capture the fact that the posterior becomes diffuse and carries no information with respect to the data.

Revised: E Additional Qualitative Analysis

Revised: This section provides additional qualitative experiments performed using the sequence VAE trained with the proposed adversarial training.

Revised: I ’m not sure what all the hype is about .

Revised: i ’ve been here a few times and it ’s just ok .

Revised: nothing special .

Revised: I would n’t go out of my way to come here .

Revised: I ’ve been here a few times and it ’s always been good .

Revised: the food is good , but the service is not so great .

Revised: I ’ve been here a few times and it ’s always been good .

Revised: i ’ve had the chicken and waffles and the service was good .

Revised: Great place to go for a quick bite to eat .

Revised: the food is great and the service is great .

Revised: i have been here a few times and have never been disappointed .

Revised: Great food , great service , and great service .

Revised: i ’ve been here a few times and have never been disappointed

Revised: <sos> do you think that you should be with someone you love ?

Revised: if you do n’t know what you are , then you should be happy .

Revised: <eos>

Revised: i am not sure if he is going to get a job .

Revised: i am not ready to get married but i am not sure how to get him to pay for it .

Revised: <eos> <sos> please help me with my homework ?

Revised: i am a _unk student

Revised: and i need to know what the average salary of a school is in the us .

Revised: i am looking for a website that has a list of the average salaries of students in the united states.

Revised: <eos> <sos>

Revised: what are some good websites to get free stuff on the net ?

Revised: <eos> <sos> how do i become a better person ?

Revised: i ’m shy , but i do n’t know how to approach a guy .

Revised: what should i do ?

Revised: you should be able to be friends with someone who is not interested in you .

Revised: if you are shy.

Revised: <sos> if you have a quadratic equation , what is the value of x ? x = -1 <eos> <sos> how to get a _unk visa ?

Revised: i have a degree in psychology and i want to know what is the process of getting a job in the us .

Revised: if you are a _unk , you can apply for a job .

Revised: <eos> <sos> the u.s. has a nuclear power to stop the war in iraq ?

Revised: what is the reason for the war ?

Revised: the war is over .

Revised: <eos> <sos> question about jesus ?

Revised: what is the name of the church that jesus is in the bible ?

Revised: what is the name of the church ?

Revised: <eos> <sos> do you think it is bad for you to have a cold sore ?

Revised: i have a cold sore and i have a bad breath .

Revised: why do i have to pee ?

Revised: it ’s

Revised: because it ’s not a bad thing .

Revised: <eos> <sos> is there any _unk in the world ?

Revised: yes , but there is no such thing as a soul mate .

Revised: <eos>

Revised: <sos> since the nfl has gone to the eight division format ?

Revised: have three teams from teh same division made the playoffs in the same year not yet but could happen this year with the cowboys , giants , and redskins in the nfc east , or steelers , bengals , and ravens in the afc north .

Revised: <sos> the dog jumps into the air to catch a toy in its mouth .

Revised: <eos> <sos> a young woman in a white shirt and black pants is playing with a young boy in a blue shirt .

Revised: <eos> <sos> the person is flying a plane .

Revised: <eos> <sos> the dogs have their owners in the air in front of a crowd of onlookers .

Revised: <eos> <sos> the people are participating in an operation .

Revised: <eos> <sos> a woman with black hair is standing in a puddle .

Revised: <eos> <sos> a young woman is riding a bike in front of a group of people in a red dress .

Revised: <sos> people are holding up their signs in their hands .

Revised: <eos>