Original: Auto-Encoding Sequential Monte Carlo
Revised: Auto-Encoding Sequential Monte Carlo

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: We build on auto-encoding sequential Monte Carlo ( aesmc ): 1 a method for model and proposal learning based on maximizing the lower bound to the log marginal likelihood in a broad family of structured probabilistic models.
Revised: We build on auto-encoding sequential Monte Carlo ( aesmc ): 1 a method for model and proposal learning based on maximizing the lower bound to the log marginal likelihood in a broad family of structured probabilistic models.

Original: Our approach relies on the eﬃciency of sequential Monte Carlo ( smc ) for performing inference in structured probabilistic models and the ﬂexibility of deep neural networks to model complex conditional probability distributions.
Revised: Our approach relies on the eﬃciency of sequential Monte Carlo ( smc ) for performing inference in structured probabilistic models and the ﬂexibility of deep neural networks to model complex conditional probability distributions.

Original: We develop additional theoretical insights and introduce a new training procedure which improves both model and proposal learning.
Revised: We develop additional theoretical insights and introduce a new training procedure which improves both model and proposal learning.

Original: We demonstrate that our approach provides a fast, easy-to-implement and scalable means for simultaneous model learning and proposal adaptation in deep generative models.
Revised: We demonstrate that our approach provides a fast, easy-to-implement and scalable means for simultaneous model learning and proposal adaptation in deep generative models.

Original:  Introduction
Revised:  Introduction

Original: We build upon aesmc (Anon, 2017), a method for model learning that itself builds on variational auto-encoders ( vae s) (Kingma & Welling, 2014; Rezende et al., 2014) and importance weighted auto-encoders ( iwae s) (Burda et al., 2016).
Revised: We build upon aesmc (Anon, 2017), a method for model learning that itself builds on variational auto-encoders ( vae s) (Kingma & Welling, 2014; Rezende et al., 2014) and importance weighted auto-encoders ( iwae s) (Burda et al., 2016).

Original: Aesmc is similarly based on maximizing a lower bound to the log marginal likelihood, but uses smc (Doucet &
Revised: Aesmc is similarly based on maximizing a lower bound to the log marginal likelihood, but uses smc (Doucet &

Original: Johansen, 2009) as the underlying marginal likelihood estimator instead of importance sampling ( is ).
Revised: Johansen, 2009) as the underlying marginal likelihood estimator instead of importance sampling ( is ).

Original: For a very wide array of models, particularly those with sequential structure, smc forms a substantially more powerful inference method than is , typically returning lower variance estimates for the marginal likelihood.
Revised: For a very wide array of models, particularly those with sequential structure, smc forms a substantially more powerful inference method than is , typically returning lower variance estimates for the marginal likelihood.

Original: Consequently, by using smc for its marginal likelihood estimation, aesmc often leads to improvements in model learning compared with vae s and iwae s.
Revised: Consequently, by using smc for its marginal likelihood estimation, aesmc often leads to improvements in model learning compared with vae s and iwae s.

Original: We provide experiments on structured time-series data that show aesmc based learning was able to learn useful representations of the latent space for both reconstruction and prediction more eﬀectively than the iwae counterpart.
Revised: We provide experiments on structured time-series data that show aesmc based learning was able to learn useful representations of the latent space for both reconstruction and prediction more eﬀectively than the iwae counterpart.

Original: Aesmc was introduced in an earlier preprint (Anon, 2017) concurrently with the closely related methods of Maddison et al.
Revised: Aesmc was introduced in an earlier preprint (Anon, 2017) concurrently with the closely related methods of Maddison et al.

Original: ; Naesseth et al.
Revised: ; Naesseth et al.

Original: In this work we take these ideas further by providing new theoretical insights for the resulting evidence lower bounds ( elbo s), extending these to explore the relative eﬃciency of diﬀerent approaches to proposal learning, and using our results to develop a new and improved training procedure.
Revised: In this work we take these ideas further by providing new theoretical insights for the resulting evidence lower bounds ( elbo s), extending these to explore the relative eﬃciency of diﬀerent approaches to proposal learning, and using our results to develop a new and improved training procedure.

Original: In particular, we introduce a method for expressing the gap between an elbo and the log marginal likelihood as a Kullback-Leibler ( kl ) divergence between two distributions on an extended sampling space.
Revised: In particular, we introduce a method for expressing the gap between an elbo and the log marginal likelihood as a Kullback-Leibler ( kl ) divergence between two distributions on an extended sampling space.

Original: Doing so allows us to investigate the behavior of this family of algorithms when the objective is maximized perfectly, which occurs only if the kl divergence becomes zero.
Revised: Doing so allows us to investigate the behavior of this family of algorithms when the objective is maximized perfectly, which occurs only if the kl divergence becomes zero.

Original: In the iwae case, this implies that the proposal distributions are equal to the posterior distributions under the learned model.
Revised: In the iwae case, this implies that the proposal distributions are equal to the posterior distributions under the learned model.

Original: In the aesmc case, it has implications for both the proposal distributions and the intermediate set of targets that are learned.
Revised: In the aesmc case, it has implications for both the proposal distributions and the intermediate set of targets that are learned.

Original: We demonstrate that, somewhat counter-intuitively, using lower variance estimates for the marginal likelihood can actually be harmful to proposal learning.
Revised: We demonstrate that, somewhat counter-intuitively, using lower variance estimates for the marginal likelihood can actually be harmful to proposal learning.

Original: In other words, we show that tighter is not necessarily better, and often substantially worse, in the context of variational bounds.
Revised: In other words, we show that tighter is not necessarily better, and often substantially worse, in the context of variational bounds.

Original: Using these insights, we propose an adaptation to the aesmc algorithm, which we call alternating elbo s, that uses diﬀerent lower bounds for updating the model parameters and proposal parameters.
Revised: Using these insights, we propose an adaptation to the aesmc algorithm, which we call alternating elbo s, that uses diﬀerent lower bounds for updating the model parameters and proposal parameters.

Original: We further show that this adaptation empirically leads to both improved model learning and proposal adaptation performance.
Revised: We further show that this adaptation empirically leads to both improved model learning and proposal adaptation performance.

Original:  Background 2. State-Space Models
Revised:  Background 2. State-Space Models

Original: State-space models ( ssm s) are probabilistic models over a set of latent variables x 1: T and observed variables y 1: T .
Revised: State-space models ( ssm s) are probabilistic models over a set of latent variables x 1: T and observed variables y 1: T .

Original: Given parameters θ , a ssm is characterized by an initial density µ θ ( x 1 ), a series of transition densities f t,θ ( x t | x 1: t − 1 ), and a series of emission densities g t,θ ( y t | x 1: t ) with the joint density being p θ ( x 1: T , y 1: T ) = µ θ ( x 1 ) Q Tt =2 f t,θ ( x t | x 1: t − 1 ) Q Tt =1 g t,θ ( y t | x 1: t ).
Revised: Given parameters θ , a ssm is characterized by an initial density µ θ ( x 1 ), a series of transition densities f t,θ ( x t | x 1: t − 1 ), and a series of emission densities g t,θ ( y t | x 1: t ) with the joint density being p θ ( x 1: T , y 1: T ) = µ θ ( x 1 ) Q Tt =2 f t,θ ( x t | x 1: t − 1 ) Q Tt =1 g t,θ ( y t | x 1: t ).

Original: We are usually interested in approximating the posterior p θ ( x 1: T | y 1: T ) or the expectation of some test function ϕ under this posterior
Revised: We are usually interested in approximating the posterior p θ ( x 1: T | y 1: T ) or the expectation of some test function ϕ under this posterior

Original: I ( ϕ ) :
Revised: I ( ϕ ) :

Original: = R ϕ ( x 1: T ) p θ ( x 1: T | y 1: T ) d x 1: T .
Revised: = R ϕ ( x 1: T ) p θ ( x 1: T | y 1: T ) d x 1: T .

Original: We refer to these two tasks as inference.
Revised: We refer to these two tasks as inference.

Original: Inference in models which are non-linear, non-discrete, and non-Gaussian is diﬃcult and one must resort to approximate methods, for which smc has been shown to be one of the most powerful approaches (Doucet & Johansen, 2009).
Revised: Inference in models which are non-linear, non-discrete, and non-Gaussian is diﬃcult and one must resort to approximate methods, for which smc has been shown to be one of the most powerful approaches (Doucet & Johansen, 2009).

Original: We will consider model learning as a problem of maximizing the marginal likelihood p θ ( y 1: T ) =
Revised: We will consider model learning as a problem of maximizing the marginal likelihood p θ ( y 1: T ) =

Original: R p θ ( x 1: T , y 1: T ) d x 1: T in the family of models parameterized by θ .
Revised: R p θ ( x 1: T , y 1: T ) d x 1: T in the family of models parameterized by θ .

Original: 2. Sequential Monte Carlo
Revised: 2. Sequential Monte Carlo

Original: smc performs approximate inference on a sequence of target distributions ( π t ( x 1: t ))
Revised: smc performs approximate inference on a sequence of target distributions ( π t ( x 1: t ))

Original: Tt =1 .
Revised: Tt =1 .

Original: In the context of ssm s, the target distributions are often taken to be ( p θ ( x 1: t | y 1: t ))
Revised: In the context of ssm s, the target distributions are often taken to be ( p θ ( x 1: t | y 1: t ))

Original: Tt =1 .
Revised: Tt =1 .

Original: Given a parameter φ and proposal distributions q 1 ,φ ( x 1 | y 1 ) and ( q t,φ ( x t | y 1: t , x 1: t − 1 ))
Revised: Given a parameter φ and proposal distributions q 1 ,φ ( x 1 | y 1 ) and ( q t,φ ( x t | y 1: t , x 1: t − 1 ))

Original: Tt =2 from which we can sample and whose densities we can evaluate, smc is described in Algorithm 1.
Revised: Tt =2 from which we can sample and whose densities we can evaluate, smc is described in Algorithm 1.

Original: Using the set of weighted particles (˜ x k 1: T , w kT )
Revised: Using the set of weighted particles (˜ x k 1: T , w kT )

Original: Kk =1 at the last time step, we can approximate the posterior as P Kk =1 ¯ w kT δ ˜ x k 1: T ( x 1: T ) and the integral I ϕ as P Kk =1 ¯ w kT ϕ (˜ x k 1: T ), where ¯ w kT := w kT / P j w jT is the normalized weight and δ z is a Dirac measure centred on z .
Revised: Kk =1 at the last time step, we can approximate the posterior as P Kk =1 ¯ w kT δ ˜ x k 1: T ( x 1: T ) and the integral I ϕ as P Kk =1 ¯ w kT ϕ (˜ x k 1: T ), where ¯ w kT := w kT / P j w jT is the normalized weight and δ z is a Dirac measure centred on z .

Original: Furthermore, one can obtain an unbiased estimator of the marginal likelihood p θ ( y 1: T ) using the intermediate particle weights:
Revised: Furthermore, one can obtain an unbiased estimator of the marginal likelihood p θ ( y 1: T ) using the intermediate particle weights:

Original: [Equation]
Revised: [Equation]

Original: Algorithm 1: Sequential Monte Carlo
Revised: Algorithm 1: Sequential Monte Carlo

Original: Data : observed values y 1: T , model parameters θ , proposal parameters φ
Revised: Data: observed values y 1: T , model parameters θ , proposal parameters φ

Original: Result : particles (˜ x k 1: T )
Revised: Result: particles (˜ x k 1: T )

Original: Kk =1 , weights ( w kT )
Revised: Kk =1 , weights ( w kT )

Original: Kk =1 , marginal likelihood estimate ˆ Z SMC begin
Revised: Kk =1 , marginal likelihood estimate ˆ Z SMC begin

Original: Sample initial particle values x k 1 ∼ q 1 ,φ ( ·| y 1 ).
Revised: Sample initial particle values x k 1 ∼ q 1 ,φ ( ·| y 1 ).

Original: Compute and normalize weights:
Revised: Compute and normalize weights:

Original: [Equation]
Revised: [Equation]

Original: Initialize particle set: ˜ x k 1 ← x kfor t = 2 , 3 , . . .
Revised: Initialize particle set: ˜ x k 1 ← x kfor t = 2 , 3 , . . .

Original: , T do
Revised: , T do

Original: Sample ancestor index a kt − 1 ∼ Discrete( ·| ¯ w 1 t − 1 , . . .
Revised: Sample ancestor index a kt − 1 ∼ Discrete( ·| ¯ w 1 t − 1 , . . .

Original: , ¯ w Kt − 1 ).
Revised: , ¯ w Kt − 1 ).

Original: Sample particle value x kt ∼ q t,φ ( ·| y 1: t , ˜ x a kt − 1 1: t − 1 ).
Revised: Sample particle value x kt ∼ q t,φ ( ·| y 1: t , ˜ x a kt − 1 1: t − 1 ).

Original: Update particle set ˜ x k 1: t ← (˜ x a kt − 1 1: t − 1 , x kt ).
Revised: Update particle set ˜ x k 1: t ← (˜ x a kt − 1 1: t − 1 , x kt ).

Original: Compute and normalize weights:
Revised: Compute and normalize weights:

Original: [Equation]
Revised: [Equation]

Original: q [Equation] Compute marginal likelihood: ˆ Z SMC = Q Tt =1 1 K P Kk =1 w kt .
Revised: Compute marginal likelihood: ˆ Z SMC = Q Tt =1 1 K P Kk =1 w kt .

Original: return particles (˜ x k 1: T )
Revised: return particles (˜ x k 1: T )

Original: Kk =1 , weights ( w kT )
Revised: Kk =1 , weights ( w kT )

Original: Kk =1 , marginal likelihood estimate ˆ Z SMC
Revised: Kk =1 , marginal likelihood estimate ˆ Z SMC

Original: The sequential nature of smc and the resampling step are crucial in making smc scalable to large T .
Revised: The sequential nature of smc and the resampling step are crucial in making smc scalable to large T .

Original: The former makes it easier to design eﬃcient proposal distributions as each step need only target the next set of variables x t .
Revised: The former makes it easier to design eﬃcient proposal distributions as each step need only target the next set of variables x t .

Original: The resampling step allows the algorithm to focus on promising particles in light of new observations, avoiding the exponential divergence between the weights of diﬀerent samples that occurs for importance sampling as T increases.
Revised: The resampling step allows the algorithm to focus on promising particles in light of new observations, avoiding the exponential divergence between the weights of diﬀerent samples that occurs for importance sampling as T increases.

Original: This can be demonstrated both empirically and theoretically (Del Moral, 2004, Chapter 9).
Revised: This can be demonstrated both empirically and theoretically (Del Moral, 2004, Chapter 9).

Original: We refer the reader to (Doucet & Johansen, 2009) for an in-depth treatment of smc .
Revised: We refer the reader to (Doucet & Johansen, 2009) for an in-depth treatment of smc .

Original: 2. Importance Weighted Auto-Encoders
Revised: 2. Importance Weighted Auto-Encoders

Original: Given a dataset of observations ( y ( n ) )
Revised: Given a dataset of observations ( y ( n ) )

Original: Nn =1 , a generative model p θ ( x, y ) and an inference network q φ ( x | y ), iwae s (Burda et al., 2016) maximize 1 N P Nn =1 elbo IS ( θ, φ, y ( n ) ) where, for a given observation y , the elbo IS (with K particles) is a lower bound on log p θ ( y ) by Jensen’s inequality:
Revised: Nn =1 , a generative model p θ ( x, y ) and an inference network q φ ( x | y ), iwae s (Burda et al., 2016) maximize 1 N P Nn =1 elbo IS ( θ, φ, y ( n ) ) where, for a given observation y , the elbo IS (with K particles) is a lower bound on log p θ ( y ) by Jensen’s inequality:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that for K = 1 particle, this objective reduces to a vae (Kingma & Welling, 2014;
Revised: Note that for K = 1 particle, this objective reduces to a vae (Kingma & Welling, 2014;

Original: Rezende et al., 2014) objective to which we will refer to as
Revised: Rezende et al., 2014) objective to which we will refer to as

Original: [Equation]
Revised: [Equation]

Original: The iwae optimization is performed using stochastic gradient ascent ( sga ) where a sample from (cid:16)Q Kk =1 q φ ( x k | y ( n ) ) (cid:17) is obtained using the reparameterization trick (Kingma &
Revised: The iwae optimization is performed using stochastic gradient ascent ( sga ) where a sample from (cid:16)Q Kk =1 q φ ( x k | y ( n ) ) (cid:17) is obtained using the reparameterization trick (Kingma &

Original: Welling, 2014) and the gradient 1 N P Nn =1 ∇ θ,φ log (cid:16)P Kk =1 p θ ( x k ,y ( n ) )
Revised: Welling, 2014) and the gradient 1 N P Nn =1 ∇ θ,φ log (cid:16)P Kk =1 p θ ( x k ,y ( n ) )

Original: q φ ( x k | y ( n ) ) (cid:17) is used to perform an optimization step.
Revised: q φ ( x k | y ( n ) ) (cid:17) is used to perform an optimization step.

Original:  Auto-Encoding Sequential Monte Carlo
Revised:  Auto-Encoding Sequential Monte Carlo

Original: aesmc implements model learning, proposal adaptation, and inference amortization in a similar manner to the vae and the iwae : it uses sga on an empirical average of the elbo over observations.
Revised: aesmc implements model learning, proposal adaptation, and inference amortization in a similar manner to the vae and the iwae : it uses sga on an empirical average of the elbo over observations.

Original: However, it varies in the form of this elbo .
Revised: However, it varies in the form of this elbo .

Original: In this section, we will introduce the aesmc elbo , explain how gradients of it can be estimated, and discuss the implications of these changes.
Revised: In this section, we will introduce the aesmc elbo , explain how gradients of it can be estimated, and discuss the implications of these changes.

Original: 3. Objective Function
Revised: 3. Objective Function

Original: Consider a family of ssm s { p θ ( x 1: T , y 1: T ) : θ ∈ Θ } and a family of proposal distributions { q φ ( x 1: T | y 1: T ) =
Revised: Consider a family of ssm s { p θ ( x 1: T , y 1: T ) : θ ∈ Θ } and a family of proposal distributions { q φ ( x 1: T | y 1: T ) =

Original: q 1 ,φ ( x 1 | y 1 ) Q T t =2 q t,φ ( x t | x 1: t − 1 , y 1: t ) : φ ∈ Φ } .
Revised: q 1 ,φ ( x 1 | y 1 ) Q T t =2 q t,φ ( x t | x 1: t − 1 , y 1: t ) : φ ∈ Φ } .

Original: Aesmc uses an elbo objective based on the smc marginal likelihood estimator (1).
Revised: Aesmc uses an elbo objective based on the smc marginal likelihood estimator (1).

Original: In particular, for a given y 1: T , the objective is deﬁned as
Revised: In particular, for a given y 1: T , the objective is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where ˆ Z SMC ( x 1: K 1: T , a 1: K 1: T − 1 ) is deﬁned in (1) and Q SMC is the sampling distribution of smc ,
Revised: where ˆ Z SMC ( x 1: K 1: T , a 1: K 1: T − 1 ) is deﬁned in (1) and Q SMC is the sampling distribution of smc ,

Original: Q SMC ( x 1: K 1: T , a 1: K 1: T − 1 ) =
Revised: Q SMC ( x 1: K 1: T , a 1: K 1: T − 1 ) =

Original: [Equation]
Revised: [Equation]

Original: Y
Revised: Y

Original: [Equation]
Revised: [Equation]

Original: elbo SMC forms a lower bound to the log marginal likelihood log p θ ( y 1: T ) due to Jensen’s inequality.
Revised: elbo SMC forms a lower bound to the log marginal likelihood log p θ ( y 1: T ) due to Jensen’s inequality.

Original: Hence, given a dataset ( y ( n ) 1: T ) Nn =1 , we can perform model parameter learning based on maximizing the lower bound of 1 N P Nn =1 log p θ ( y ( n ) 1: T ):
Revised: Hence, given a dataset ( y ( n ) 1: T ) Nn =1 , we can perform model parameter learning based on maximizing the lower bound of 1 N P Nn =1 log p θ ( y ( n ) 1: T ):

Original: [Equation]
Revised: [Equation]

Original: For notational convenience, we will talk about optimizing elbo s in the rest of this section.
Revised: For notational convenience, we will talk about optimizing elbo s in the rest of this section.

Original: However, we note that the main intended use of aesmc is to amortize over datasets, for which we replace the elbo gradient with an empirical sum as per (7).
Revised: However, we note that the main intended use of aesmc is to amortize over datasets, for which we replace the elbo gradient with an empirical sum as per (7).

Original: 3. Gradient Estimation
Revised: 3. Gradient Estimation

Original: We describe a gradient estimator used for optimizing elbo SMC ( θ, φ, y 1: T ) using sga .
Revised: We describe a gradient estimator used for optimizing elbo SMC ( θ, φ, y 1: T ) using sga .

Original: The smc sampler in Algorithm 1 proceeds by sampling x 1: K 1 , a 1: K 1 , x 1: K 2 , . . .
Revised: The smc sampler in Algorithm 1 proceeds by sampling x 1: K 1 , a 1: K 1 , x 1: K 2 , . . .

Original: sequentially from their respective distributions Q Kk =1 q 1 ( x k 1 ) , Q Kk =1 Discrete( a k 1 | w 1: K) , Q Kk =1 q 2 ( x k 2 | x a k 1 1 ) , . . .
Revised: sequentially from their respective distributions Q Kk =1 q 1 ( x k 1 ) , Q Kk =1 Discrete( a k 1 | w 1: K) , Q Kk =1 q 2 ( x k 2 | x a k 1 1 ) , . . .

Original: until the whole particle-weight trajectory ( x 1: T 1: K , a 1: K 1: T − 1 ) is sampled.
Revised: until the whole particle-weight trajectory ( x 1: T 1: K , a 1: K 1: T − 1 ) is sampled.

Original: From this trajectory, using equation (1), we can compute an estimation for the marginal likelihood.
Revised: From this trajectory, using equation (1), we can compute an estimation for the marginal likelihood.

Original: Assuming that the sampling of latent variables x 1: K 1: T is reparameterizable, we can make their sampling independent of ( θ, φ ).
Revised: Assuming that the sampling of latent variables x 1: K 1: T is reparameterizable, we can make their sampling independent of ( θ, φ ).

Original: In particular, assume that there exists a set of auxiliary random variables (cid:15) 1: K 1: T where (cid:15) kt ∼ s t and a set of reparameterization functions r t .
Revised: In particular, assume that there exists a set of auxiliary random variables (cid:15) 1: K 1: T where (cid:15) kt ∼ s t and a set of reparameterization functions r t .

Original: We can simulate the smc sampler as follows: sample (cid:15) 1: K∼ Q Kk =1 s 1 and set x k 1 = r 1 ( (cid:15) k 1 ), then sample a 1: Kfrom Q K k =

Revised: We can simulate the smc sampler as follows: sample (cid:15) 1: K∼ Q Kk =1 s 1 and set x k 1 = r 1 ( (cid:15) k 1 ), then sample a 1: Kfrom Q K k =

Original: Discrete( a k 1 | w 1: K), then sample (cid:15) 1: K∼ Q K k =1 s 2 and set x k 2 = r 2 ( (cid:15) k 2 , x a k 1 1 ), until we obtain ( x 1: T 1: K , a 1: K 1: T − 1 ).
Revised: Discrete( a k 1 | w 1: K), then sample (cid:15) 1: K∼ Q K k =1 s 2 and set x k 2 = r 2 ( (cid:15) k 2 , x a k 1 1 ), until we obtain ( x 1: T 1: K , a 1: K 1: T − 1 ).

Original: We use this reparameterized sample of ( x 1: T 1: K , a 1: K 1: T − 1 ) to evaluate the gradient estimator ∇ θ,φ log ˆ Z SMC ( x 1: K 1: T , a 1: K 1: T − 1 ).
Revised: We use this reparameterized sample of ( x 1: T 1: K , a 1: K 1: T − 1 ) to evaluate the gradient estimator ∇ θ,φ log ˆ Z SMC ( x 1: K 1: T , a 1: K 1: T − 1 ).

Original: To account for the discrete choices of ancestor indices a kt we can additionally use the reinforce (Williams, 1992) trick, however in practice, we ﬁnd that the additional term in the estimator has problematically high variance.
Revised: To account for the discrete choices of ancestor indices a kt we can additionally use the reinforce (Williams, 1992) trick, however in practice, we ﬁnd that the additional term in the estimator has problematically high variance.

Original: We explore various other possible gradient estimators and empirical assessments of their variances in Appendix A.
Revised: We explore various other possible gradient estimators and empirical assessments of their variances in Appendix A.

Original: This exploration conﬁrms that including the additional reinforce terms leads to problematically high variance, justifying our decision to omit them, despite introducing a small bias into the gradient estimates.
Revised: This exploration conﬁrms that including the additional reinforce terms leads to problematically high variance, justifying our decision to omit them, despite introducing a small bias into the gradient estimates.

Original: 3. Bias & Implications on the Proposals
Revised: 3. Bias & Implications on the Proposals

Original: In this section, we express the gap between elbo s and the log marginal likelihood as a kl divergence and study implications on the proposal distributions.
Revised: In this section, we express the gap between elbo s and the log marginal likelihood as a kl divergence and study implications on the proposal distributions.

Original: We present a set of claims and propositions whose full proofs are in Appendix B.
Revised: We present a set of claims and propositions whose full proofs are in Appendix B.

Original: These give insight into the behavior of aesmc , compared with alternatives and show the advantages, and disadvantages, of using our diﬀerent elbo .
Revised: These give insight into the behavior of aesmc , compared with alternatives and show the advantages, and disadvantages, of using our diﬀerent elbo .

Original: This insight motivates Section 4 which proposes an algorithm for improving proposal learning.
Revised: This insight motivates Section 4 which proposes an algorithm for improving proposal learning.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Given an unnormalized target density ˜ P : X → [0 , ∞ ) with normalizing constant Z P > 0 , P := ˜ P/Z P , and a proposal density Q : X → [0 , ∞ ) , the elbo
Revised: Given an unnormalized target density ˜ P : X → [0 , ∞ ) with normalizing constant Z P > 0 , P := ˜ P/Z P , and a proposal density Q : X → [0 , ∞ ) , the elbo

Original: [Equation]
Revised: [Equation]

Original: is a lower bound on log Z P and satisﬁes
Revised: is a lower bound on log Z P and satisﬁes

Original: [Equation]
Revised: [Equation]

Original: This is a standard identity used in variational inference (Wainwright et al., 2008) and vae s.
Revised: This is a standard identity used in variational inference (Wainwright et al., 2008) and vae s.

Original: In the case of vae s, applying Deﬁnition 1 with P being p θ ( x | y ), ˜ P being p θ ( x, y ),
Revised: In the case of vae s, applying Deﬁnition 1 with P being p θ ( x | y ), ˜ P being p θ ( x, y ),

Original: Z P being p θ ( y ), and Q being q φ ( x | y ), we directly can rewrite (4) as elbo VAE ( θ, φ, y )
Revised: Z P being p θ ( y ), and Q being q φ ( x | y ), we directly can rewrite (4) as elbo VAE ( θ, φ, y )

Original: = log p θ ( y ) − kl ( q φ ( x
Revised: = log p θ ( y ) − kl ( q φ ( x

Original: [Equation]
Revised: [Equation]

Original: The key observation for expressing such a bound for general elbo s such as elbo IS and elbo
Revised: The key observation for expressing such a bound for general elbo s such as elbo IS and elbo

Original: SMC is that the target density P and the proposal density Q need not directly correspond to p θ ( x | y ) and q φ ( x | y ).
Revised: SMC is that the target density P and the proposal density Q need not directly correspond to p θ ( x | y ) and q φ ( x | y ).

Original: This allows us to view the underlying sampling distributions of the marginal likelihood Monte Carlo estimators such as Q IS in (3) and Q SMC in (6) as proposal distributions on an extended space X .
Revised: This allows us to view the underlying sampling distributions of the marginal likelihood Monte Carlo estimators such as Q IS in (3) and Q SMC in (6) as proposal distributions on an extended space X .

Original: The following claim uses this observation to express the bound between a general elbo and the log marginal likelihood as kl divergence from the extended space sampling distribution to a corresponding target distribution.
Revised: The following claim uses this observation to express the bound between a general elbo and the log marginal likelihood as kl divergence from the extended space sampling distribution to a corresponding target distribution.

Original: Claim 1.
Revised: Claim 1.

Original: Given a non-negative unbiased estimator ˆ Z P ( x ) ≥ 0 of the normalizing constant
Revised: Given a non-negative unbiased estimator ˆ Z P ( x ) ≥ 0 of the normalizing constant

Original: Z P where x is distributed from the proposal distribution Q ( x ) , the following holds:
Revised: Z P where x is distributed from the proposal distribution Q ( x ) , the following holds:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: is a normalized target density.
Revised: is a normalized target density.

Original: In the case of iwae s
Revised: In the case of iwae s

Original: [Equation]
Revised: [Equation]

Original: we can apply Claim 1 with Q and ˆ Z P being Q IS and ˆ Z IS deﬁned in (3) and Z P being p θ ( y ).
Revised: we can apply Claim 1 with Q and ˆ Z P being Q IS and ˆ Z IS deﬁned in (3) and Z P being p θ ( y ).

Original: This yields
Revised: This yields

Original: [Equation]
Revised: [Equation]

Original: , where
Revised: , where

Original: [Equation]
Revised: [Equation]

Original: Similarly, in the case of aesmc
Revised: Similarly, in the case of aesmc

Original: [Equation]
Revised: [Equation]

Original: we obtain elbo
Revised: we obtain elbo

Original: [Equation]
Revised: [Equation]

Original: , where
Revised: , where

Original: [Equation]
Revised: [Equation]

Original: Having expressions for the target distribution P and the sampling distribution Q for a given elbo allows us to investigate what happens when we maximize that elbo , remembering that the kl term is strictly non-negative and zero if and only if P = Q .
Revised: Having expressions for the target distribution P and the sampling distribution Q for a given elbo allows us to investigate what happens when we maximize that elbo , remembering that the kl term is strictly non-negative and zero if and only if P = Q .

Original: For the vae and iwae cases then, provided the proposal is suﬃciently ﬂexible, one can always perfectly maximize the elbo by setting p θ ( x | y )
Revised: For the vae and iwae cases then, provided the proposal is suﬃciently ﬂexible, one can always perfectly maximize the elbo by setting p θ ( x | y )

Original: =
Revised: =

Original: q φ ( x | y ) for all x .
Revised: q φ ( x | y ) for all x .

Original: The reverse implication also holds: if elbo
Revised: The reverse implication also holds: if elbo

Original: VAE = log Z P then it must be the case that p θ ( x | y ) =
Revised: VAE = log Z P then it must be the case that p θ ( x | y ) =

Original: q φ ( x | y ).
Revised: q φ ( x | y ).

Original: However, for aesmc , achieving elbo = log Z P is only possible when one also has suﬃcient ﬂexibility to learn a particular series of intermediate target distributions, namely the marginals of the ﬁnal target distribution.
Revised: However, for aesmc , achieving elbo = log Z P is only possible when one also has suﬃcient ﬂexibility to learn a particular series of intermediate target distributions, namely the marginals of the ﬁnal target distribution.

Original: In other words, it is necessary to learn a particular factorization of the generative model, not just the correct individual proposals, to achieve P = Q .
Revised: In other words, it is necessary to learn a particular factorization of the generative model, not just the correct individual proposals, to achieve P = Q .

Original: These observations are formalized in Propositions 1 and 2 below.
Revised: These observations are formalized in Propositions 1 and 2 below.

Original: Proposition 1.
Revised: Proposition 1.

Original: Q IS ( x 1: K ) =
Revised: Q IS ( x 1: K ) =

Original: P IS ( x 1: K ) for all x 1: K
Revised: P IS ( x 1: K ) for all x 1: K

Original: if and only if q ( x | y )
Revised: if and only if q ( x | y )

Original: =
Revised: =

Original: p ( x | y ) for all x .
Revised: p ( x | y ) for all x .

Original: Proposition 2.
Revised: Proposition 2.

Original: If K > 1 , then P SMC ( x 1: K 1: T , a 1: K 1: T − 1 ) = Q SMC ( x 1: K 1: T , a 1: K 1: T − 1 ) for all ( x 1: K 1: T , a 1: K 1: T − 1 ) if and only if
Revised: If K > 1 , then P SMC ( x 1: K 1: T , a 1: K 1: T − 1 ) = Q SMC ( x 1: K 1: T , a 1: K 1: T − 1 ) for all ( x 1: K 1: T , a 1: K 1: T − 1 ) if and only if

Original: π t ( x 1: t ) =
Revised: π t ( x 1: t ) =

Original: R p ( x 1: T | y 1: T ) d x t +1: T = p ( x 1: t | y 1: T ) for all x 1: t and t = 1 , . .
Revised: R p ( x 1: T | y 1: T ) d x t +1: T = p ( x 1: t | y 1: T ) for all x 1: t and t = 1 , . .

Original: , T , and 2.
Revised: , T , and 2.

Original: q 1 ( x 1 | y 1 )
Revised: q 1 ( x 1 | y 1 )

Original: =
Revised: =

Original: p ( x 1 | y 1: T ) for all x 1 and q t ( x t | x 1: t − 1 , y 1: t ) = p ( x 1: t | y 1: T ) /p
Revised: p ( x 1 | y 1: T ) for all x 1 and q t ( x t | x 1: t − 1 , y 1: t ) = p ( x 1: t | y 1: T ) /p

Original: ( x 1: t − 1 | y 1: T ) for t = 2 , . .
Revised: ( x 1: t − 1 | y 1: T ) for t = 2 , . .

Original: , T for all x 1: t ,
Revised: , T for all x 1: t ,

Original: where π t ( x 1: t ) are the intermediate targets used by smc .
Revised: where π t ( x 1: t ) are the intermediate targets used by smc .

Original: Proposition 2 has the consequence that if the family of generative models is such that condition 1 does not hold, we will not be able to make the bound tight elbo SMC =
Revised: Proposition 2 has the consequence that if the family of generative models is such that condition 1 does not hold, we will not be able to make the bound tight elbo SMC =

Original: Z P .
Revised: Z P .

Original: This means that, except for a very small class of models, then, for most convenient parameterizations, it will be impossible to learn a perfect proposal that gives a tight bound, i.e. there will be no θ and φ such that the above conditions can be satisﬁed.
Revised: This means that, except for a very small class of models, then, for most convenient parameterizations, it will be impossible to learn a perfect proposal that gives a tight bound, i.e. there will be no θ and φ such that the above conditions can be satisﬁed.

Original: However, it also means that elbo
Revised: However, it also means that elbo

Original: SMC encodes important additional information about the implications the factorization of the generative model has on the inference—the model depends only on the ﬁnal target π T ( x 1: T )
Revised: SMC encodes important additional information about the implications the factorization of the generative model has on the inference—the model depends only on the ﬁnal target π T ( x 1: T )

Original: =
Revised: =

Original: p θ ( x 1: T | y 1: T ), but some choices of the intermediate targets π t ( x 1: t ) will lead to much more eﬃcient inference than others.
Revised: p θ ( x 1: T | y 1: T ), but some choices of the intermediate targets π t ( x 1: t ) will lead to much more eﬃcient inference than others.

Original: Perhaps more importantly, smc is usually a far more powerful inference algorithm than importance sampling and so the aesmc setup allows for more ambitious model learning problems to be eﬀectively tackled than the vae or iwae .
Revised: Perhaps more importantly, smc is usually a far more powerful inference algorithm than importance sampling and so the aesmc setup allows for more ambitious model learning problems to be eﬀectively tackled than the vae or iwae .

Original: After all, even though it is well known in the smc literature that, unlike for importance sampling, most problems have no perfect set of smc proposals which will generate exact samples from the posterior (Doucet & Johansen, 2009), smc is still gives superior performance on most problems with more than a few dimensions.
Revised: After all, even though it is well known in the smc literature that, unlike for importance sampling, most problems have no perfect set of smc proposals which will generate exact samples from the posterior (Doucet & Johansen, 2009), smc is still gives superior performance on most problems with more than a few dimensions.

Original: These intuitions are backed up by our experiments that show that using elbo SMC regularly learns better models than using elbo
Revised: These intuitions are backed up by our experiments that show that using elbo SMC regularly learns better models than using elbo

Original: IS .
Revised: IS .

Original:  Improving Proposal Learning
Revised:  Improving Proposal Learning

Original: Given the implications from the previous section, we now ask whether optimizing elbo IS and elbo
Revised: Given the implications from the previous section, we now ask whether optimizing elbo IS and elbo

Original: SMC actually improves the proposal distribution?
Revised: SMC actually improves the proposal distribution?

Original: In other words, does the optimization procedure make q φ ( x 1: T | y 1: T ) closer to p θ ( x 1: T | y 1: T ) and how does the number of particles K aﬀect this?
Revised: In other words, does the optimization procedure make q φ ( x 1: T | y 1: T ) closer to p θ ( x 1: T | y 1: T ) and how does the number of particles K aﬀect this?

Original: In the vae case, we are directly optimizing kl ( q φ ( x | y )
Revised: In the vae case, we are directly optimizing kl ( q φ ( x | y )

Original: ||
Revised: ||

Original: p θ ( x | y )) for ﬁxed model parameters and so it is straight forward to see that we will induce proposal learning.
Revised: p θ ( x | y )) for ﬁxed model parameters and so it is straight forward to see that we will induce proposal learning.

Original: In the iwae and aesmc cases, such optimization only minimizes kl ( Q IS || P IS ) and kl ( Q SMC || P SMC ) respectively, which does not directly imply that kl ( q φ ( x | y )
Revised: In the iwae and aesmc cases, such optimization only minimizes kl ( Q IS || P IS ) and kl ( Q SMC || P SMC ) respectively, which does not directly imply that kl ( q φ ( x | y )

Original: ||
Revised: ||

Original: p θ ( x | y )) is small.
Revised: p θ ( x | y )) is small.

Original: [Figure]
Revised: [Figure]

Original: Counter-intuitively, it transpires that the tighter bounds implied by using a larger K is often harmful to proposal learning for both iwae and aesmc .
Revised: Counter-intuitively, it transpires that the tighter bounds implied by using a larger K is often harmful to proposal learning for both iwae and aesmc .

Original: At a high-level, this is because an accurate estimate for ˆ
Revised: At a high-level, this is because an accurate estimate for ˆ

Original: Z P can be achieved for a wide range of proposal parameters φ and so the magnitude of ∇ φ elbo reduces as K increases.
Revised: Z P can be achieved for a wide range of proposal parameters φ and so the magnitude of ∇ φ elbo reduces as K increases.

Original: Typically, this shrinkage happens faster than increasing K reduces the standard deviation of the estimate and so the signal-to-noise ratio ( snr ) for the gradient estimate actually decreases, even though it is a lower variance estimate.
Revised: Typically, this shrinkage happens faster than increasing K reduces the standard deviation of the estimate and so the signal-to-noise ratio ( snr ) for the gradient estimate actually decreases, even though it is a lower variance estimate.

Original: This eﬀect is demonstrated in Figure 1 which shows a kernel density estimator for the distribution of the elbo gradient estimate for diﬀerent K and the model given in Section 5.2.
Revised: This eﬀect is demonstrated in Figure 1 which shows a kernel density estimator for the distribution of the elbo gradient estimate for diﬀerent K and the model given in Section 5.2.

Original: Here we see that as we increase K , both the expected gradient estimate (i.e. true gradient) and standard deviation of the estimate decrease.
Revised: Here we see that as we increase K , both the expected gradient estimate (i.e. true gradient) and standard deviation of the estimate decrease.

Original: However, the former decreases faster and so the snr decreases.
Revised: However, the former decreases faster and so the snr decreases.

Original: This is perhaps easiest to appreciate by noting that for K ≥ 10, there is a roughly equal probability of the estimate being positive or negative, such that we are equally likely to increase or decrease the parameter value at the next sga iteration, inevitably leading to poor performance.
Revised: This is perhaps easiest to appreciate by noting that for K > 10, there is a roughly equal probability of the estimate being positive or negative, such that we are equally likely to increase or decrease the parameter value at the next sga iteration, inevitably leading to poor performance.

Original: On the other hand, when K = 1, it is far more likely that the gradient estimate is positive than negative, and so there is clear drift to the gradient steps.
Revised: On the other hand, when K = 1, it is far more likely that the gradient estimate is positive than negative, and so there is clear drift to the gradient steps.

Original: We add to the empirical evidence for this behavior is Section 5.
Revised: We add to the empirical evidence for this behavior is Section 5.

Original: Note the critical diﬀerence for model learning is that ∇ θ elbo does not, in general, decrease in magnitude as K increases.
Revised: Note the critical diﬀerence for model learning is that ∇ θ elbo does not, in general, decrease in magnitude as K increases.

Original: Note also that using a larger K should always give better performance at test time—the implication of our result is that it may be better to learn φ using a smaller K .
Revised: Note also that using a larger K should always give better performance at test time—the implication of our result is that it may be better to learn φ using a smaller K .

Original: We can further demonstrate this result using an informal theoretical argument.
Revised: We can further demonstrate this result using an informal theoretical argument.

Original: Turner &
Revised: Turner &

Original: Sahani (2011) previously showed that tighter bounds can be detrimental in some scenarios for variational inference.
Revised: Sahani (2011) previously showed that tighter bounds can be detrimental in some scenarios for variational inference.

Original: As well as its diﬀerent context, our distinct analysis goes beyond their observations by showing that for the case of the iwae , the gradient estimates for the proposal degrade arbitrarily for suﬃciently large K .
Revised: As well as its diﬀerent context, our distinct analysis goes beyond their observations by showing that for the case of the iwae , the gradient estimates for the proposal degrade arbitrarily for suﬃciently large K .

Original: Our gradient estimate for the K particle iwae is
Revised: Our gradient estimate for the K particle iwae is

Original: [Equation]
Revised: [Equation]

Original: where I =
Revised: where I =

Original: lim K →∞
Revised: lim K →∞

Original: I K
Revised: I K

Original: = 0 because with inﬁnite samples, the estimate is exact and thus independent of the proposal parameters.
Revised: = 0 because with inﬁnite samples, the estimate is exact and thus independent of the proposal parameters.

Original: Now adapting the iwae result of Rainforth et al.
Revised: Now adapting the iwae result of Rainforth et al.

Original: (2017) shows that
Revised: (2017) shows that

Original: [Equation]
Revised: [Equation]

Original: (17) where C 0 , κ 0 ( K 0 in Rainforth et al.
Revised: (17) where C 0 , κ 0 ( K 0 in Rainforth et al.

Original: and ς 1 are constants and we have set N =in their formulation.
Revised: and ς 1 are constants and we have set N =in their formulation.

Original: Here the ﬁrst C 20 ς 41 4 K 2 and O (cid:0) 1 K 3 (cid:1) are “bias terms”, i.e. ( E [ I K ]) 

Revised: Here the ﬁrst C 20 ς 41 4 K 2 and O (cid:0) 1 K 3 (cid:1) are “bias terms”, i.e. ( E [ I K ])

Original: = ( ∇ φ elbo ) 2 , and the rest are variance terms.
Revised: = ( ∇ φ elbo ) 2 , and the rest are variance terms.

Original: We can now deﬁne our snr as follows
Revised: We can now deﬁne our snr as follows

Original: [Equation]
Revised: [Equation]

Original: where we have substituted in the bounds for the bias and variance from (17) and the approximations will, in general, become increasingly exact as K increases.
Revised: where we have substituted in the bounds for the bias and variance from (17) and the approximations will, in general, become increasingly exact as K increases.

Original: We thus see that increasing K reduces the snr and so a lower K is preferable for proposal learning.
Revised: We thus see that increasing K reduces the snr and so a lower K is preferable for proposal learning.

Original: 4. Alternating elbo s
Revised: 4. Alternating elbo s

Original: To address these issues, we propose the alternating elbo s ( alt ) algorithm which updates ( θ, φ ) in a coordinate descent fashion using diﬀerent elbo s, and thus gradient estimates, for each.
Revised: To address these issues, we propose the alternating elbo s ( alt ) algorithm which updates ( θ, φ ) in a coordinate descent fashion using diﬀerent elbo s, and thus gradient estimates, for each.

Original: We pick a θ -optimizing pair and a φ -optimizing pair ( A θ , K θ ) , ( A φ , K φ ) ∈ { is , smc } × { 1 , 2 , . . . } , corresponding to a inference type number of particles.
Revised: We pick a θ -optimizing pair and a φ -optimizing pair ( A θ , K θ ) , ( A φ , K φ ) ∈ { is , smc } × { 1 , 2 , . . . } , corresponding to a inference type number of particles.

Original: In an optimization step, we obtain an estimator for ∇ θ elbo A θ with K θ particles and an estimator for ∇ φ elbo A φ with K
Revised: In an optimization step, we obtain an estimator for ∇ θ elbo A θ with K θ particles and an estimator for ∇ φ elbo A φ with K

Original: φ particles which we call g θ and g φ respectively.
Revised: φ particles which we call g θ and g φ respectively.

Original: We use g θ to update the current θ and g φ to update the current φ .
Revised: We use g θ to update the current θ and g φ to update the current φ .

Original: The results from the previous sections suggest that using
Revised: The results from the previous sections suggest that using

Original: A θ = smc
Revised: A θ = smc

Original: and A φ = is with a large K θ and a small K φ should perform better model and proposal learning than just ﬁxing ( A θ , K θ ) =
Revised: and A φ = is with a large K θ and a small K φ should perform better model and proposal learning than just ﬁxing ( A θ , K θ ) =

Original: ( A φ , K φ ) to ( smc , large) since using A φ = is with small K φ helps learning φ and using A θ
Revised: ( A φ , K φ ) to ( smc , large) since using A φ = is with small K φ helps learning φ and using A θ

Original: = smc with large K θ helps learning θ .
Revised: = smc with large K θ helps learning θ .

Original: We experimentally ﬁnd that this procedure in fact improves both model and proposal learning, leading to a new algorithm that improves the aesmc approach even further.
Revised: We experimentally ﬁnd that this procedure in fact improves both model and proposal learning, leading to a new algorithm that improves the aesmc approach even further.

Original:  Experiments
Revised:  Experiments

Original: We now present a series of experiments designed to answer the following questions: 1) Does reducing the gap by using more particles or a better inference procedure lead to an adverse eﬀect on proposal learning? 2)
Revised: We now present a series of experiments designed to answer the following questions: 1) Does reducing the gap by using more particles or a better inference procedure lead to an adverse eﬀect on proposal learning? 2)

Original: Can aesmc , despite this eﬀect, outperform iwae ?
Revised: Can aesmc , despite this eﬀect, outperform iwae ?

Original: Can we further improve the learned model and proposal by using our newly proposed algorithm alt ?
Revised: Can we further improve the learned model and proposal by using our newly proposed algorithm alt ?

Original: First we investigate a linear Gaussian state space model ( lgssm ) for model learning and a latent variable model for proposal adaptation.
Revised: First we investigate a linear Gaussian state space model ( lgssm ) for model learning and a latent variable model for proposal adaptation.

Original: This allows us to compare the learned parameters to the optimal ones.
Revised: This allows us to compare the learned parameters to the optimal ones.

Original: Doing so, we conﬁrm our conclusions for this simple problem.
Revised: Doing so, we conﬁrm our conclusions for this simple problem.

Original: We then extend those results to more complex, high dimensional observation spaces that require models and proposals parameterized by neural networks.
Revised: We then extend those results to more complex, high dimensional observation spaces that require models and proposals parameterized by neural networks.

Original: We do so by investigating the Moving Agents dataset, a set of partially occluded video sequences.
Revised: We do so by investigating the Moving Agents dataset, a set of partially occluded video sequences.

Original: 5. Linear Gaussian State Space Model
Revised: 5. Linear Gaussian State Space Model

Original: Given the following lgssm
Revised: Given the following lgssm

Original: [Equation]
Revised: [Equation]

Original: we ﬁnd that optimizing elbo SMC ( θ, φ, y 1: T ) w.r.t θ leads to better generative models than optimizing elbo IS ( θ, φ, y 1: T ).
Revised: we ﬁnd that optimizing elbo SMC ( θ, φ, y 1: T ) w.r.t θ leads to better generative models than optimizing elbo IS ( θ, φ, y 1: T ).

Original: The same is true for using more particles.
Revised: The same is true for using more particles.

Original: To do so, we generate a sequence y 1: T for T = 200 by sampling from the model with θ =
Revised: To do so, we generate a sequence y 1: T for T = 200 by sampling from the model with θ =

Original: ( θ 1 , θ 2 ) =
Revised: ( θ 1 , θ 2 ) =

Original: We then optimize the diﬀerent elbo s w.r.t θ using the bootstrap proposal q 1 ( x 1 | y 1 ) =
Revised: We then optimize the diﬀerent elbo s w.r.t θ using the bootstrap proposal q 1 ( x 1 | y 1 ) =

Original: µ θ ( x 1 ) and q t ( x t | x 1: t − 1 , y 1: t ) =
Revised: µ θ ( x 1 ) and q t ( x t | x 1: t − 1 , y 1: t ) =

Original: f t,θ ( x t | x 1: t − 1 ).
Revised: f t,θ ( x t | x 1: t − 1 ).

Original: Because we use the bootstrap proposal, gradients w.r.t to θ are not backpropagated through q .
Revised: Because we use the bootstrap proposal, gradients w.r.t to θ are not backpropagated through q .

Original: We use a ﬁxed learning rate of 0 . 01 and optimize for 500 steps.
Revised: We use a ﬁxed learning rate of 0 . 01 and optimize for 500 steps.

Original: Figure 2 shows that the convergence of both log p θ ( y 1: T ) to max θ log p θ ( y 1: T ) and θ to argmax θ log p θ ( y 1: T ) is faster when elbo SMC and more particles are used.
Revised: Figure 2 shows that the convergence of both log p θ ( y 1: T ) to max θ log p θ ( y 1: T ) and θ to argmax θ log p θ ( y 1: T ) is faster when elbo SMC and more particles are used.

Original: 5. Proposal Learning
Revised: 5. Proposal Learning

Original: We now investigate how learning φ , i.e. the proposal, is aﬀected by the the choice of elbo and the number of particles.
Revised: We now investigate how learning φ , i.e. the proposal, is aﬀected by the the choice of elbo and the number of particles.

Original: Consider a simple, ﬁxed generative model p ( µ ) p ( x | µ )
Revised: Consider a simple, ﬁxed generative model p ( µ ) p ( x | µ )

Original: =
Revised: =

Original: Normal( µ ; 0 , 1 2 )Normal( x ; µ, 1 2 ) where µ and x are the latent and observed variables respectively and a family of proposal distributions
Revised: Normal( µ ; 0 , 1 2 )Normal( x ; µ, 1 2 ) where µ and x are the latent and observed variables respectively and a family of proposal distributions

Original: q φ ( µ ) =
Revised: q φ ( µ ) =

Original: Normal( µ ; µ q , σ 2 q ) parameterized by φ = ( µ q , log σ 2 q ).
Revised: Normal( µ ; µ q , σ 2 q ) parameterized by φ = ( µ q , log σ 2 q ).

Original: For a ﬁxed observation x = 2 .
Revised: For a ﬁxed observation x = 2 .

Original: 3, we initialize φ = (0 . 01 , 0 . 01) and optimize elbo IS with respect to φ .
Revised: 3, we initialize φ = (0 . 01 , 0 . 01) and optimize elbo IS with respect to φ .

Original: We investigate the quality of the learned parameter φ as we increase the number of particles
Revised: We investigate the quality of the learned parameter φ as we increase the number of particles

Original: K during training.
Revised: K during training.

Original: Figure 3 (left) clearly demonstrates that the quality of φ compared to the analytic posterior decreases as we increase K .
Revised: Figure 3 (left) clearly demonstrates that the quality of φ compared to the analytic posterior decreases as we increase K .

Original: Similar behavior is observed in Figure 3 (middle, right) where we optimize elbo SMC with respect to both θ and φ for the lgssm described in Section 5.1.
Revised: Similar behavior is observed in Figure 3 (middle, right) where we optimize elbo SMC with respect to both θ and φ for the lgssm described in Section 5.1.

Original: We see that using more
Revised: We see that using more

Original: [Figure]
Revised: [Figure]

Original: particles helps model learning but makes proposal learning worse.
Revised: particles helps model learning but makes proposal learning worse.

Original: Using our alt algorithm alleviates this problem and at the same time makes model learning faster as it proﬁts from a more accurate proposal distribution
Revised: Using our alt algorithm alleviates this problem and at the same time makes model learning faster as it proﬁts from a more accurate proposal distribution

Original: [Figure]
Revised: [Figure]

Original: 5. Moving Agents
Revised: 5. Moving Agents

Original: To show that our results are applicable to complex, high dimensional data we compare alt , aesmc and iwae on stochastic, partially observable video sequences.
Revised: To show that our results are applicable to complex, high dimensional data we compare alt , aesmc and iwae on stochastic, partially observable video sequences.

Original: Figure 7 in the appendix shows an example of such a sequence.
Revised: Figure 7 in the appendix shows an example of such a sequence.

Original: The dataset consists of 5000 sequences of images ( y ( i ) 0:39 ) 50001 of which 1000 are randomly held out as test set.
Revised: The dataset consists of 5000 sequences of images ( y ( i ) 0:39 ) 50001 of which 1000 are randomly held out as test set.

Original: Each sequence contains 40 images represented as a 3 dimensional array of size 1 × 32 × 32.
Revised: Each sequence contains 40 images represented as a 3 dimensional array of size 1 × 32 × 32.

Original: In each sequence there is one agent, represented as circle, whose starting position is sampled randomly along the top and bottom of the image.
Revised: In each sequence there is one agent, represented as circle, whose starting position is sampled randomly along the top and bottom of the image.

Original: The dataset is inspired by (Ondruska & Posner, 2016), however with the crucial diﬀerence that the movement of the agent is stochastic .
Revised: The dataset is inspired by (Ondruska & Posner, 2016), however with the crucial diﬀerence that the movement of the agent is stochastic .

Original: The agent performs a directed random walk through the image.
Revised: The agent performs a directed random walk through the image.

Original: At each timestep, it moves according to
Revised: At each timestep, it moves according to

Original: [Equation]
Revised: [Equation]

Original: where ( x t , y t ) are the coordinates in frame t in a unit square that is then projected onto 32 × 32 pixels.
Revised: where ( x t , y t ) are the coordinates in frame t in a unit square that is then projected onto 32 × 32 pixels.

Original: In addition to the stochasticity of the movement, half of the image is occluded, preventing the agent to be observed.
Revised: In addition to the stochasticity of the movement, half of the image is occluded, preventing the agent to be observed.

Original: As generative model and proposal distribution we use a Variational Recurrent Neural Network ( vrnn ) (Chung et al., 2015).
Revised: As generative model and proposal distribution we use a Variational Recurrent Neural Network ( vrnn ) (Chung et al., 2015).

Original: It extends recurrent neural networks ( rnn s) by introducing a stochastic latent state x t at each timestep t .
Revised: It extends recurrent neural networks ( rnn s) by introducing a stochastic latent state x t at each timestep t .

Original: Together with the observation y t , this state conditions the deterministic transition of the rnn .
Revised: Together with the observation y t , this state conditions the deterministic transition of the rnn .

Original: By introducing this unobserved stochastic state, the vrnn is able to better model complex long range variability in stochastic sequences.
Revised: By introducing this unobserved stochastic state, the vrnn is able to better model complex long range variability in stochastic sequences.

Original: Architecture and hyperparameter details are given in the appendix.
Revised: Architecture and hyperparameter details are given in the appendix.

Original: One can see in Figure 4 that models trained using the alternating training schedule alt outperform aesmc which in turn outperform iwae .
Revised: One can see in Figure 4 that models trained using the alternating training schedule alt outperform aesmc which in turn outperform iwae .

Original: Using more particles improves the elbo , but more so for aesmc and alt as the diﬀerences in estimators and training methods become more pronounced with higher particle numbers.
Revised: Using more particles improves the elbo , but more so for aesmc and alt as the diﬀerences in estimators and training methods become more pronounced with higher particle numbers.

Original: In the appendix we inspect diﬀerent learned generative models by using them for prediction, conﬁrming the results presented here.
Revised: In the appendix we inspect diﬀerent learned generative models by using them for prediction, conﬁrming the results presented here.

Original: [Figure]
Revised: [Figure]

Original: model.
Revised: model.

Original:  Conclusions
Revised:  Conclusions

Original: We have developed aesmc —a method for performing model learning using a new elbo objective which is based on the smc marginal likelihood estimator.
Revised: We have developed aesmc —a method for performing model learning using a new elbo objective which is based on the smc marginal likelihood estimator.

Original: This elbo objective is optimized using sga and the reparameterization trick.
Revised: This elbo objective is optimized using sga and the reparameterization trick.

Original: Our approach utilizes the eﬃciency of smc in models with intermediate observations and hence is suitable for a highly structured models.
Revised: Our approach utilizes the eﬃciency of smc in models with intermediate observations and hence is suitable for a highly structured models.

Original: We experimentally demonstrated that this objective has a tighter gap to the log marginal likelihood than the iwae objective and that it works well on structured problems such as learning the generative model of moving balls which can be used for tasks such as reconstruction or prediction.
Revised: We experimentally demonstrated that this objective has a tighter gap to the log marginal likelihood than the iwae objective and that it works well on structured problems such as learning the generative model of moving balls which can be used for tasks such as reconstruction or prediction.

Original: Additionally, in Claim 1, we provide a simple way to express the bias of objectives induced by log of marginal likelihood estimators as a kl divergence.
Revised: Additionally, in Claim 1, we provide a simple way to express the bias of objectives induced by log of marginal likelihood estimators as a kl divergence.

Original: In Propositions 1 and 2, we investigate the implications of these kl s being zero in the case of iwae and aesmc .
Revised: In Propositions 1 and 2, we investigate the implications of these kl s being zero in the case of iwae and aesmc .

Original: In the latter case, we ﬁnd that we can achieve zero kl only if we are able to learn smc intermediate target distributions corresponding to marginals of the target distribution.
Revised: In the latter case, we ﬁnd that we can achieve zero kl only if we are able to learn smc intermediate target distributions corresponding to marginals of the target distribution.

Original: We then built on these results to show that tighter in not necessarily better when it comes to variational bounds and used this insight to develop a new method, alternating elbo s, that address some of the issues of aesmc to improve proposal learning.
Revised: We then built on these results to show that tighter in not necessarily better when it comes to variational bounds and used this insight to develop a new method, alternating elbo s, that address some of the issues of aesmc to improve proposal learning.

Original: A Gradients
Revised: A Gradients

Original: The goal is to obtain an unbiased estimator for the gradient
Revised: The goal is to obtain an unbiased estimator for the gradient

Original: [Equation]
Revised: [Equation]

Original: A. Full Reinforce
Revised: A. Full Reinforce

Original: We express the required quantity as
Revised: We express the required quantity as

Original: [Equation]
Revised: [Equation]

Original: which we can estimate by sampling ( x 1: K 1: T , a 1: K 1: T − 1 ) directly from Q SMC and evaluating h ∇ θ,φ log Q SMC ( x 1: K
Revised: which we can estimate by sampling ( x 1: K 1: T , a 1: K 1: T − 1 ) directly from Q SMC and evaluating h ∇ θ,φ log Q SMC ( x 1: K

Original: [Equation]
Revised: [Equation]

Original: A. Reinforce & Reparameterization
Revised: A. Reinforce & Reparameterization

Original: We express the required quantity as ∇ θ,φ
Revised: We express the required quantity as ∇ θ,φ

Original: [Equation]
Revised: [Equation]

Original: (33) where r ( (cid:15) 1: K 1: T ) denotes a sample with identical distribution as x 1: K 1: T obtained by passing the auxiliary samples (cid:15) 1: K 1: T through the reparameterization function.
Revised: (33) where r ( (cid:15) 1: K 1: T ) denotes a sample with identical distribution as x 1: K 1: T obtained by passing the auxiliary samples (cid:15) 1: K 1: T through the reparameterization function.

Original: We can thus estimate the gradient by sampling (cid:15) 1: K 1: T from the auxiliary distribution, reparameterizing and evaluating h ∇ θ,φ log (cid:16)Q
Revised: We can thus estimate the gradient by sampling (cid:15) 1: K 1: T from the auxiliary distribution, reparameterizing and evaluating h ∇ θ,φ log (cid:16)Q

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Kk =1 Discrete
Revised: Kk =1 Discrete

Original: [Equation]
Revised: [Equation]

Original: a kt
Revised: a kt

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: : K
Revised: : K

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: : K 1: T −
Revised: : K 1: T −

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: φ
Revised: φ

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: (cid:15) 1: K 1: T )
Revised: (cid:15) 1: K 1: T )

Original: [Equation]
Revised: [Equation]

Original: a 1: K 1: T − 1 )
Revised: a 1: K 1: T − 1 )

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: In Figure 5, we demonstrate that the estimator in (33) has much higher variance if we include the ﬁrst term.
Revised: In Figure 5, we demonstrate that the estimator in (33) has much higher variance if we include the ﬁrst term.

Original: [Figure]
Revised: [Figure]

Original: r e i n f o r ce
Revised: r e i n f o r ce

Original: B Proofs for Bias & Implications on the Proposals
Revised: B Proofs for Bias & Implications on the Proposals

Original: Derivation of (9) .
Revised: Derivation of (9) .

Original: [Equation]
Revised: [Equation]

Original: Proof of Claim 1.
Revised: Proof of Claim 1.

Original: Since ˆ Z P ( x ) ≥ 0, Q ( x ) ≥ 0 and R Q ( x ) ˆ Z P ( x ) d x = Z P , we can let the unnormalized target density in Deﬁnition 1 be ˜ P
Revised: Since ˆ Z P ( x ) ≥ 0, Q ( x ) ≥ 0 and R Q ( x ) ˆ Z P ( x ) d x = Z P , we can let the unnormalized target density in Deﬁnition 1 be ˜ P

Original: ( x ) =
Revised: ( x ) =

Original: Q ( x ) ˆ Z P ( x ).
Revised: Q ( x ) ˆ Z P ( x ).

Original: Hence, the normalized target density is P ( x ) =
Revised: Hence, the normalized target density is P ( x ) =

Original: Q ( x ) ˆ Z P
Revised: Q ( x ) ˆ Z P

Original: ( x ) /Z
Revised: ( x ) /Z

Original: P .
Revised: P .

Original: Substituting these quantities into (8) and (9) yields the two equalities in (10).
Revised: Substituting these quantities into (8) and (9) yields the two equalities in (10).

Original: Proof of Proposition 1.
Revised: Proof of Proposition 1.

Original: ( = ⇒ )
Revised: ( = ⇒ )

Original: Substituting for Q IS ( x 1: K )
Revised: Substituting for Q IS ( x 1: K )

Original: =
Revised: =

Original: P IS ( x 1: K ), we obtain
Revised: P IS ( x 1: K ), we obtain

Original: K Y k
Revised: K Y k

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Integrating both sides with respect to ( x 2 , . .
Revised: Integrating both sides with respect to ( x 2 , . .

Original: , x K ) over the whole support (i.e. marginalize out everything except x 1 ), we obtain:
Revised: , x K ) over the whole support (i.e. marginalize out everything except x 1 ), we obtain:

Original: [Equation]
Revised: [Equation]

Original: Rearranging gives us q ( x 1 | y )
Revised: Rearranging gives us q ( x 1 | y )

Original: =
Revised: =

Original: p ( x 1 | y ) for all x 1 .
Revised: p ( x 1 | y ) for all x 1 .

Original: ( ⇐ = )
Revised: ( ⇐ = )

Original: Substituting p ( x k | y )
Revised: Substituting p ( x k | y )

Original: =
Revised: =

Original: q ( x k | y ), we obtain
Revised: q ( x k | y ), we obtain

Original: [Equation]
Revised: [Equation]

Original: Proof of Proposition 2.
Revised: Proof of Proposition 2.

Original: We consider the general sequence of target distributions π t ( x 1: t ) ( p θ ( x 1: t | y 1: t ) in the case of ssm s), their unnormalized versions γ t ( x 1: t ) ( p θ ( x 1: t , y 1: t ) in the case of ssm s), their normalizing constants Z t =
Revised: We consider the general sequence of target distributions π t ( x 1: t ) ( p θ ( x 1: t | y 1: t ) in the case of ssm s), their unnormalized versions γ t ( x 1: t ) ( p θ ( x 1: t , y 1: t ) in the case of ssm s), their normalizing constants Z t =

Original: R γ t ( x 1: t ) d x 1: t ( p θ ( y 1: t ) in the case of ssm s), where Z = Z T = p ( y 1: T ).
Revised: R γ t ( x 1: t ) d x 1: t ( p θ ( y 1: t ) in the case of ssm s), where Z = Z T = p ( y 1: T ).

Original: ( = ⇒ )
Revised: ( = ⇒ )

Original: It suﬃces to show that ˆ Z SMC ( x 1: K 1: T , a 1: K 1: T − 1 )
Revised: It suﬃces to show that ˆ Z SMC ( x 1: K 1: T , a 1: K 1: T − 1 )

Original: =
Revised: =

Original: Z for all ( x 1: K 1: T , a 1: K 1: T − 1 ) implies 1 andin Proposal 2 due to equation (11).
Revised: Z for all ( x 1: K 1: T , a 1: K 1: T − 1 ) implies 1 andin Proposal 2 due to equation (11).

Original: We ﬁrst prove that ˆ Z SMC ( x 1: K 1: T , a 1: K 1: T − 1 )
Revised: We ﬁrst prove that ˆ Z SMC ( x 1: K 1: T , a 1: K 1: T − 1 )

Original: =
Revised: =

Original: Z for all ( x 1: K 1: T , a 1: K 1: T − 1 ) implies that the weights
Revised: Z for all ( x 1: K 1: T , a 1: K 1: T − 1 ) implies that the weights

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for t = 2 , . . .
Revised: for t = 2 , . . .

Original: , T
Revised: , T

Original: [Equation]
Revised: [Equation]

Original: 43) are constant with respect to x 1: t .
Revised: 43) are constant with respect to x 1: t .

Original: Pick t ∈ { 1 , . .
Revised: Pick t ∈ { 1 , . .

Original: , T } and distinct k, ‘ ∈ { 1 , . . .
Revised: , T } and distinct k, ‘ ∈ { 1 , . . .

Original: , K } .
Revised: , K } .

Original: Also, pick x 1: t and x 0 1: t .
Revised: Also, pick x 1: t and x 0 1: t .

Original: Now, consider two sets of particle sets (¯ x 1: K 1: T , ¯ a 1: K 1: T − 1 ) and (˜ x 1: K 1: T , ˜ a 1: K 1: T − 1 ), illustrated in Figure 6, such that ¯ x κτ =   x 0 τ if κ = ‘ and τ < t x 0 τ if ( κ, τ ) =
Revised: Now, consider two sets of particle sets (¯ x 1: K 1: T , ¯ a 1: K 1: T − 1 ) and (˜ x 1: K 1: T , ˜ a 1: K 1: T − 1 ), illustrated in Figure 6, such that ¯ x κτ =   x 0 τ if κ = ‘ and τ < t x 0 τ if ( κ, τ ) =

Original: ( k, t )
Revised: ( k, t )

Original: x τ if κ = k
Revised: x τ if κ = k

Original: and τ < t x κτ otherwise for τ = 1 , . . .
Revised: and τ < t x κτ otherwise for τ = 1 , . . .

Original: , T, κ = 1 , . . .
Revised: , T, κ = 1 , . . .

Original: , K, (44) ¯ a κτ = (cid:26) ‘ if ( κ, τ ) =
Revised: , K, (44) ¯ a κτ = (cid:26) ‘ if ( κ, τ ) =

Original: ( k, t − 1) or ( k, t ) κ otherwise for τ = 1 , . .
Revised: ( k, t − 1) or ( k, t ) κ otherwise for τ = 1 , . .

Original: , T − 1 , κ = 1 , . .
Revised: , T − 1 , κ = 1 , . .

Original: , K, (45) ˜ x κτ =   x 0 τ if κ = ‘ and τ < t x τ if ( κ, τ ) =
Revised: , K, (45) ˜ x κτ =   x 0 τ if κ = ‘ and τ < t x τ if ( κ, τ ) =

Original: ( k, t )
Revised: ( k, t )

Original: x τ if κ = k
Revised: x τ if κ = k

Original: and τ < t x κτ otherwise for τ = 1 , . . .
Revised: and τ < t x κτ otherwise for τ = 1 , . . .

Original: , T, κ = 1 , . . .
Revised: , T, κ = 1 , . . .

Original: , K, (46) ˜ a κτ = (cid:26) ‘ if ( κ, τ ) =
Revised: , K, (46) ˜ a κτ = (cid:26) ‘ if ( κ, τ ) =

Original: ( k, t ) κ otherwise for τ = 1 , . .
Revised: ( k, t ) κ otherwise for τ = 1 , . .

Original: , T − 1 , κ = 1 , . . .
Revised: , T − 1 , κ = 1 , . . .

Original: , K.
Revised: , K.

Original: (47) x (cid:48)··· ··· ··· ··· (cid:96) k x (cid:48)x (cid:48)
Revised: (47) x (cid:48)··· ··· ··· ··· (cid:96) k x (cid:48)x (cid:48)

Original: t −xxx t −...
Revised: t −xxx t −...

Original: x (cid:48) t ... x (cid:48)··· ··· ··· ··· (cid:96) k x (cid:48)x (cid:48)
Revised: x (cid:48) t ... x (cid:48)··· ··· ··· ··· (cid:96) k x (cid:48)x (cid:48)

Original: t −xxx t −... x t ...
Revised: t −xxx t −... x t ...

Original: The weights ¯ w κτ and ˜ w κτ for the respective particle sets are identical except when ( τ, κ )
Revised: The weights ¯ w κτ and ˜ w κτ for the respective particle sets are identical except when ( τ, κ )

Original: = ( t, k ) where
Revised: = ( t, k ) where

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since ˆ Z (¯ x 1: K 1: T , ¯ a 1: K 1: T − 1 )
Revised: Since ˆ Z (¯ x 1: K 1: T , ¯ a 1: K 1: T − 1 )

Original: =
Revised: =

Original: ˆ Z (˜ x 1: K 1: T , ˜ a 1: K 1: T − 1 ), we have w t ( x 0 1: t ) =
Revised: ˆ Z (˜ x 1: K 1: T , ˜ a 1: K 1: T − 1 ), we have w t ( x 0 1: t ) =

Original: w t ( x 1: t ).
Revised: w t ( x 1: t ).

Original: We have proven that w t ( x 1: t ) is constant with respect to x 1: t for all t = 1 , . . .
Revised: We have proven that w t ( x 1: t ) is constant with respect to x 1: t for all t = 1 , . . .

Original: , T .
Revised: , T .

Original: Now, for x 1: t , consider the proposal by rearranging (42) and (43)
Revised: Now, for x 1: t , consider the proposal by rearranging (42) and (43)

Original: [Equation]
Revised: [Equation]

Original: where w t := w t ( x 1: t ) is constant.
Revised: where w t := w t ( x 1: t ) is constant.

Original: For this to be a normalized density with respect to x t , we must have
Revised: For this to be a normalized density with respect to x t , we must have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and for t = 2 , . . .
Revised: and for t = 2 , . . .

Original: , T :
Revised: , T :

Original: [Equation]
Revised: [Equation]

Original: Since R π t +1 ( x 1: t +1 )
Revised: Since R π t +1 ( x 1: t +1 )

Original: d x t +1 and π t ( x 1: t ) are both normalized densities
Revised: d x t +1 and π t ( x 1: t ) are both normalized densities

Original: [Equation]
Revised: [Equation]

Original: we must have π t ( x 1: t ) =
Revised: we must have π t ( x 1: t ) =

Original: R π t +1 ( x 1: t +1 )
Revised: R π t +1 ( x 1: t +1 )

Original: d x t +1 for all t = 1 , . . .
Revised: d x t +1 for all t = 1 , . . .

Original: , T − 1 for all x 1: t .
Revised: , T − 1 for all x 1: t .

Original: For a given t ∈ { 1 , . . .
Revised: For a given t ∈ { 1 , . . .

Original: , T − 1 } and x 1: t , applying this repeatedly yields
Revised: , T − 1 } and x 1: t , applying this repeatedly yields

Original: [Equation]
Revised: [Equation]

Original: We also have
Revised: We also have

Original: [Equation]
Revised: [Equation]

Original: ( ⇐ = )
Revised: ( ⇐ = )

Original: We substitute identities in 1 and 2 of Proposal 2 back to the expression of ˆ Z ( x 1: K 1: T , a 1: K 1: T − 1 ) to obtain ˆ Z ( x 1: K 1: T , a 1: K 1: T − 1 )
Revised: We substitute identities in 1 and 2 of Proposal 2 back to the expression of ˆ Z ( x 1: K 1: T , a 1: K 1: T − 1 ) to obtain ˆ Z ( x 1: K 1: T , a 1: K 1: T − 1 )

Original: =
Revised: =

Original: Z .
Revised: Z .

Original: C VRNN
Revised: C VRNN

Original: In the following we give the details of our vrnn architecture.
Revised: In the following we give the details of our vrnn architecture.

Original: The generative model is given by:
Revised: The generative model is given by:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: and the proposal distribution is given by
Revised: and the proposal distribution is given by

Original: [Equation]
Revised: [Equation]

Original: The functions µ xθ and σ xθ are computed by networks with two fully connected layers of size 128 whose ﬁrst layer is shared.
Revised: The functions µ xθ and σ xθ are computed by networks with two fully connected layers of size 128 whose ﬁrst layer is shared.

Original: ϕ
Revised: ϕ

Original: xθ is one fully connected layer of size 128.
Revised: xθ is one fully connected layer of size 128.

Original: For visual input, the encoding ϕ yθ is a convolutional network with conv-4x4-2-1-32, conv4x4-2-1-64, conv-4x4-2-1-128 where conv-wxh-s-p-n denotes a convolutional network with n ﬁlters of size w × h , stride s , padding p .
Revised: For visual input, the encoding ϕ yθ is a convolutional network with conv-4x4-2-1-32, conv4x4-2-1-64, conv-4x4-2-1-128 where conv-wxh-s-p-n denotes a convolutional network with n ﬁlters of size w × h , stride s , padding p .

Original: Between convolutions we use leaky ReLUs with slope 0.2 as nonlinearity and batch norms.
Revised: Between convolutions we use leaky ReLUs with slope 0.2 as nonlinearity and batch norms.

Original: The decoding µ yθ uses transposed convolutions of the same dimensions but in reversed order, however with stride s = 1 and padding p = 0 for the ﬁrst layer.
Revised: The decoding µ yθ uses transposed convolutions of the same dimensions but in reversed order, however with stride s = 1 and padding p = 0 for the ﬁrst layer.

Original: A Gated Recurrent Unit (GRU) is used as RNN and if not stated otherwise ReLUs are used in between fully connected layers.
Revised: A Gated Recurrent Unit (GRU) is used as RNN and if not stated otherwise ReLUs are used in between fully connected layers.

Original: For the proposal distribution, the functions µ pφ and σ pφ are neural networks with three fully connected layers of size 128 that are sharing the ﬁrst two layers.
Revised: For the proposal distribution, the functions µ pφ and σ pφ are neural networks with three fully connected layers of size 128 that are sharing the ﬁrst two layers.

Original: Sigmoid and softplus functions are used where values in (0 , 1) or R + are required.
Revised: Sigmoid and softplus functions are used where values in (0 , 1) or R + are required.

Original: We use a minibatch size of 40.
Revised: We use a minibatch size of 40.

Original: For the moving agents dataset we use sga with a learning rate of 1 e − 5.
Revised: For the moving agents dataset we use sga with a learning rate of 1 e − 5.

Original: A speciﬁc feature of the vrnn architecture is that the proposal and the generative model share the component ϕ
Revised: A speciﬁc feature of the vrnn architecture is that the proposal and the generative model share the component ϕ

Original: yφ,θ .
Revised: yφ,θ .

Original: Consequently, we set φ = θ for the parameters belonging to this module and train it using gradients for both θ and φ .
Revised: Consequently, we set φ = θ for the parameters belonging to this module and train it using gradients for both θ and φ .

Original: D Experiments D. Moving Agents
Revised: D Experiments D. Moving Agents

Original: In ﬁgure 7 we investigate the quality of the generative model by comparing visual predictions.
Revised: In ﬁgure 7 we investigate the quality of the generative model by comparing visual predictions.

Original: We do so for models learned by iwae (top) and aesmc (bottom) .
Revised: We do so for models learned by iwae (top) and aesmc (bottom) .

Original: The models were learned using ten particles but for easier visualization we only predict using ﬁve particles.
Revised: The models were learned using ten particles but for easier visualization we only predict using ﬁve particles.

Original: The ﬁrst row in each graphic shows the ground truth.
Revised: The ﬁrst row in each graphic shows the ground truth.

Original: The second row shows the averaged predictions of all ﬁve particles.
Revised: The second row shows the averaged predictions of all ﬁve particles.

Original: The next ﬁve rows show the predictions made by each particle individually.
Revised: The next ﬁve rows show the predictions made by each particle individually.

Original: The observations (i.e. the top row) up to t = 19 are shown to the model.
Revised: The observations (i.e. the top row) up to t = 19 are shown to the model.

Original: Up to this timestep the latent values x 0:19 are drawn from the proposal distribution q ( x t | y t , h t − 1 ).
Revised: Up to this timestep the latent values x 0:19 are drawn from the proposal distribution q ( x t | y t , h t − 1 ).

Original: From t = 20 onwards the latent values x 20:37 are drawn from the generative model p ( x t | x t − 1 ).
Revised: From t = 20 onwards the latent values x 20:37 are drawn from the generative model p ( x t | x t − 1 ).

Original: Consequently, the model predicts the partially occluded, stochastic movement overtimesteps into the future.
Revised: Consequently, the model predicts the partially occluded, stochastic movement overtimesteps into the future.

Original: We note that most particles predict a viable future trajectory.
Revised: We note that most particles predict a viable future trajectory.

Original: However, the model learned by iwae is not as consistent in the quality of its predictions, often ’forgetting’ the particle.
Revised: However, the model learned by iwae is not as consistent in the quality of its predictions, often ’forgetting’ the particle.

Original: This does not happen in every predicted sequence but the behavior shown here is very typical.
Revised: This does not happen in every predicted sequence but the behavior shown here is very typical.

Original: Models learned by aesmc are much more consistent in the quality of their predictions.
Revised: Models learned by aesmc are much more consistent in the quality of their predictions.

Revised: E Additional Experiments E. Optimizing Only Proposal Parameters

Revised: We have run experiments where we optimize various elbo objectives with respect to φ with θ ﬁxed in order to see how various objectives have an eﬀect on proposal learning.

Revised: In particular, we train elbo IS and elbo SMC with number of particles K ∈ { 10 , 100 , 1000 } .

Revised: Once the training is done, we use the trained proposal network to perform inference using both is and smc with number of particles K test ∈ { 10 , 100 , 1000 } .

Revised: In Figure 8, we see experimental results for the lgssm described in Section 5.1.

Revised: We measure the quality of the inference network using a proxy qP

Revised: Tt =1 ( µ kalman t − µ approx t ) 2 where µ kalman t is the true marginal mean E p ( x 1: T | y 1: T ) [ x t ] obtained from the Kalman smoothing algorithm and µ approx t = (cid:16)

Revised: P

Revised: Kk =1 w kT x t (cid:17) / (cid:16) P

Revised: Kk =1 w kT (cid:17) is an approximate marginal mean obtained from the proposal parameterized by φ .

Revised: We see that if we train using elbo SMC with K train = 1000, the performance for inference using smc (with whichever K test ∈ { 10 , 100 , 1000 } ) is worse than if we train with elbo IS with any number of particles K train ∈ { 10 , 100 , 1000 } .

Revised: Examining the other axes of variation:

Revised: [Figure]

Revised: • Increasing K test (moving up in Figure 8 (Right)) improves inference.

Revised: • Increasing K train (moving to the right in Figure 8 (Right))

Revised: worsens inference.

Revised: • Among diﬀerent possible combinations of (training algorithm, testing algorithm), ( is , smc ) (cid:31) ( smc , smc ) (cid:31) ( is , is ) (cid:31) ( smc , is ), where we use “ a (cid:31) b ” to denote that the combination a results in better inference than combination b .