Original: Nested Variational Inference
Revised: Nested Variational Inference

Original: Anonymous Authors

Original: Anonymous Institution

Original: Abstract
Revised: Abstract

Original: We develop Nested Variational Inference (NVI), a family of methods that learn proposals for nested importance samplers by minimizing an inclusive or exclusive KL divergence at each level of nesting.
Revised: We develop nested variational inference (NVI), a family of methods that learn proposals for nested importance samplers by minimizing an inclusive or exclusive KL divergence at each level of nesting.

Original: This formulation enables us to learn proposals for a general class of samplers that admits most commonly used importance sampling strategies as special cases. NVI additionally provides a mechanism for learning intermediate densities, which can serve as heuristics to guide the sampler.
Revised: NVI is applicable to many commonly-used importance sampling strategies and additionally provides a mechanism for learning intermediate densities, which can serve as heuristics to guide the sampler.

Original: Our experiments apply NVI to learn samplers targeting (a) an unnormalized density using annealing and (b) the posterior of a hidden Markov model.
Revised: Our experiments apply NVI to learn samplers targeting (a) an unnormalized density using annealing and (b) the posterior of a hidden Markov model.

Original: We observe improved sample quality in terms of log average weight and eﬀective sample size.
Revised: We observe improved sample quality in terms of log average weight and eﬀective sample size.

Original: 1. Introduction
Revised: 1. Introduction

Original: Deep generative models provide a mechanism for incorporating priors into methods for unsupervised representation learning.
Revised: Deep generative models provide a mechanism for incorporating priors into methods for unsupervised representation learning.

Original: This is particularly useful in settings where the prior deﬁnes a meaningful inductive bias that reﬂects a known structure of the underlying domain.
Revised: This is particularly useful in settings where the prior deﬁnes a meaningful inductive bias that reﬂects a known structure of the underlying domain.

Original: Training models with structured priors, however, poses some challenges.
Revised: Training models with structured priors, however, poses some challenges.

Original: A standard strategy for training these models is to maximize a reparameterized variational lower bound, which serves to jointly train the generative model and an inference model that approximates its posterior (Kingma and Welling, 2013; Rezende et al., 2014).
Revised: A standard strategy for training these models is to maximize a reparameterized variational lower bound with respect to a generative model and an inference model that approximates its posterior (Kingma and Welling, 2013; Rezende et al., 2014).

Original: This approach works well in variational autoencoders (VAEs) with isotropic Gaussian priors, but often fails in models with high-dimensional and correlated latent variables.
Revised: This approach works well in variational autoencoders (VAEs) with isotropic Gaussian priors but often fails in models with highdimensional and correlated latent variables.

Original: In recent years, a range of strategies for improving upon standard reparameterized variational inference have been put forward.
Revised: In recent years a range of strategies for improving upon standard reparameterized variational inference have been put forward.

Original: These include wake-sleep style variational methods that minimize the inclusive KL-divergence (Bornschein and Bengio, 2014; Le et al., 2019), as well as sampling schemes that incorporate annealing (Huang et al., 2018), Sequential Monte
Revised: These include wake-sleep style variational methods that minimize the inclusive KL-divergence (Bornschein and Bengio, 2014; Le et al., 2019), as well as sampling schemes that incorporate annealing (Huang et al., 2018), Sequential Monte

Original: Carlo (Le et al., 2017; Naesseth et al., 2017; Maddison et al., 2017), Gibbs sampling (Wu et al., 2019; Wang et al., 2018), and MCMC updates (Salimans et al., 2015; Hoﬀman, 2017;
Revised: Carlo (Le et al., 2017; Naesseth et al., 2017; Maddison et al., 2017), Gibbs sampling (Wu et al., 2019; Wang et al., 2018), and MCMC updates (Salimans et al., 2015; Hoﬀman, 2017;

Original: Li et al., 2017).
Revised: Li et al., 2017).

Original: While these methods oﬀer ﬂexible inference, typically resulting in better approximations to the posterior compared to traditional Variational Inference methods, they are also model-speciﬁc, requiring specialized sampling schemes and gradient estimators, and can not be easily composed with other techniques.
Revised: While these methods oﬀer ﬂexible inference, typically resulting in better approximations to the posterior compared to traditional variational inference methods, they are also model-speciﬁc, requiring specialized sampling schemes and gradient estimators, and can not be easily composed with other techniques.

Original: In this paper, we propose Nested Variational Inference (NVI), a framework for combining nested importance sampling and variational inference.
Revised: In this paper, we propose nested variational inference (NVI), a framework for combining nested importance sampling and variational inference.

Original: Nested importance sampling formalize the construction of proposals by way of nested calls to other importance samplers (Naesseth et al., 2015, 2019), and hereby deﬁne a general class of importance sampling strategies that admits methods based on annealing (Neal, 2001) and sequential Monte Carlo (Del Moral et al., 2006) as special cases.
Revised: Nested importance sampling formalizes the construction of proposals by way of nested calls to other importance samplers (Naesseth et al., 2015, 2019), and admits many existing importance sampling strategies as special cases, including methods based on annealing (Neal, 2001) and sequential Monte Carlo (Del Moral et al., 2006).

Original: NVI learns proposals by optimizing an inclusive or exclusive KL divergence at each level of nesting.
Revised: NVI learns proposals by optimizing an inclusive or exclusive KL divergence at each level of nesting.

Original: Combining nested variational objectives with importance resampling allows us to compute gradient estimates based on incremental weights, which depend only on variables that are sampled locally, rather than on all variables in the model.
Revised: Combining nested variational objectives with importance resampling allows us to compute gradient estimates based on incremental weights, which depend only on variables that are sampled locally, rather than on all variables in the model.

Original: Doing so yields lower variance weights, whilst maintaining a high sample diversity relative to existing variational methods based on sequential Monte Carlo.
Revised: Doing so yields lower variance weights, whilst maintaining a high sample diversity relative to existing variational methods based on sequential Monte Carlo.

Original: NVI extends beyond existing methods in that it deﬁnes objectives for learning intermediate densities in addition to proposals.
Revised: NVI extends beyond existing methods in that it deﬁnes objectives for learning intermediate densities in addition to proposals.

Original: In a nested importance sampler, the target density at each level of nesting deﬁnes the proposal at the next level of nesting.
Revised: In a nested importance sampler, the target density at each level of nesting deﬁnes the proposal at the next level of nesting.

Original: These learned intermediate densities can serve as heuristics that guide the sampler.
Revised: These learned intermediate densities can serve as heuristics that guide the sampler.

Original: To demonstrate the potential of this approach.
Revised: To demonstrate the potential of this approach.

Original: We compare NVI to existing techniques in two settings.
Revised: We compare NVI to existing techniques in two settings.

Original: The ﬁrst is an annealed sampling task, where we use NVI to learn proposals and optimize the annealing schedule which speciﬁes the intermediate densities.
Revised: The ﬁrst is an annealed sampling task, where we use NVI to learn proposals and optimize the annealing schedule which speciﬁes the intermediate densities.

Original: The second task is performing posterior inference on a Hidden Markov model, in which NVI is used to learn proposals and a heuristic factor that incorporates future observations.
Revised: The second task is performing posterior inference on a Hidden Markov model, in which NVI is used to learn proposals and a heuristic factor that incorporates future observations.

Original: In both cases,
Revised: In both cases,

Original: NVI results in substantial improvements to sample quality.
Revised: NVI results in substantial improvements to sample quality.

Original: 2. Nested Variational Inference
Revised: 2. Nested Variational Inference

Original: Problem setting.
Revised: Problem setting.

Original: Nested Variational Inference makes use of nested importance samplers (Naesseth et al., 2015, 2019), which provide a means of reasoning about methods that recursively use importance samplers to generate proposals.
Revised: Nested variational inference makes use of nested importance samplers (Naesseth et al., 2015, 2019), which provide a means of reasoning about methods that recursively use importance samplers to generate proposals.

Original: Here, we will consider a sequence of intractable densities { π k ( z k ; θ k ) }
Revised: Here, we will consider a sequence of intractable densities { π k ( z k ; θ k ) }

Original: Kk =1 and corresponding unnormalized densities { γ k ( z k ; θ k ) }
Revised: Kk =1 and corresponding unnormalized densities { γ k ( z k ; θ k ) }

Original: Kk =1 with intractible normalizing constant such that
Revised: Kk =1 with intractible normalizing constant such that

Original: [Equation]
Revised: [Equation]

Original:  We are typically interested in the case where the ﬁnal density γ K ( z ; θ ) corresponds to the posterior distribution p θ ( z | x ).
Revised: dz k [Equation] We are typically interested in the case where the ﬁnal density γ K ( z ; θ ) corresponds to the posterior distribution p θ ( z | x ).

Original: To simplify notation, we will in the following omit parameters when they are not needed for context (i.e. we write γ ( z ) instead of γ ( z ; θ )).
Revised: To simplify notation, we will in the following omit parameters when they are not needed for context (i.e. we write γ ( z ) instead of γ ( z ; θ )).

Original: We deﬁne the sequence of intermediate densities to interpolate between a density π 1 ( z 1 ), for which sampling is easy, to the ﬁnal density π K ( z K ) for which sampling is diﬃcult.
Revised: We deﬁne the sequence of intermediate densities to interpolate between a density π 1 ( z 1 ), for which sampling is easy, to the ﬁnal density π K ( z K ) for which sampling is diﬃcult.

Original: Two standard strategies are to deﬁne variables z k ∈ Z k on (1) a ﬁxed sample space Z 1 = · · · = Z K with increasingly tightly-peaked densities, e.g. when performing annealing, or(2) on sample spaces with increasing dimensionality Z 1 = Z (cid:48) 1 , Z [Equation]
Revised: Two standard strategies are to deﬁne variables z k ∈ Z k on (1) a ﬁxed sample space Z 1 = · · · = Z K with increasingly tightly-peaked densities, e.g. when performing annealing, or

Original: = Z[Equation] 1 × Z (cid:48)2 , . . .
Revised: (2) on sample spaces with increasing dimensionality Z 1 = Z (cid:48)

Original: [Equation]
Revised: [Equation]

Original: =

Original: [Equation]
Revised: [Equation]

Original: (cid:48)
Revised: (cid:48)

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ×
Revised: ×

Original: Z (cid:48) k , which is common when sampling from state-space models.
Revised: Z [Equation] k , which is common when sampling from state-space models.

Original: In the ﬁrst case, we introduce a forward kernel q k ( z k | z k − 1 ; ˆ φ k ), and a reverse kernel r k ( z k − 1 | z k ; ˇ φ k ).
Revised: In the ﬁrst case, we introduce a forward kernel q k ( z k | z k − 1 ; ˆ φ k ), and a reverse kernel r k ( z k − 1 | z k ; ˇ φ k ).

Original: This yields a forward and reverse density on the extended space
Revised: This yields a forward and reverse density on the extended space

Original: Z k ×Z k − 1 , ˆ γ k
Revised: Z k ×Z k − 1 , ˆ γ k

Original: ( z k , z k − 1 ) =
Revised: ( z k , z k − 1 ) =

Original: q k ( z k | z k − 1 ) γ k − 1 ( z k − 1 ) , 
Revised: q k ( z k | z k − 1 ) γ k − 1 ( z k − 1 ) , ˇ γ k ( z k , z k − 1 )

Original: [Equation]
Revised: = γ k ( z k ) r k ( z k − 1 | z k ) .

Original: In the latter case, we can omit the construction of a reverse kernel as the reverse density at every step is fully speciﬁed by the corresponding intermediate density on Z (cid:48)
Revised: In the latter case, we can omit the construction of a reverse kernel as the reverse density at every step is fully speciﬁed by the corresponding intermediate density on Z (cid:48)

Original: k
Revised: k

Original: × Z k −ˆ γ k ( z 1: k ) =
Revised: × Z k −ˆ γ k ( z 1: k ) =

Original: q k ( z (cid:48) k
Revised: q k ( z (cid:48) k

Original: | z k − 1 ) γ k ( z k − 1 ) =
Revised: | z k − 1 ) γ k ( z k − 1 ) =

Original: q k ( z (cid:48) k
Revised: q k ( z (cid:48) k

Original: | z (cid:48) 1: k − 1 ) γ k
Revised: | z (cid:48) 1: k [Equation]

Original: ( z (cid:48) 1: k − 1 ) , ˇ γ k ( z k ) =
Revised: [Equation]

Original: γ k ( z k ) =
Revised: k [Equation]

Original: γ k ( z (cid:48) 1: k ) .
Revised: 1: k [Equation] [Equation]

Original: Nested Importance Samplers.
Revised: Nested Importance Samplers.

Original: Nested importance samplers (Naesseth et al., 2015, 2019) deﬁne a sampling problem recursively by generalizing from standard sequential importance sampling (SIS) methods.
Revised: Nested importance samplers (Naesseth et al., 2015, 2019) deﬁne a sampling problem recursively by generalizing from standard sequential importance sampling (SIS) methods.

Original: Suppose that we have a mechanism by which we can generate weighted samples ( w k − 1 , z k − 1 ) from the target density π k − 1 ( z k − 1 ).
Revised: Suppose that we have a mechanism by which we can generate weighted samples ( w k − 1 , z k − 1 ) from the target density π k − 1 ( z k − 1 ).

Original: We can construct samples ( w k , z k ) that target the next density by the following construction,
Revised: We can construct samples ( w k , z k ) that target the next density by the following construction,

Original: [Equation]
Revised: [Equation]

Original: where ˜ v k denotes the incremental weight computed w.r.t.
Revised: where ˜ v k denotes the incremental weight computed w.r.t.

Original: the normalized densities.
Revised: the normalized densities.

Original: This notion of compositionality can be formalized by introducing the concept of proper weighting.
Revised: This notion of compositionality can be formalized by introducing the concept of proper weighting.

Original: Deﬁnition 1 (Proper weighting)
Revised: Deﬁnition 1 (Proper weighting)

Original: Let π be a probability density.
Revised: Let π be a probability density.

Original: A random pair ( w, z ) ∼ Π is properly weighted (p.w.) for an unnormalized probability density
Revised: A random pair ( w, z ) ∼ Π is properly weighted (p.w.) for an unnormalized probability density

Original: γ ≡ Zπ if w ≥ 0 and for all measurable functions g it holds that
Revised: γ ≡ Zπ if w ≥ 0 and for all measurable functions g it holds that

Original: [Equation]
Revised: [Equation]

Original: for some constant c > 0 .
Revised: for some constant c > 0 .

Original: Hence, given properly weighted samples ( z l , w l ) ∼ Π this ensures that the we can compute strongly consistent self-normalized estimates
Revised: Hence, given properly weighted samples ( z l , w l ) ∼ Π this ensures that the we can compute strongly consistent self-normalized estimates

Original: [Equation]
Revised: [Equation]

Original: In other words, proper weighting ensures that the bias of our self-normalized estimators vanishes in the limit of inﬁnite samples.
Revised: In other words, proper weighting ensures that the bias of our self-normalized estimators vanishes in the limit of inﬁnite samples.

Original: Compositionality of NVI steps.
Revised: Compositionality of Proper Weighting.

Original: The proper weighting property allows us to reason about the validity of compositions of sampling operations in a straight-forward manner.
Revised: The proper weighting property allows us to reason about the validity of compositions of sampling operations in a straight-forward manner.

Original: If we can show that individual operations preserve proper weighting, then any composition of these operations also preserves proper weighting.
Revised: If we can show that individual operations preserve proper weighting, then any composition of these operations also preserves proper weighting.

Original: Importance sampling and hence the sequential construction in Equation 1 preserves proper weighting, which is to say that if ( w k − 1 , z k − 1 ) is properly weighted w.r.t.
Revised: Importance sampling and hence the sequential construction in Equation 1 preserves proper weighting, which is to say that if ( w k − 1 , z k − 1 ) is properly weighted w.r.t.

Original: γ k − 1 ( z k − 1 ), then ( w k , z k ) is properly weighted w.r.t.
Revised: γ k − 1 ( z k − 1 ), then ( w k , z k ) is properly weighted w.r.t.

Original: γ k ( z k ).
Revised: γ k ( z k ).

Original: Other operations that preserve proper weighting are rejection sampling, the application of an MCMC transition operator, and most notably importance resampling, which forms the basis for sequential Monte Carlo methods.
Revised: Other operations that preserve proper weighting are rejection sampling, the application of an MCMC transition operator, and most notably importance resampling, which forms the basis for sequential Monte Carlo methods.

Original: Composition of these operations leads to a broad class of veriﬁably correct importance samplers that admit many existing methods as special cases.
Revised: Composition of these operations leads to a broad class of veriﬁably correct importance samplers that admit many existing methods as special cases.

Original: Nested Variational Objectives.
Revised: Nested Variational Objectives.

Original: We are interested in deﬁning variational objectives that can be used to optimize the parameters of nested importance samplers.
Revised: We are interested in deﬁning variational objectives that can be used to optimize the parameters of nested importance samplers.

Original: As before, we will for purposes of exposition restrict ourselves to samplers that follow the sequential construction in Equation 1.
Revised: As before, we will for purposes of exposition restrict ourselves to samplers that follow the sequential construction in Equation 1.

Original: Given an initial target density γ 1 ( z 1 ) and initial proposal q 1 ( z 1 ), we deﬁne objectives which minimizes an f -divergence based term at each level of nesting
Revised: Given an initial target density γ 1 ( z 1 ) and initial proposal q 1 ( z 1 ), we deﬁne objectives which minimizes an f -divergence based term at each level of nesting

Original: [Equation]
Revised: [Equation]

Original: An f -divergence D f
Revised: An f -divergence D f

Original: ( p || q ) =
Revised: ( p || q ) =

Original: E z ∼ q [ f ( p ( z ) /q
Revised: E z ∼ q [ f ( p ( z ) /q

Original: ( z ))] is parameterized by a convex function f
Revised: ( z ))] is parameterized by a convex function f

Original: that satisﬁes f (1) = 0.
Revised: that satisﬁes f (1) = 0.

Original: When f exc ( w ) :
Revised: When f exc ( w ) :

Original: = f ( w ) = − log( w ) we recover the exclusive KL divergence, whereas f inc ( w ) :
Revised: = f ( w ) = − log( w ) we recover the exclusive KL divergence, whereas f inc ( w ) :

Original: = f ( w ) = w log( w ) recovers the inclusive KL divergence.
Revised: = f ( w ) = w log( w ) recovers the inclusive KL divergence.

Original: We can write each f -divergence in terms of the incremental weight v k
Revised: We can write each f -divergence in terms of the incremental weight v k

Original: [Equation]
Revised: [Equation]

Original: In the following we assume that all parameters θ K of the ﬁnal target density are known or estimated by way of maximum likelihood estimation.
Revised: In the following we assume that all parameters θ K of the ﬁnal target density are known or estimated by way of maximum likelihood estimation.

Original: Our goal is to minimize D with respect to parameters { θ k } K − 1 k =1 of the intermediate densities and parameters { ˆ φ k } Kk =2 and { ˇ φ k } Kk =2 of the forward and reverse kernels at each level of nesting.
Revised: Our goal is to minimize D with respect to parameters { θ k } K − 1 k =1 of the intermediate densities and parameters { ˆ φ k } Kk =2 and { ˇ φ k } Kk =2 of the forward and reverse kernels at each level of nesting.

Original: Intuitively, placing consecutive intermediate densities closer to each other should result in an easier learning problem for the corresponding forward and reverse kernels, while bringing the forward and reverse densities closer should result in a easier sampling problem, e.g optimizing a Pearson χ 2 -divergence can be shown to minimize the variance of the importance weight (M¨uller et al., 2019).
Revised: Intuitively, placing consecutive intermediate densities closer to each other should result in an easier learning problem for the corresponding forward and reverse kernels, while bringing the forward and reverse densities closer should result in a easier sampling problem, e.g optimizing a Pearson χ 2 -divergence can be shown to minimize the variance of the importance weight (M¨uller et al., 2019).

Original: Here we consider KL-divergences only but still take motivation from this intuition.
Revised: Here we consider KL-divergences only but still take motivation from this intuition.

Original: To the best of our knowledge, optimizing a sequence of divergences combined with the ability to learn the parameters of the intermediate densities is novel to NVI.
Revised: To the best of our knowledge, optimizing a sequence of divergences combined with the ability to learn the parameters of the intermediate densities is novel to NVI.

Original: Gradient Computation
Revised: Gradient Computation

Original: Building on the NIS framework described above we are able to compute consistent self-normalized gradient estimators as shown in Equation 2.
Revised: Building on the NIS framework described above, we are able to compute consistent self-normalized gradient estimators as shown in Equation 2.

Original: While the computation of gradient estimates for the parameters ˇ φ k of the reverse kernel and reparameterized gradients estimates for parameters ˆ φ k of the forward kernel is straightforward, estimating the gradients w.r.t.
Revised: While the computation of gradient estimates for the parameters ˇ φ k of the reverse kernel and reparameterized gradients estimates for parameters ˆ φ k of the forward kernel is straightforward, estimating the gradients w.r.t.

Original: parameters of the intermediate densities θ k − 1 and θ k is less convenient.
Revised: parameters of the intermediate densities θ k − 1 and θ k is less convenient.

Original: E.g. in case of the exclusive KL-divergence, the gradient w.r.t.
Revised: E.g. in case of the exclusive KL-divergence, the gradient w.r.t.

Original: θ k − 1 is computed using a score function estimator, which uses an additional baseline, and both, the estimators for θ k − 1 and θ k require to estimate the gradient of the respective log normalizing constants.
Revised: θ k − 1 is computed using a score function estimator, which uses an additional baseline, and both, the estimators for θ k − 1 and θ k require to estimate the gradient of the respective log normalizing constants.

Original: Detailed deviation of the gradient estimators can be found in Appendix D.
Revised: Detailed deviation of the gradient estimators can be found in Appendix D.

Original: 3. Experiments
Revised: 3. Experiments

Original: We evaluate NVI on two tasks, (1) learning to sample form an unnormalized target density where intermediate densities are generated along a geometric annealing path, and (2) learning intermediate densities to generate posterior samples of a hidden Markov model.
Revised: We evaluate NVI on two tasks, (1) learning to sample form an unnormalized target density where intermediate densities are generated along a geometric annealing path, and (2) learning intermediate densities to generate posterior samples of a hidden Markov model.

Original: 3.1. Sampling from an unnormalized target density via annealing
Revised: 3.1. Sampling from an unnormalized target density via annealing

Original: We are targeting a 2-dimensional unnormalized Gaussian mixture model (GMM) γ K with
Revised: We are targeting a 2-dimensional unnormalized Gaussian mixture model (GMM) γ K with

Original: M = 8 equidistantly spaced modes along a circle a round the origin.
Revised: M = 8 equidistantly spaced modes along a circle a round the origin.

Original: Starting from a initial proposal q 1 ( z 1 ) = γ 1 ( z 1 ) we construct a sequence of K annealed densitiesγ k ( z ) =
Revised: Starting from a initial proposal q 1 ( z 1 ) = γ 1 ( z 1 ) we construct a sequence of K annealed densities

Original: q 1 ( z ) 1 − β k γ K ( z ) β k , β k =
Revised: [Equation]

Original: k −
Revised: for

Original: K − 1 , for k = 1 . . .
Revised: [Equation]

Original: K equi-distantly scheduled along a geometric annealing path.
Revised: K equi-distantly scheduled along a geometric annealing path.

Original: We deﬁne the corresponding forward and reverse densities using forward and reverse kernel,ˆ [Equation] [Equation] =
Revised: We deﬁne the corresponding forward and reverse densities using forward and reverse kernel,

Original: [Equation]
Revised: [Equation]

Original: 1 | z k ) .

Original: We compare 4 diﬀerent variants of Nested Variational Inference (NVI, NVIR, NVI*, and
Revised: We compare 4 diﬀerent variants of Nested Variational Inference (NVI, NVIR, NVI*, and

Original: NVIR*), which optimize an exclusive KL-divergence at every step, and Annealed Variational objectives (AVO)
Revised: NVIR*), which optimize an exclusive KL-divergence at every step, and Annealed Variational objectives (AVO)

Original: (Huang et al., 2018).
Revised: (Huang et al., 2018).

Original: NVIR employs additional resampling after every step, NVI* additionally learns the annealing schedule of the intermediate densities, and
Revised: NVIR employs additional resampling after every step, NVI* additionally learns the annealing schedule of the intermediate densities, and

Original: NVIR* combines both.
Revised: NVIR* combines both.

Original: All methods use the architecture described above for the forward and reverse kernels and are trained for 20 , 000 iteration using Adam with a learning rate of 1e − 3 .
Revised: All methods use the architecture described above for the forward and reverse kernels and are trained for 20 , 000 iteration using Adam with a learning rate of 1e − 3 .

Original: A detailed description of the model and architecture can be found in Appendix C.1.
Revised: A detailed description of the model and architecture can be found in Appendix C.1.

Original: We report the sample quality of the learned samplers in terms of the log average weight and eﬀective sample size in Table 1 and show the learned annealing schedules and samples from the intermediate densities in Figure 1.
Revised: We report the sample quality of the learned samplers in terms of the log average weight and eﬀective sample size in Table 1 and show the learned annealing schedules and samples from the intermediate densities in Figure 1.

Original: Our results show that samplers trained with
Revised: Our results show that samplers trained with

Original: NVI are able to more accurately estimate the log normalizing constant whilst maintaining a higher eﬀective sample sizes compared to AVO.
Revised: NVI are able to more accurately estimate the log normalizing constant whilst maintaining a higher eﬀective sample sizes compared to AVO.

Original: Moreover, NVI* and NVIR* learn more equi-distantly spaced annealing schedules in terms of KL-divergence.
Revised: Moreover, NVI* and NVIR* learn more equi-distantly spaced annealing schedules in terms of KL-divergence.

Original: Both learning the annealing schedule and resampling empirically helps to learn better samplers.
Revised: Both learning the annealing schedule and resampling empirically helps to learn better samplers.

Original: 3.2. Learning Intermediate Heuristic for Hidden Markov Models
Revised: 3.2. Learning Intermediate Heuristic for Hidden Markov Models

Original: Here, we are considering a Hidden Markov Model (HMM) with a GMM likelihood consisting of M cluster (see Appendix C.2) with data points x 1: T , global variables η , and hidden states z 1: T .
Revised: Here, we are considering a Hidden Markov Model (HMM) with a GMM likelihood consisting of M cluster (see Appendix C.2) with data points x 1: T , global variables η , and hidden states z 1: T .

Original: We can deﬁne a sequence of unnormalized target densities { γ t } Tt =0 as γ 0 ( x 1: T , η ) = p ( η ) , γ t ( z 1: t , x 1: T , η ) = p ( z 1: t , x 1: t , η ) .
Revised: We can deﬁne a sequence of unnormalized target densities { γ t } Tt =0 as γ 0 ( x 1: T , η ) = p ( η ) , γ t ( z 1: t , x 1: T , η ) = p ( z 1: t , x 1: t , η ) .

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Intuitively, this evaluates the utility of a sample for the global variables η based on the probability under the HMM up to current step t , neglecting future observations x t : T .
Revised: Intuitively, this evaluates the utility of a sample for the global variables η based on the probability under the HMM up to current step t , neglecting future observations x t : T .

Original: If the observations up to step t do only contain a small subset of possible states this can lead to mode collapse.
Revised: If the observations up to step t do only contain a small subset of possible states this can lead to mode collapse.

Original: Here we design a sequence of target densities by learning a heuristic factor Ψ( · ; θ ) as part of our target densities
Revised: Here we design a sequence of target densities by learning a heuristic factor Ψ( · ; θ ) as part of our target densities

Original: [Equation]
Revised: [Equation]

Original: : t ,
Revised: : t ,

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The heuristic factor evaluates the likelihood of future data points given the global variables.
Revised: The heuristic factor evaluates the likelihood of future data points given the global variables.

Original: We consider two heuristic factors which enumerate over individual clusters: (1) a GMMstyle heuristic factor and (2) a neural heuristic factor
Revised: We consider two heuristic factors which enumerate over individual clusters: (1) a GMMstyle heuristic factor and (2) a neural heuristic factor

Original: [Equation]
Revised: [Equation]

Original: We compare four diﬀerent variants of NVI, NVIR without a heuristic factor, NVIR-GMM using the GMM-style heuristic factor, NVIR* and NVIR × , which both use the neural heuristic factor.
Revised: We compare four diﬀerent variants of NVI, NVIR without a heuristic factor, NVIR-GMM using the GMM-style heuristic factor, NVIR* and NVIR × , which both use the neural heuristic factor.

Original: We found NVIR × , a variant of NVIR* which detaches a part of the gradient, described in appendix C.2, using an inclusive KL-divergence on a hidden
Revised: We found NVIR × , a variant of NVIR* which detaches a part of the gradient, described in appendix C.2, using an inclusive KL-divergence on a hidden

Original: Table 1 shows that, as expected, employing a heuristic factor results in better sample quality in terms of log average importance weight and eﬀective sample size.
Revised: Table 1 shows that, as expected, employing a heuristic factor results in better sample quality in terms of log average importance weight and eﬀective sample size.

Original: Moreover the neural heuristic factor outperforms the GMM-style heuristic factor.
Revised: Moreover the neural heuristic factor outperforms the GMM-style heuristic factor.

Original: 4. Conclusion
Revised: 4. Conclusion

Original: We develop NVI, a framework that combines nested importance sampling and variational inference by optimizing a variational objective at every level of nesting.
Revised: We develop NVI, a framework that combines nested importance sampling and variational inference by optimizing a variational objective at every level of nesting.

Original: The formulation allows to learn proposals and intermediate densities for a general class of samplers, which admit most commonly used importance sampling strategies as special cases.
Revised: The formulation allows to learn proposals and intermediate densities for a general class of samplers, which admit most commonly used importance sampling strategies as special cases.

Original: Our experiments demonstrate that samplers, targeting (a) an unnormalized GMM and (b) the posterior of a
Revised: Our experiments demonstrate that samplers, targeting (a) an unnormalized GMM and (b) the posterior of a

Original: HMM, trained with NVI are able to outperform baselines in terms of log average weight and eﬀective sampling size.
Revised: HMM, trained with NVI are able to outperform baselines in terms of log average weight and eﬀective sampling size.

Original: Moreover, we found that learning intermediate distributions based on the inclusive and exclusive KL-divergence results in better samplers in our experiments.
Revised: Moreover, we found that learning intermediate distributions based on the inclusive and exclusive KL-divergence results in better samplers in our experiments.

Revised: Acknowledgments

Revised: We would like to thank our reviewers for their thoughtful comments.

Revised: This work was supported by the Intel Corporation, the 3M Corporation, NSF award 1835309, startup funds from Northeastern University, the Air Force Research Laboratory (AFRL) and DARPA

Original: Appendix A. Notation
Revised: Appendix A. Notation

Original: v k = γ k
Revised: v k = γ k

Original: ( z k ) r k ( z k − 1 |
Revised: ( z k ) r k ( z k − 1 |

Original: z k , ˇ φ k ) γ k − 1 ( z k − 1 ) q k
Revised: z k , ˇ φ k ) γ k − 1 ( z k − 1 ) q k

Original: ( z k | z k − 1 , ˆ φ k ) k -th incremental weight ˜ v k
Revised: ( z k | z k − 1 , ˆ φ k ) k -th incremental weight ˜ v k

Original: = v k
Revised: = v k

Original: Z k − 1 Z k
Revised: Z k − 1 Z k

Original: =
Revised: =

Original: π
Revised: π

Original: ( z k ) r k ( z k − 1 |
Revised: ( z k ) r k ( z k − 1 |

Original: z
Revised: z

Original: k , ˇ φ k ) πk − 1( z k − 1 ) q k
Revised: k , ˇ φ k ) πk − 1( z k − 1 ) q k

Original: ( z k | z k − 1 , ˆ φ k ) k
Revised: ( z k | z k − 1 , ˆ φ k ) k

Original: -th incremental weight w k = (cid:81) kk (cid:48)
Revised: -th incremental weight w k = (cid:81) kk (cid:48)

Original: =1 v k (cid:48)
Revised: =1 v k (cid:48)

Original: k -th cumulative weight π k ( z k ) k
Revised: k -th cumulative weight π k ( z k ) k

Original: -th target density on Z k γ k
Revised: -th target density on Z k γ k

Original: ( z k ) =
Revised: ( z k ) =

Original: Z k π k ( z k ) k -th unnormalized target ˇ π k ( z k , z k − 1 ) =
Revised: Z k π k ( z k ) k -th unnormalized target ˇ π k ( z k , z k − 1 ) =

Original: π k ( z k ) r k ( z k − 1 |
Revised: π k ( z k ) r k ( z k − 1 |

Original: z
Revised: z

Original: k , ˇ φ k ) k -th extended target on Z k − 1 ×
Revised: k , ˇ φ k ) k -th extended target on Z k − 1 ×

Original: Z k ˇ γ k ( z k , z k − 1 ) =
Revised: Z k ˇ γ k ( z k , z k − 1 ) =

Original: Z k ˇ π k ( z k , z
Revised: Z k ˇ π k ( z k , z

Original: k − 1 ) k -th extended unnormalized proposal ˆ π k ( z k , z k − 1 ) =
Revised: k − 1 ) k -th extended unnormalized proposal ˆ π k ( z k , z k − 1 ) =

Original: π k − 1 ( z k − 1 ) q k
Revised: π k − 1 ( z k − 1 ) q k

Original: ( z k | z k − 1 , ˆ φ k ) k -th extended proposal on Z k − 1 ×
Revised: ( z k | z k − 1 , ˆ φ k ) k -th extended proposal on Z k − 1 ×

Original: Z k ˆ γ k
Revised: Z k ˆ γ k

Original: ( z k , z k − 1 ) =
Revised: ( z k , z k − 1 ) =

Original: Z k ˆ π k
Revised: Z k ˆ π k

Original: ( z k , z k − 1 ) k -th extended unnormalized proposal
Revised: ( z k , z k − 1 ) k -th extended unnormalized proposal

Original: Appendix B. Important Identities
Revised: Appendix B. Important Identities

Original: Thermodynamic Identity:
Revised: Thermodynamic Identity:

Original: [Equation]
Revised: [Equation]

Original: Log-derivative trick a.k.a.
Revised: Log-derivative trick a.k.a.

Original: reinforce trick:
Revised: reinforce trick:

Original: [Equation]
Revised: [Equation]

Original: Consequently, it holds that
Revised: Consequently, it holds that

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Fisher’s Identity:
Revised: Fisher’s Identity:

Original: [Equation]
Revised: [Equation]

Original: Appendix C. Experiment Details C.1. Experiment 1: Annealing
Revised: Appendix C. Experiment Details C.1. Experiment 1: Annealing

Original: We are targeting an unnormalized Gaussian mixture model (GMM) γ K with M = 8 equidistantly spaced modes along a circle with radius r = 10, γ K ( z K ) =
Revised: We are targeting an unnormalized Gaussian mixture model (GMM) γ K with M = 8 equidistantly spaced modes along a circle with radius r = 10, γ K ( z K ) =

Original: M (cid:88) k
Revised: M (cid:88) k

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: , for m = 1 , 2 , ..., M and σ = 0 . 5.
Revised: , for m = 1 , 2 , ..., M and σ = 0 . 5.

Original: The model the forward and backward kernel as conditional
Revised: The model the forward and backward kernel as conditional

Original: Gaussian, where the mappings for the means µ k and standard deviations σ k consist of a multilayer perceptron with a single share hidden layer of 50 neurons with sigmoid activation functions
Revised: Gaussian, where the mappings for the means µ k and standard deviations σ k consist of a multilayer perceptron with a single share hidden layer of 50 neurons with sigmoid activation functions

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: z k
Revised: z k

Original: C.2. Experiment 2: Hidden Markov Models
Revised: C.2. Experiment 2: Hidden Markov Models

Original: In the second set of experiments, we evaluate NVIR on a hidden Markov model with a GMM likelihood over data points x 1: T , global variables for the GMM components
Revised: In the second set of experiments, we evaluate NVIR on a hidden Markov model with a GMM likelihood over data points x 1: T , global variables for the GMM components

Original: η = { τ 1: M , µ 1: M } , and local states z 1: T ,
Revised: η = { τ 1: M , µ 1: M } , and local states z 1: T ,

Original: [Equation]
Revised: [Equation]

Original: We then construct forward densities by deﬁning proposals q 0 ( η | x 1: T ; φ ) , q 1 ( z 1 | x 1 , η ; φ ) and q t ( z t | z t − 1 , x t , η ; φ ) for t = 2 , . . .
Revised: We then construct forward densities by deﬁning proposals q 0 ( η | x 1: T ; φ ) , q 1 ( z 1 | x 1 , η ; φ ) and q t ( z t | z t − 1 , x t , η ; φ ) for t = 2 , . . .

Original: T .
Revised: T .

Original: At each step, we deﬁne a inclusive KL divergence
Revised: At each step, we deﬁne a inclusive KL divergence

Original: L 0 ( φ ) =
Revised: L 0 ( φ ) =

Original: KL( π ( x 1: T , η ) || q 0 ( η | x 1: T ; φ )) ,
Revised: KL( π ( x 1: T , η ) || q 0 ( η | x 1: T ; φ )) ,

Original: L 1 ( φ, θ 0 )
Revised: L 1 ( φ, θ 0 )

Original: =
Revised: =

Original: KL( π ( z 1 , x 1: T , η ) ||
Revised: KL( π ( z 1 , x 1: T , η ) ||

Original: π ( x 1: T , η ; θ 0 ) q 1 ( z 1 | x 1 , η ; φ ))
Revised: π ( x 1: T , η ; θ 0 ) q 1 ( z 1 | x 1 , η ; φ ))

Original: L t ( φ, θ t − 1 ) =
Revised: L t ( φ, θ t − 1 ) =

Original: KL( π ( z 1: t , x 1: T , η ) ||
Revised: KL( π ( z 1: t , x 1: T , η ) ||

Original: π ( z 1: t − 1 , x 1: T , η ; θ t − 1 ) q
Revised: π ( z 1: t − 1 , x 1: T , η ; θ t − 1 ) q

Original: ( z t | z t − 1 , x t , η ; φ )) , t=2, 3, ..., T .
Revised: ( z t | z t − 1 , x t , η ; φ )) , t=2, 3, ..., T .

Original: We empirically found that we achieve better results when detaching the target density (i.e. the left-hand-side density in each KL) in the objective.
Revised: We empirically found that we achieve better results when detaching the target density (i.e. the left-hand-side density in each KL) in the objective.

Original: This version of NVIR* is denoted NVIR × .
Revised: This version of NVIR* is denoted NVIR × .

Original: We compute self-normalized gradient estimates
Revised: We compute self-normalized gradient estimates

Original: [Equation]
Revised: [Equation]

Original: where the importance weights are deﬁned as
Revised: where the importance weights are deﬁned as

Original: [Equation]
Revised: [Equation]

Original: C.3. Architectures of the Proposals
Revised: C.3. Architectures of the Proposals

Original: We model the the proposal for global variables η := { τ 1: M , µ 1: M } as Normal-Gamma distribution, where the mapping for its parameters α 1: M , β 1: M , µ 1: M , ν 1: M consists of a LSTM with 2 layers, followed by two individual multilayer perceptron with a single hidden layer of 128 neurons,
Revised: We model the the proposal for global variables η := { τ 1: M , µ 1: M } as Normal-Gamma distribution, where the mapping for its parameters α 1: M , β 1: M , µ 1: M , ν 1: M consists of a LSTM with 2 layers, followed by two individual multilayer perceptron with a single hidden layer of 128 neurons,

Original: [Equation]
Revised: [Equation]

Original: We model the initial state proposal and consecutive state proposal as Categorical distributions, where the mapping for its parameters π t consists of a multilayer perceptron with a single hidden layer of 128 neurons and Tanh activation function,
Revised: We model the initial state proposal and consecutive state proposal as Categorical distributions, where the mapping for its parameters π t consists of a multilayer perceptron with a single hidden layer of 128 neurons and Tanh activation function,

Original: [Equation]
Revised: [Equation]

Original: Appendix D. Gradient estimation
Revised: Appendix D. Gradient estimation

Original: To compute the gradient of the nested variational objective (NVO) we need to compute the gradients of the individual terms D f (ˇ π k ||
Revised: To compute the gradient of the nested variational objective (NVO) we need to compute the gradients of the individual terms D f (ˇ π k ||

Original: ˆ π k ) w.r.t.
Revised: ˆ π k ) w.r.t.

Original: parameters ˇ φ k , ˆ φ k , θ k , and θ k − 1 .
Revised: parameters ˇ φ k , ˆ φ k , θ k , and θ k − 1 .

Original: [Equation]
Revised: [Equation]

Original: In the following we are deriving the relevant gradients for the general f-divergence, exclusive
Revised: In the following we are deriving the relevant gradients for the general f-divergence, exclusive

Original: KL-divergence ( f ( w ) = − log w ), and inclusive KL-divergence ( f ( w ) =
Revised: KL-divergence ( f ( w ) = − log w ), and inclusive KL-divergence ( f ( w ) =

Original: w log w ).
Revised: w log w ).

Original: D.1. Gradients for general f-divergences
Revised: D.1. Gradients for general f-divergences

Original: Gradient w.r.t.
Revised: Gradient w.r.t.

Original: parameters ˆ φ k of the forward kernel:
Revised: parameters ˆ φ k of the forward kernel:

Original: Reparameterizing the sample z k ≡ z k
Revised: Reparameterizing the sample z k ≡ z k

Original: (cid:15) k ; ˆ φ k ) allows us, under mild conditions 1 , to interchange the order of integration and diﬀerentiation and compute path-wise derivatives.
Revised: (cid:15) k ; ˆ φ k ) allows us, under mild conditions 1 , to interchange the order of integration and diﬀerentiation and compute path-wise derivatives.

Original: [Equation]
Revised: [Equation]

Original: w = v k
Revised: w = v k

Original: Zk − 1 Zk
Revised: Zk − 1 Zk

Original: Z k −
Revised: Z k −

Original: Z k ∂v k
Revised: Z k ∂v k

Original: ∂z k
Revised: ∂z k

Original: ∂z k ∂
Revised: ∂z k ∂

Original: ˆ φ k (cid:12)(cid:12)(cid:12)(cid:12)
Revised: ˆ φ k (cid:12)(cid:12)(cid:12)(cid:12)

Original: z k
Revised: z k

Original: =
Revised: =

Original: z k
Revised: z k

Original: ( (cid:15) k ; φ k )
Revised: ( (cid:15) k ; φ k )

Original: + ∂f ∂w (cid:12)(cid:12)(cid:12)(cid:12)
Revised: + ∂f ∂w (cid:12)(cid:12)(cid:12)(cid:12)

Original: w = v k
Revised: w = v k

Original: Zk − 1 Zk
Revised: Zk − 1 Zk

Original: Z k −
Revised: Z k −

Original: Z k ∂v k
Revised: Z k ∂v k

Original: ∂ ˆ φ k   
Revised: ∂ ˆ φ k   

Original:  =
Revised:  =

Original: E z k − 1 ∼ π k −  E (cid:15)
Revised: E z k − 1 ∼ π k −  E (cid:15)

Original: k ∼ p k  
Revised: k ∼ p k  

Original: ∂f ∂w (cid:12)(cid:12)(cid:12)(cid:12) w = v k
Revised: ∂f ∂w (cid:12)(cid:12)(cid:12)(cid:12) w = v k

Original: Zk − 1 Zk
Revised: Zk − 1 Zk

Original: Z k −
Revised: Z k −

Original: Z k (cid:32) ∂v k
Revised: Z k (cid:32) ∂v k

Original: ∂z k
Revised: ∂z k

Original: ∂z k ∂
Revised: ∂z k ∂

Original: ˆ φ k (cid:12)(cid:12)(cid:12)(cid:12)
Revised: ˆ φ k (cid:12)(cid:12)(cid:12)(cid:12)

Original: z k
Revised: z k

Original: =
Revised: =

Original: z k
Revised: z k

Original: ( (cid:15) k ; φ k ) − ∂q k ∂ ˆ φ k (cid:33)  
Revised: ( (cid:15) k ; φ k ) − ∂q k ∂ ˆ φ k (cid:33)  

Original:  =
Revised:  =

Original: E z k − 1 ∼ π k −  E (cid:15)
Revised: E z k − 1 ∼ π k −  E (cid:15)

Original: k ∼ p k  
Revised: k ∼ p k  

Original: ∂f ∂w (cid:12)(cid:12)(cid:12)(cid:12) w = v k
Revised: ∂f ∂w (cid:12)(cid:12)(cid:12)(cid:12) w = v k

Original: Zk − 

Revised: Zk −

Original: Zk v k
Revised: Zk v k

Original: Z k −
Revised: Z k −

Original: Z k (cid:32) ∂ log v k ∂z k ∂z k
Revised: Z k (cid:32) ∂ log v k ∂z k ∂z k

Original: ∂
Revised: ∂

Original: ˆ φ k (cid:12)(cid:12)(cid:12)(cid:12)
Revised: ˆ φ k (cid:12)(cid:12)(cid:12)(cid:12)

Original: z k
Revised: z k

Original: =
Revised: =

Original: z k
Revised: z k

Original: ( (cid:15) k ; φ k ) − ∂ log q k ∂ ˆ
Revised: ( (cid:15) k ; φ k ) − ∂ log q k ∂ ˆ

Original: φ k
Revised: φ k

Original: (cid:33)  
Revised: (cid:33)  

Original:  p.w. =
Revised:  p.w. =

Original: E w k
Revised: E w k

Original: [Equation]
Revised: [Equation]

Original: 1 ,z k − 1 ∼ Π k −  w k
Revised: 1 ,z k − 1 ∼ Π k −  w k

Original: [Equation]
Revised: [Equation]

Original: cZ k
Revised: cZ k

Original: [Equation]
Revised: [Equation]

Original: E (cid:15) k ∼ p k
Revised: E (cid:15) k ∼ p k

Original: [Equation]
Revised: [Equation]

Original: Alternatively, we can compute a score function gradient which does not require the target density γ k to be diﬀerentiable w.r.t.
Revised: Alternatively, we can compute a score function gradient which does not require the target density γ k to be diﬀerentiable w.r.t.

Original: the sample z k and hence can also be computed for discrete variable models.
Revised: the sample z k and hence can also be computed for discrete variable models.

Original: [Equation]
Revised: [Equation]

Original: Gradient w.r
Revised: Gradient w.r

Original: [Equation]
Revised: [Equation]

Original: t. parameters ˇ φ k of the reverse kernel:
Revised: t. parameters ˇ φ k of the reverse kernel:

Original: [Equation]
Revised: [Equation]

Original: .w.
Revised: .w.

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: Gradient
Revised: Gradient

Original: [Equation]
Revised: [Equation]

Original: p.w. =
Revised: p.w. =

Original: [Equation]
Revised: [Equation]

Original: Gradient w.r.t. parameters θ k −
Revised: Gradient w.r.t. parameters θ k −

Original: [Equation]
Revised: [Equation]

Original: log
Revised: log

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: ) (cid:20) f (cid:18)
Revised: ) (cid:20) f (cid:18)

Original: [Equation]
Revised: [Equation]

Original: (cid:19)(cid:21)
Revised: (cid:19)(cid:21)

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: k ) (cid:20)
Revised: k ) (cid:20)

Original: [Equation]
Revised: [Equation]

Original: (cid:18)
Revised: (cid:18)

Original: [Equation]
Revised: [Equation]

Original: (cid:19)(cid:21)(cid:35) =
Revised: (cid:19)(cid:21)(cid:35) =

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)
Revised: (cid:12)(cid:12)(cid:12)(cid:12)

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: v k Zk −
Revised: v k Zk −

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original:  =
Revised:  =

Original: [Equation]
Revised: [Equation]

Original: Zk
Revised: Zk

Original: [Equation]
Revised: [Equation]

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original:  (cid:18)
Revised:  (cid:18)

Original: [Equation]
Revised: [Equation]

Original: k −∂θ k −
Revised: k −∂θ k −

Original: [Equation]
Revised: [Equation]

Original: − ∂ log Z k −∂θ k −
Revised: − ∂ log Z k −∂θ k −

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: k − 1 ,z k ∼ ˆ π k     f (cid:18) v k Z k − 1 Z k (cid:19) − ∂f ∂w (cid:12)(cid:12)(cid:12)(cid:12)
Revised: k − 1 ,z k ∼ ˆ π k     f (cid:18) v k Z k − 1 Z k (cid:19) − ∂f ∂w (cid:12)(cid:12)(cid:12)(cid:12)

Original: w = v k
Revised: w = v k

Original: Zk − 

Revised: Zk −

Original: Zk v k
Revised: Zk v k

Original: Z k −
Revised: Z k −

Original: Z k   ∂ log γ k −∂θ k −  −
Revised: Z k   ∂ log γ k −∂θ k −  −

Original: E z k − 1 ,z k ∼ ˆ π k     f (cid:18) v k Z k − 1 Z k (cid:19) − ∂f ∂w (cid:12)(cid:12)(cid:12)(cid:12) w = v k
Revised: E z k − 1 ,z k ∼ ˆ π k     f (cid:18) v k Z k − 1 Z k (cid:19) − ∂f ∂w (cid:12)(cid:12)(cid:12)(cid:12) w = v k

Original: Zk − 

Revised: Zk −

Original: Zk v k
Revised: Zk v k

Original: Z k −
Revised: Z k −

Original: Z k   
Revised: Z k   

Original: 
Revised: 

Original: E z k − 1 ∼ π k −  ∂ log γ k −∂θ k −  =
Revised: E z k − 1 ∼ π k −  ∂ log γ k −∂θ k −  =

Original: Cov ˆ
Revised: Cov ˆ

Original: π k  
Revised: π k  

Original: f (cid:18) v k Z k − 1 Z k (cid:19) − ∂f ∂w (cid:12)(cid:12)(cid:12)(cid:12) w = v k
Revised: f (cid:18) v k Z k − 1 Z k (cid:19) − ∂f ∂w (cid:12)(cid:12)(cid:12)(cid:12) w = v k

Original: Zk − 

Revised: Zk −

Original: Zk v k
Revised: Zk v k

Original: Z k −
Revised: Z k −

Original: Z k , ∂ log γ k −∂θ k −  =
Revised: Z k , ∂ log γ k −∂θ k −  =

Original: Cov ˆ
Revised: Cov ˆ

Original: π k  
Revised: π k  

Original: f (cid:18) v k Z k − 1 Z k (cid:19) , ∂ log γ k −∂θ k −  − Cov
Revised: f (cid:18) v k Z k − 1 Z k (cid:19) , ∂ log γ k −∂θ k −  − Cov

Original: ˆ π k  
Revised: ˆ π k  

Original: ∂f ∂w (cid:12)(cid:12)(cid:12)(cid:12) w = v k
Revised: ∂f ∂w (cid:12)(cid:12)(cid:12)(cid:12) w = v k

Original: Zk − 

Revised: Zk −

Original: Zk v k Z k − 1 Z k , ∂ log γ k −∂θ k −  p.w. =
Revised: Zk v k Z k − 1 Z k , ∂ log γ k −∂θ k −  p.w. =

Original: E w
Revised: E w

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:   w
Revised:   w

Original: [Equation]
Revised: [Equation]

Original: cZ
Revised: cZ

Original: [Equation]
Revised: [Equation]

Original: E z
Revised: E z

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: | z k − 1 , ˆ φ k )     f (cid:18) v
Revised: | z k − 1 , ˆ φ k )     f (cid:18) v

Original: [Equation]
Revised: [Equation]

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: Z k (cid:19) − ∂f ∂w (cid:12)(cid:12)(cid:12)(cid:12)
Revised: Z k (cid:19) − ∂f ∂w (cid:12)(cid:12)(cid:12)(cid:12)

Original: w = v k
Revised: w = v k

Original: Zk − 

Revised: Zk −

Original: Zk v k
Revised: Zk v k

Original: Z k −
Revised: Z k −

Original: Z k   ∂ log γ k −∂θ k −  
Revised: Z k   ∂ log γ k −∂θ k −  

Original:  −
Revised:  −

Original: E w
Revised: E w

Original: [Table]
Revised: [Table]

Original: − 1 ,z k − 1 ∼ Π k −
Revised: − 1 ,z k − 1 ∼ Π k −

Original: [Equation]
Revised: [Equation]

Original: w k
Revised: w k

Original: [Equation]
Revised: [Equation]

Original: cZ
Revised: cZ

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ˆ
Revised: ˆ

Original: [Equation]
Revised: [Equation]

Original: k )   
Revised: k )   

Original: [Equation]
Revised: [Equation]

Original: f (cid:18) v
Revised: f (cid:18) v

Original: [Equation]
Revised: [Equation]

Original: k (cid:19) −
Revised: k (cid:19) −

Original: [Equation]
Revised: [Equation]

Original: ∂w (cid:12) (cid:12)(cid:12)(cid:12) w =
Revised: ∂w (cid:12) (cid:12)(cid:12)(cid:12) w =

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: v
Revised: v

Original: [Table]
Revised: [Table]

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original:   
Revised:   

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: −
Revised: −

Original: [Table]
Revised: [Table]

Original:   ∂ log γ k −∂θ k −  
Revised:   ∂ log γ k −∂θ k −  

Original: 
Revised: 

Original: D.2. Gradients for the exclusive KL-divergence ( f ( w ) = − log( w ) )
Revised: D.2. Gradients for the exclusive KL-divergence ( f ( w ) = − log( w ) )

Original: Building on the deviations for the general case derived in D.1 we derive the gradients for the exclusive KL-divergence as special cases by substituting f ( w ) = − log( w ).
Revised: Building on the deviations for the general case derived in D.1 we derive the gradients for the exclusive KL-divergence as special cases by substituting f ( w ) = − log( w ).

Original: Gradient w.r
Revised: Gradient w.r

Original: [Equation]
Revised: [Equation]

Original: t. parameters ˆ φ k of the forward kernel:
Revised: t. parameters ˆ φ k of the forward kernel:

Original: The reparameterized gradient takes the form
Revised: The reparameterized gradient takes the form

Original: [Equation]
Revised: [Equation]

Original: whereas the score function gradient takes the form
Revised: whereas the score function gradient takes the form

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal equalities holds due to the reinforce property (Appendix B Equation 3)
Revised: The ﬁnal equalities holds due to the reinforce property (Appendix B Equation 3)

Original: [Equation]
Revised: [Equation]

Original: Gradient w.r.t.
Revised: Gradient w.r.t.

Original: parameters ˇ φ of the reverse kernel
Revised: parameters ˇ φ of the reverse kernel

Original: [Equation]
Revised: [Equation]

Original: Gradient w.r.t.
Revised: Gradient w.r.t.

Original: parameters θ k of the current target
Revised: parameters θ k of the current target

Original: [Equation]
Revised: [Equation]

Original: Gradient w.r.t.
Revised: Gradient w.r.t.

Original: parameters θ k − 1 of the current proposal
Revised: parameters θ k − 1 of the current proposal

Original: [Equation]
Revised: [Equation]

Original: D.3. Gradients for the inclusive KL-divergence ( f ( w ) = w log( w ) )
Revised: D.3. Gradients for the inclusive KL-divergence ( f ( w ) = w log( w ) )

Original: First notice that
Revised: First notice that

Original: [Equation]
Revised: [Equation]

Original: Hence the gradients for the inclusive KL-divergence follow by symmetry from the gradient of the exclusive KL-divergence by swapping the arguments and identifying the components r k , π k and corresponding parameters ˆ φ k , θ k with the components of the forward density q k , π k − 1 and parameters ˇ φ k , θ k − 1 respectively.
Revised: Hence the gradients for the inclusive KL-divergence follow by symmetry from the gradient of the exclusive KL-divergence by swapping the arguments and identifying the components r k , π k and corresponding parameters ˆ φ k , θ k with the components of the forward density q k , π k − 1 and parameters ˇ φ k , θ k − 1 respectively.

Original: Gradient w.r.t.
Revised: Gradient w.r.t.

Original: parameters ˆ φ k of the forward kernel:
Revised: parameters ˆ φ k of the forward kernel:

Original: [Equation]
Revised: [Equation]

Original: Gradient w.r.t.
Revised: Gradient w.r.t.

Original: parameters ˇ φ k of the reverse kernel:
Revised: parameters ˇ φ k of the reverse kernel:

Original: Note that the sample z k −is assumed to be not reparameterized.
Revised: Note that the sample z k −is assumed to be not reparameterized.

Original: Hence we only state the score-function gradient for the inclusive KL-divergence.
Revised: Hence we only state the score-function gradient for the inclusive KL-divergence.

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Gradient w.r.t.
Revised: Gradient w.r.t.

Original: parameters θ k of the current target
Revised: parameters θ k of the current target

Original: [Equation]
Revised: [Equation]

Original: Gradient w.r.t.
Revised: Gradient w.r.t.

Original: parameters θ k − 1 of the current proposal
Revised: parameters θ k − 1 of the current proposal

Original: [Equation]
Revised: [Equation]