D OUBLY R EPARAMETERIZED G RADIENT E STIMATORS FOR M ONTE C ARLO O BJECTIVES
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep latent variable models have become a popular model choice due to the scalable learning algorithms introduced by (Kingma & Welling, 2013; Rezende et al., 2014).
These approaches maximize a variational lower bound on the intractable log likelihood of the observed data.
Burda et al.
(2015) introduced a multi-sample variational bound, IWAE, that is at least as tight as the standard variational lower bound and becomes increasingly tight as the number of samples increases.
Counterintuitively, the typical inference network gradient estimator for the IWAE bound performs poorly as the number of samples increases (Rainforth et al., 2018; Le et al., 2018).
Roeder et al.
(2017) propose an improved gradient estimator, however, are unable to show it is unbiased.
We show that it is in fact biased and that the bias can be estimated efﬁciently with a second application of the reparameterization trick.
The doubly reparameterized gradient (DReG) estimator does not suffer as the number of samples increases, resolving the previously raised issues.
The same idea can be used to improve many recently introduced training techniques for latent variable models.
In particular, we show that this estimator reduces the variance of the IWAE gradient, the reweighted wake-sleep update (RWS) (Bornschein & Bengio, 2014), and the jackknife variational inference (JVI) gradient (Nowozin, 2018).
Finally, we show that this computationally efﬁcient, unbiased drop-in gradient estimator translates to improved performance for all three objectives on several modeling tasks.
I NTRODUCTION
Following the inﬂuential work by (Kingma & Welling, 2013; Rezende et al., 2014), deep generative models with latent variables have been widely used to model data such as natural images (Rezende & Mohamed, 2015; Kingma et al., 2016; Chen et al., 2016; Gulrajani et al., 2016), speech and music time-series (Chung et al., 2015; Fraccaro et al., 2016; Krishnan et al., 2015), and video (Babaeizadeh et al., 2017; Ha & Schmidhuber, 2018; Denton & Fergus, 2018).
The power of these models lies in combining learned nonlinear function approximators with a principled probabilistic approach, resulting in expressive models that can capture complex distributions.
Unfortunately, the nonlinearities that empower these model also make marginalizing the latent variables intractable, rendering direct maximum likelihood training inapplicable.
Instead of directly maximizing the marginal likelihood, a common approach is to maximize a tractable lower bound on the likelihood such as the variational evidence lower bound (ELBO) (Jordan et al., 1999; Blei et al., 2017).
The tightness of the bound is determined by the expressiveness of the variational family.
For tractability, a factorized variational family is commonly used, which can cause the learned model to be overly simplistic.
Burda et al.
(2015) introduced a multi-sample bound, IWAE, that is at least as tight as the ELBO and becomes increasingly tight as the number of samples increases.
Counterintuitively, although the bound is tighter, Rainforth et al.
(2018) theoretically and empirically showed that the standard inference network gradient estimator for the IWAE bound performs poorly as the number of samples increases due to a diminishing signal-to-noise ratio (SNR).
This motivates the search for novel gradient estimators.
Roeder et al.
(2017) proposed a lower-variance estimator of the gradient of the IWAE bound.
They speculated that their estimator was unbiased, however, were unable to prove the claim.
We show
: anonymized .
that it is in fact biased, but that it is possible to construct an unbiased estimator with a second application of the reparameterization trick which we call the IWAE doubly reparameterized gradient (DReG) estimator.
Our estimator is an unbiased, computationally efﬁcient drop-in replacement, and does not suffer as the number of samples increases, resolving the counterintuitive behavior from previous work (Rainforth et al., 2018).
Furthermore, our insight is applicable to alternative multisample training techniques for latent variable models: reweighted wake-sleep (RWS) (Bornschein & Bengio, 2014) and jackknife variational inference (JVI) (Nowozin, 2018).
In this work, we derive DReG estimators for IWAE, RWS, and JVI and demonstrate improved scaling with the number of samples on a simple example.
Then, we evaluate DReG estimators on MNIST generative modeling, Omniglot generative modeling, and MNIST structured prediction tasks.
In all cases, we demonstrate substantial unbiased variance reduction, which translates to improved performance over the original estimators.
B ACKGROUND
Our goal is to learn a latent variable generative model p θ ( x, z )
= p θ ( z ) p θ ( x | z ) where x are observed data and z are continuous latent variables.
The marginal likelihood of the observed data, p θ ( x ) = (cid:82) p θ ( x, z ) dz , is generally intractable.
Instead, we maximize a variational lower bound on log p θ ( x ) such as the ELBO
[Equation]
where q ( z | x ) is a variational distribution.
Following the inﬂuential work by (Kingma & Welling, 2013; Rezende et al., 2014), we consider the amortized inference setting where q φ ( z | x ) , referred to as the inference network, is a learnable function parameterized by φ that maps from x to a distribution over z .
The tightness of the bound is coupled to the expressiveness of the variational family (i.e., { q φ } φ ).
As a result, limited expressivity of { q φ } φ , can negatively affect the learned model.
Burda et al.
(2015) introduced the importance weighted autoencoder (IWAE) bound which alleviates this coupling
[Equation]
with z 1: K ∼ (cid:81) i q φ ( z i | x ) .
The IWAE bound reduces to the ELBO when K = 1 , is non-decreasing as K increases, and converges to log p θ ( x ) as K → ∞ under mild conditions (Burda et al., 2015).
When q φ is reparameterizable 1 , the standard gradient estimator of the IWAE bound is ∇ θ,
[Equation]
where w i = p θ ( x, z i ) /q
φ ( z i | x ) .
A single sample estimator of this expectation is typically used as the gradient estimator.
As K increases, the bound becomes increasingly tight, however, Rainforth et al.
(2018) show that the signal-to-noise ratio (SNR) of the inference network gradient estimator goes to 0 .
This does not happen for the model parameters ( θ ).
Following up on this work, Le et al.
(2018) demonstrate that this deteriorates the performance of learned models on practical problems.
Because the IWAE bound converges to log p θ ( x ) (as K → ∞ ) regardless of q φ , φ ’s affect on the bound must diminish as K increases.
It may be tempting to conclude that the SNR of the inference network gradient estimator must also decrease as K → ∞ .
However, low SNR is a limitation of the gradient estimator, not necessarily of the bound.
Although the magnitude of the gradient converges to 0 , if the variance of the gradient estimator decreases more quickly, then the SNR of the gradient estimator need not degrade.
This motivates the search for lower variance inference network gradient estimators.
To derive improved gradient estimators for φ , it is informative to expand the total derivative 2 of the
IWAE bound with respect to φ
[Equation]
Previously, Roeder et al.
(2017) found that the ﬁrst term within the parentheses of Eq.
3 can contribute signiﬁcant variance to the gradient estimator.
When K = 1 , this term analytically vanishes in expectation, so when K > 1 they suggested dropping it.
Below, we abbreviate this estimator as
STL.
As we show in Section 6.1, the STL estimator introduces bias when K > 1 .
D OUBLY R EPARAMETERIZED G RADIENT E STIMATORS (DR E G S )
Our insight is that we can estimate the ﬁrst term within the parentheses of Eq.
3 efﬁciently with a second application of the reparameterization trick.
To see this, ﬁrst note that
[Equation]
so it sufﬁces to focus on one of the K terms.
Because the derivative is a partial derivative ∂∂φ , it treats z i
=
z ( (cid:15) i , φ ) as a constant, so we can freely change the random variable that the expectation is over to z 1: K .
Now,
[Equation]
where z − i =
z 1: i − 1 ,i +1: K is the set of z 1: K without z i .
The inner expectation resembles a REIN- FORCE gradient term (Williams, 1992), where we interpret
[Equation]
j as the “reward”.
Now, we can use the following well-known equivalence between the REINFORCE gradient and the reparameterization trick gradient (See Appendix 8.1 for a derivation)
[Equation]
This holds even when f depends on φ .
Typically, the reparameterization gradient estimator has lower variance than the REINFORCE gradient estimator because it directly takes advantage of the derivative of f .
Applying the identity from Eq.
5 to the right hand side of Eq.
4 gives
[Equation]
This last expression can be efﬁciently estimated with a single Monte Carlo sample.
When z i is not reparameterizable (e.g., the models in (Mnih & Rezende, 2016)), we can use a control variate (e.g.,
K ∂∂φ log q φ ( z i | x ) ).
In both cases, when K = 1 , this term vanishes exactly and we recover the estimator proposed in (Roeder et al., 2017) for the ELBO.
However, when K > 1 , there is no reason to believe this term will analytically vanish.
Substituting Eq.
6 into Eq.
3, we obtain a simpliﬁcation due to cancellation of terms
[Equation]
We call the algorithm that uses the single sample Monte Carlo estimator of this expression for the inference network gradient the IWAE doubly reparameterized gradient estimator (IWAE-DReG).
This estimator has the property that when q ( z | x ) is optimal (i.e., q ( z | x )
= p ( z | x ) ), the estimator vanishes exactly and has zero variance, whereas this does not hold for the standard IWAE gradient estimator.
We provide an asymptotic analysis of the IWAE-DReG estimator in Appendix 8.2.
The conclusion of that analysis is that, in contrast to the standard IWAE gradient estimator, the SNR of the IWAE-DReG estimator exhibits the same scaling behaviour of O ( √ K ) for both the generation and inference network gradients (i.e., improving in K ).
A LTERNATIVE TRAINING ALGORITHMS
Now, we review alternative training algorithms for deep generative models and derive their doubly reparameterized versions.
4. R EWEIGHTED W AKE S LEEP (RWS)
Bornschein & Bengio (2014) introduced RWS, an alternative multi-sample update for latent variable models that uses importance sampling.
Computing the gradient of the log marginal likelihood
[Equation]
requires samples from p θ ( z | x ) , which is generally intractable.
We can approximate the gradient with a self-normalized importance sampling estimator
[Equation]
, where z 1: K ∼ (cid:81) i q φ ( z i | x ) .
Interestingly, this is precisely the same as the IWAE gradient of θ , so the RWS update for θ can be interpreted as maximizing the IWAE lower bound in terms of θ .
Instead of optimizing a joint objective for p and q , RWS optimizes a separate objective for the inference network.
(Bornschein & Bengio, 2014) propose a “wake” update and a “sleep” update for the inference network.
Le et al.
(2018) provide empirical support for solely using the wake update for the inference network, so we focus on that update.
The wake update approximately minimizes the KL divergence from p θ ( z | x ) to q φ ( z | x ) .
The gradient of the KL term is
[Equation]
The wake update of the inference network approximates the intractable expectation by selfnormalized importance sampling
[Equation]
with z i ∼ q φ ( z i | x ) .
Le et al.
(2018) note that this update does not suffer from diminishing SNR as K increases.
However, a downside is that the updates for p and q are not gradients of a uniﬁed objective, so could potentially lead to instability or divergence.
D OUBLY R EPARAMETERIZED R EWEIGHTED W AKE UPDATE
The wake update gradient for the inference network (Eq.
8) can be reparameterized − E z 1: K (cid:34) K (cid:88) i =
[Equation]
We call the algorithm that uses the single sample Monte Carlo estimator of this expression as the wake update for the inference network
RWS-DReG.
Interestingly, the inference network gradient estimator from (Roeder et al., 2017) can be seen as the sum of the IWAE gradient estimator and the wake update of the inference network (as the wake update minimizes, we add the negative of Eq. 9).
Their positive results motivate further exploration of convex combinations of IWAE-DReG and RWS-DReG
[Equation]
We refer to the algorithm that uses the single sample Monte Carlo estimator of this expression as
DReG( α ).
When α = 1 , this reduces to RWS-DReG, when α = 0 , this reduces to IWAE-DReG and when α = 0 .
5 , this reduces STL.
4. J ACKKNIFE V ARIATIONAL I NFERENCE (JVI)
Alternatively, Nowozin (2018) reinterprets the IWAE lower bound as a biased estimator for the log marginal likelihood.
He analyzes the bias and introduces a novel family of estimators, Jackknife
Variational Inference (JVI), which trade off reduction in bias for increased variance.
This additional ﬂexibility comes at the cost of no longer being a stochastic lower bound on the log marginal likelihood.
The ﬁrst-order JVI has signiﬁcantly reduced bias compared to IWAE, which empirically results in a better estimate of the log marginal likelihood with fewer samples (Nowozin, 2018).
For simplicity, we focus on the ﬁrst-order JVI estimator
[Equation]
K
K (cid:88) i =
E z − i   log  
K −(cid:88)
j (cid:54) =
i w j     .
It is straightforward to apply our approach to higher order JVI estimators.
D OUBLY R EPARAMETERIZED J ACKKNIFE V ARIATIONAL I NFERENCE (JVI)
The JVI estimator is a linear combination of K and K − 1 sample IWAE estimators, so we can use the doubly reparameterized gradient estimator (Eq. 7) for each term.
R ELATED W ORK
Mnih & Rezende (2016) introduced a generalized framework of Monte Carlo objectives (MCO).
The log of an unbiased marginal likelihood estimator is a lower bound on the log marginal likelihood by
Jensen’s inequality.
In this view, the ELBO can be seen as the MCO corresponding to a single importance sample estimator of the marginal likelihood with q θ as the proposal distribution.
Similarly,
IWAE corresponds to the K -sample estimator.
Maddison et al.
(2017) show that the tightness of an
MCO is directly related to the variance of the underlying estimator of the marginal likelihood.
However, Rainforth et al.
(2018) point out issues with gradient estimators of multi-sample lower bounds.
In particular, they show that although the IWAE bound is tighter, the standard IWAE gradient estimator’s SNR scales poorly with large numbers of samples, leading to degraded performance.
Le et al.
(2018) experimentally investigate this phenomenon and provide empirical evidence of this degradation across multiple tasks.
They ﬁnd that RWS (Bornschein & Bengio, 2014) does not suffer from this issue and ﬁnd that it can outperform models trained with the IWAE bound.
We conclude that it is not sufﬁcient to just tighten the bound; it is important to understand the gradient estimators of the tighter bound as well.
Wake-sleep is an alternative approach to ﬁtting deep generative models, ﬁrst introduced in (Hinton et al., 1995) as a method for training Hemholtz machines.
It was extended to the multi-sample setting by (Bornschein & Bengio, 2014) and the sequential setting in (Gu et al., 2015).
It has been applied to generative modeling of images (Ba et al., 2015).
E XPERIMENTS
To evaluate DReG estimators, we ﬁrst measure variance and signal-to-noise ratio (SNR) 3 of gradient estimators on a toy example which we can carefully control.
Then, we evaluate gradient variance and model learning on MNIST generative modeling, Omniglot generative modeling, and MNIST structured prediction tasks.
6. T OY G AUSSIAN
We reimplemented the Gaussian example from (Rainforth et al., 2018).
Consider the generative model with z ∼ N ( θ, I ) and x | z ∼ N ( z, I ) and inference network q φ ( z | x ) ∼ N ( Ax + b, 23 I ) , where φ = { A, b } .
As in (Rainforth et al., 2018), we sample a set of parameters for the model and inference network close to the optimal parameters (perturbed by zero-mean Gaussian noise with standard deviation 0 . 01 ), then estimate the gradient of the inference network parameters for increasing number of samples ( K ).
In addition to signal-to-noise ratio (SNR), we plot the squared bias and variance of the gradient estimators 4 in Fig.
The bias is computed relative to the expected value of the IWAE gradient estimator.
As a result, although the average of K ELBO gradient estimators is an unbiased estimator of the ELBO gradient, it is a biased gradient estimator of the IWAE objective.
Importantly, SNR does not penalize estimators that are biased, so trivial constant estimators can have inﬁnite SNR.
Thus, it is important to consider additional evaluation measures as well.
As K increases, the SNR of the IWAE-DReG estimator increases, whereas the SNR of the standard gradient estimator of IWAE goes to 0 , as previously reported.
Furthermore, we can see the bias present in the STL estimator.
As a check of our implementation, we veriﬁed that the observed “bias” for IWAE-DReG was statistically indistinguishable from 0 with a paired t-test.
For the biased estimators (e.g., STL), we could easily reject the null hypothesis with few samples.
[Figure]
6. G ENERATIVE MODELING
Training generative models of the binarized MNIST digits dataset is a standard benchmark task for latent variable models.
For this evaluation, we used the single latent layer architecture from (Burda et al., 2015).
The generative model used 50 Gaussian latent variables with an isotropic prior and passed z through two deterministic layers of 200 tanh units to parameterize factorized Bernoulli outputs.
The inference network passed x through two deterministic layers of 200 tanh units to parameterize a factorized Gaussian distribution over z .
Because our interest was in improved gradient estimators and optimization performance, we used the dynamically binarized MNIST dataset, which minimally suffers from overﬁtting.
We used the standard split of MNIST into train, validation
[Figure]
and test sets.
All dimensions of φ behaved qualitatively similarly, so for clarity, we show curves for a single randomly chosen dimension of φ .
We trained models with the IWAE gradient, the RWS wake update
[Figure]
and with the JVI estimator.
In all three cases, the doubly reparameterized gradient estimator reduced variance 5 and as a result substantially improved performance (Fig. 2)
[Figure]
We found similar behavior with different numbers of samples (Fig. 3 and Appendix Fig.
Interestingly, the biased gradient estimators STL and RWS-DReG perform best on this task with RWSDReG slightly outperforming STL.
As observed in (Le et al., 2018), RWS increasingly outperforms
IWAE as K increases.
Finally, we experimented with convex combinations of IWAE-DReG and
RWS-DReG (right Fig. 3).
On this dataset, convex combinations that heavily weighted RWS-DReG had the best performance.
However, as we show below, this is task dependent
[Figure]
Next, we performed the analogous experiment with the dynamically binarized Omniglot dataset using the same model architecture.
Again, we found that the doubly reparameterized gradient estimator reduced variance and as a result improved test performance (Figs.
and 6 in the Appendix).
6. S TRUCTURED PREDICTION ON MNIST
Structured prediction is another common benchmark task for latent variable models.
In this task, our goal is to model a complex observation x given a context c (i.e., model the conditional distribution p ( x | c ) ).
We can use a conditional latent variable model p θ ( x, z | c )
= p θ ( x | z, c ) p θ ( z | c ) , however, as before, computing the marginal likelihood is generally intractable.
It is straightforward to adapt the bounds and techniques from the previous section to this problem.
To evaluate our method in this context, we use the standard task of modeling the bottom half of a binarized MNIST digit from the top half.
We use a similar architecture, but now learn a conditional prior distribution p θ ( z | c ) where c is the top half of the MNIST digit.
The conditional prior feeds c to two deterministic layers of 200 tanh units to parameterize a factorized Gaussian distribution over z .
To model the conditional distribution p θ ( x | c, z ) , we concatenate z with c and feed it to two deterministic layers of 200 tanh units to parameterize factorized Bernoulli outputs.
As in the previous tasks, the doubly reparameterized gradient estimator improves across all three updates (IWAE, RWS, and JVI; Appendix Fig. 7).
However, on this task, the biased estimators (STL and RWS) underperform unbiased IWAE gradient estimators (Fig. 4).
In particular, RWS becomes unstable later in training.
We suspect that this is because RWS does not directly optimize a consistent objective.
[Figure]
j e c t i v e ( n a t s )
[Figure]
D ISCUSSION
In this work we introduce doubly reparameterized estimators for the updates in IWAE, RWS and
JVI.
We demonstrate that across tasks they provide unbiased variance reduction, which leads to improved performance.
Furthermore, DReG estimators have the same computational cost as the original estimators.
As a result, we recommend that DReG estimators be used instead of the typical gradient estimators.
Variational Sequential Monte Carlo (Maddison et al., 2017; Naesseth et al., 2018; Le et al., 2018) and Neural Adapative Sequential Monte Carlo (Gu et al., 2015) extend IWAE and RWS to sequential latent variable models, respectively.
It would be interesting to develop DReG estimators for these approaches as well.
We found that a convex combination of IWAE-DReG and RWS-DReG performed best, however, the weighting was task dependent.
In future work, we intend to apply ideas from (Baydin et al., 2017) to automatically adapt the weighting based on the data.
Finally, the form of the IWAE-DReG estimator (Eq. 7) is surprisingly simple and suggests that there may be a more direct derivation that is applicable to general MCOs.
A CKNOWLEDGMENTS
We thank Ben Poole and Diederik P. Kingma for helpful discussion and comments on drafts of this paper.
We thank Sergey Levine and Jascha Sohl-Dickstein for insightful discussion
A
[Figure] [Figure]
j
[Figure]
c t i v e
[Figure]
t s
[Figure]
8. E QUIVALENCE BETWEEN REINFORCE GRADIENT AND REPARAMETERIZATION TRICK GRADIENT
Given a function f
( z, φ ) we have
[Equation]
[Figure] [Figure]
d i e n t V a r i a n c
[Figure]
( R e l a t i v e )
[Figure]
for a reparameterizable distribution q φ ( z ) .
To see this, note that d
[Equation] [Equation]
=
[Equation] [Equation]
[Equation] [Equation]
, via the REINFORCE gradient.
On the other hand, d dφ
[Equation]
=
[Equation] [Equation]
(cid:20)
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
) (cid:21)
[Equation] [Equation]
[Equation] [Equation]
(cid:21) , via the reparameterization trick.
Thus, we conclude that
E [Equation] φ ( z )
[Equation]
(cid:20)
[Equation]
∂φ (cid:21) , from which the identity follows.
8. I NFORMAL ASYMPTOTIC ANALYSIS
At a high level, Rainforth et al.
(2018) show that the expected value of the IWAE gradient of the inference network collapses to zero with rate 1 /K , while its standard deviation is only shrinking at a rate of 1 / √ K .
This is the essence of the problem that results in the SNR (expectation divided by standard deviation) of the inference network gradients going to zero at a rate
O
((1 /K ) / (1 / √ K ))
=
O (1 / √ K ) , worsening with K .
In contrast, Rainforth et al.
(2018) show that the generation network gradients scales like O ( √ K ) , improving with K .
Because the IWAE-DReG estimator is unbiased, we cannot hope to change the scaling of the expected value in K , but we can hope to change the scaling of the variance.
In particular, in this subsection, we provide an informal argument, via the delta method, that the standard deviation of
IWAE-DReG scales like K − 3 / 2 , which results in an overall scaling of O ( √ K ) for the inference network gradient’s SNR (i.e., increasing with K ).
Thus, the SNR of the IWAE-DReG estimator improves similarly in K for both inference and generation networks.
We will appeal to the delta method on a two-variable function g : R 2 → R .
Deﬁne the following notation for the partials of g evaluated at the mean of random variables
X, Y ,
[Equation]
The delta method approximation of Var( g ( X, Y )) is given by (Section 5.5 of Casella & Berger), Var [Equation]
g [Equation] X, Y ))
≈ g [Equation] X, Y ) 2 Var [Equation]
X ) + [Equation] g
[Equation]
X, [Equation] X, Y
[Equation]
X, Y ) + g y
[Equation]
X, Y ) 2 Var( Y )
Now, assume without loss of generality that φ is a single real-valued parameter.
Let u i =
w 2 i ∂ log w i ∂z
i ∂z i ∂φ ,
X = (cid:80)
Ki
=1 u i , and Y =
(cid:80)
Ki
=1 w i .
Let g ( X, Y ) =
X/Y 2 , then g ( X, Y ) is the
IWAE-DReG estimator whose variance we seek to understand.
Letting Z = E ( w i ) and U
=
E ( u i ) we get in this case after cancellations,
[Equation]
Because w i are all mutually independent, we get Var( Y ) =
K Var( w i ) .
Similarly for Var( X ) and u i .
Because the w i and u i are identically distributed and independent for i (cid:54) =
j , we have
Cov( X, Y ) =
K Cov( w i , u i ) .
All together we can see that Var( g ( X, Y )) scales like K − 3 .
Thus, the standard deviation scales like K − 3 / 2 .
8. U NIFIED S URROGATE O BJECTIVES FOR E STIMATORS
In the main text, we assumed that θ and φ were disjoint, however, it can be helpful to share parameters between p and q (e.g., (Fraccaro et al., 2016)).
With the IWAE bound, we differentiate a single objective with respect to both the p and q parameters.
Thus it is straightforward to adapt IWAE and
IWAE-DReG to the shared parameter setting.
In this section, we discuss how to deal with shared parameters in RWS.
Suppose that both p and q are parameterized by θ .
If we denote the unshared parameters of q by φ , then we can restrict the RWS wake update to only φ .
Alternatively, with a modiﬁed RWS wake update, we can derive a single surrogate objective for each scenario such that taking the gradient with respect to θ results in the proper update.
For clarity, we introduce the following modiﬁer notation for p θ ( x, z i ) , q θ ( z i | x ) , and w i which are functions of θ and z i =
z ( θ, (cid:15) i ) .
We use ˜ X to mean X with stopped gradients with respect to z i , ˆ X to mean X with stopped gradients with respect to θ (but not θ is not stopped in z ( θ, (cid:15) i ) ), and ¯ X to mean X with stopped gradients for all variables.
Then, we can use the following surrogate objectives:
IWAE:
[Equation]
IWAE
[Equation]
: K (cid:34)
[Equation]
i =
[Equation]
i (cid:35) (11)
DReG IWAE:
[Equation]
DReG
[Equation]
IWAE (
[Equation]
: K
[Equation]
K (cid:88) i =
¯ w i (cid:80)
j ¯ w j log ˜
p
[Equation]
( x
[Equation]
z i ) + (cid:32) ¯ w i (cid:80)
j ¯ w j (cid:33)log ˆ w i 
 (12)
RWS:
[Equation]
RWS ( θ ) = E (cid:15) 1: K (cid:34) K (cid:88) i =
¯ w i (cid:80)
j ¯ w j (log ˜ p
θ ( x, z i )
+
log ˜
q θ ( z i | x )) (cid:35) (13)
DReG RWS:
L DReG − RWS ( θ ) =
E (cid:15) 1: K  
[Equation]
(cid:88) i = ¯
[Equation]
i (cid:80) j ¯
[Equation]
j
[Equation]
˜ p
[Equation]
[Equation]
, z i )
[Equation]
  ¯
[Equation]
i (cid:80) j ¯ w j − (cid:32) ¯ w i (cid:80)
j ¯ w j (cid:33) 2 
 log ˆ w i 
 (14)
STL:
[Equation]
: K
[Equation]
(cid:35) (15)
DReG( α ):
L DReG ( α ) ( θ ) = E (cid:15) 1: K  
K (cid:88)
[Equation]
The only subtle difference is that DReG( α = 0 .
5 ) does not correspond exactly to STL due to the scaling between terms:
[Equation]