P ARTIAL R EJECTION C ONTROL FOR R OBUST V ARIATIONAL I NFERENCE IN S EQUENTIAL L ATENT V ARIABLE M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Effective variational inference crucially depends on a ﬂexible variational family of distributions.
Recent work has explored sequential Monte-Carlo (SMC) methods to construct variational distributions, which can, in principle, approximate the target posterior arbitrarily well, which is especially appealing for models with inherent sequential structure.
However, SMC, which represents the posterior using a weighted set of particles, often suffers from particle weight degeneracy, leading to a large variance of the resulting estimators.
To address this issue, we present a novel approach that leverages the idea of partial rejection control (PRC) and enables us to develop a robust variational inference (VI) framework.
Although PRC constructs a low variance estimator of the marginal likelihood, its applicability in VI has not been considered before.
In addition to developing a superior VI bound, we also propose a novel marginal likelihood estimator constructed via a dice-enterprise : a generalization of the Bernoulli factory to construct unbiased estimators for SMC-PRC.
The resulting variational lower bound can be optimized efﬁciently with respect to the variational parameters.
We show theoretical properties of the lower bound and report experiments on various sequential models, such as the Gaussian state-space model and variational RNN, on which our approach outperforms existing methods.
I NTRODUCTION
Exact inference in latent variable models is usually intractable.
Markov Chain Monte-Carlo (MCMC) (Andrieu et al., 2003) and variational methods (Blei et al., 2017), are commonly employed in such models to make inference tractable.
While MCMC has been the traditional method of choice, often with provable guarantees, optimization based variational inference (VI) methods have also enjoyed considerable recent interest due to their excellent scalability on large-scale datasets.
VI is based on maximizing a lower bound constructed through a marginal likelihood estimator.
For latent variable models with sequential structure, sequential Monte Carlo (SMC) (Doucet & Johansen, 2009) returns a much lower variance estimator for the log marginal likelihood than importance sampling (IS) (Bérard et al., 2014; Cérou et al., 2011).
In this work, we focus our attention on designing a low variance, unbiased , and computationally efﬁcient estimator of the marginal likelihood.
The performance of SMC based methods is strongly dependent on the choice of the proposal distribution.
Inadequate proposal distributions propose values in areas of low probability under the target, leading to particle depletion (Doucet & Johansen, 2009).
An effective solution is to use rejection control (Liu et al., 1998; Peters et al., 2012) which is based on an approximate rejection sampling step within SMC to reject samples with low importance weights.
In this work, we leverage the idea of partial rejection control (PRC) within the framework of SMC based VI for sequential latent variable models.
To this end, we construct a novel lower bound, VSMC-PRC, and propose an efﬁcient optimization strategy for selecting the variational parameters.
Compared to other recent SMC based VI approaches (Naesseth et al., 2017; Maddison et al., 2017; Le et al., 2017), our approach consists of an inbuilt accept-reject mechanism within SMC to prevent particle depletion and obtains an unbiased marginal likelihood estimator.
Recently an unbiased estimator for SMC-PRC was developed by Kudlicka et al. (2020) who normalized their estimator by the number of propogations used until an acceptance. We use a different approach that utilizes an efﬁcient implementation of a generalization of a Bernoulli factory.
Although the idea of combining VI with an inbuilt accept-reject mechanism is not new (Salimans et al., 2015; Ruiz & Titsias, 2019; Grover et al., 2018; Gummadi, 2014), a key distinction of our approach is to incorporate a partial accept-reject mechanism.
In contrast to standard sampling algorithms that may reject the entire stream of particles, we use accept-reject on the most recent update, increasing the sampling efﬁciency.
Another key distinction is that, aside from Kudlicka et al. existing approaches using PRC (Liu et al., 1998; Peters et al., 2012) construct a biased marginal likelihood estimator, while we leverage insights from dice-enterprise (Morina et al., 2019) and Bernoulli factory (Asmussen et al., 1992) to obtain an unbiased estimator.
The rest of the paper is organized as follows:
In Section 2, we provide a brief review on SMC, partial rejection control, and dice enterprise.
In Section 3, we introduce our VSMC-PRC bound and provide new theoretical insights into the Monte-Carlo estimator and design efﬁcient ways to optimize it.
Finally, we discuss related work and present experiments on the Gaussian state-space model (SSM) and variational recurrent neural networks (RNN).
B ACKGROUND
We denote a sequence of T real-valued observations as x 1: T = ( x 1 , x 2 , . . .
, x T ) , and assume that there is an associated sequence of latent variables
z 1: T = ( z 1 , z 2 , . . .
, z T ) .
We are interested in inferring the posterior distribution of the latent variables, i.e., p ( z 1: T | x 1: T ) .
The task is, in general, intractable.
For the rest of the paper we have used some common notations from SMC and VI literature where z it : i th particle at time t ; A it − 1 : ancestor variable for the
i th particle at time t ; θ and φ : model and variational parameters, respectively.
2. S EQUENTIAL M ONTE C ARLO WITH P ARTIAL R EJECTION C ONTROL
An SMC sampler approximates a sequence of densities { p θ ( z 1: t | x 1: t ) }
Tt =1 through a set of N weighted samples generated from a proposal distribution.
Let the proposal density be
[Equation]
Consider time t − 1 at which we have uniformly weighted samples { N − 1 , z i 1: t − 1 , A it − 1 } Ni =1 estimating p θ ( z 1: t − 1 | x 1: t − 1 ) .
We want to estimate p θ ( z 1: t | x 1: t ) such that particles with a low importance weight are automatically rejected.
PRC achieves this by using an approximate rejection sampling step (Liu et al., 1998; Peters et al., 2012).
The overall procedure is as follows:
Generate z it ∼ q φ ( z t | x 1: t , z
A it − 1 1: t − 1 ) where i = 1 , 2 , . . .
, N . 2.
Accept z it with probability
[Equation]
where M ( i, t − 1) is a hyperparameter controlling the acceptance rate (see Propositionand Section (3.3) for more details).
Note that PRC applies accept-reject only on z it , not on the entire trajectory.
If z it is rejected go to step 1. 4.
The new incremental importance weight of the accepted sample is
[Equation]
where c it is
[Equation]
and the intractable normalization constant Z ( . ) (For simplicity of notation, we have ignored the dependence of Z ( . ) on M ( i, t − 1) )
[Equation]
5. Compute Monte-Carlo estimator of unnormalized weight
[Equation]
where δ i,kt ∼ q φ ( z t | x 1: t , z A
it − 1 1: t − 1 ) and k = 1 , 2 , . . , K .
Note that (cid:101) w it would be essential for constructing unbiased estimator for p θ ( x 1: T ) .
Generate ancestor variables A it through dice-enterprise and set new weights w it =
N − 1 for i = 1 , 2 , . .
, N : A it ∼ Categorical
[Equation]
2. D ICE E NTERPRISE Simulation of ancestor
variables in Eq.
(7) is non-trivial due to intractable normalization constants in the incremental importance weight (see Eq.
Vanilla Monte-Carlo estimation of α t ( . ) yields biased samples of ancestor variables from Eq.
To address this issue, we leverage a generalization of Bernoulli factory (Asmussen et al., 1992), called dice-enterprise (Morina et al., 2019).
Suppose we can simulate Bernoulli ( p it ) outcomes where p it is intractable.
Bernoulli factory problem simulates an event with probability f
( p it ) , where f ( . ) is some desired function.
In our case, the intractable coin probability p it is the intractable normalization constant,
[Equation]
Since p it ∈ [0 , 1] and we can easily simulate this coin, we obtain the dice-enterprise algorithm below.
Required:
Constants { c it } Ni =1 see (4).
Sample C ∼ Categorical
[Equation]
If C = i , generate U i ∼ U [0 , 1] and z t ∼ q φ ( z t | x 1: t ,
z A it
[Equation]
[Equation]
t − 1 ) • If U
i < a θ,φ ( z t | z
A i t − 1 1: t − 1 , x 1: t ) output i • Else go to step
The dice-enterprise produces unbiased ancestor variables.
For details on efﬁciency and correctness, please refer to Section 3.1.
Our proposed VSMC-PRC bound is constructed through a marginal likelihood estimator obtained by combining the SMC sampler with a PRC step and dice-enterprise .
The variance of estimators obtained through SMC-PRC particle ﬁlter is usually low (Kudlicka et al., 2020; Peters et al., 2012).
Therefore, we expect VSMC-PRC to be a tighter bound compared to the standard SMC based bounds used in recent works (Maddison et al., 2017; Naesseth et al., 2017; Le et al., 2017).
Algorithm (1) summarizes the generative process to simulate the VSMC-PRC bound.
P ARTIAL R EJECTION C ONTROL BASED VI FOR S EQUENTIAL L ATENT V ARIABLE M ODELS
We now show how to leverage PRC to develop a robust VI framework for sequential latent variable models.
Our framework is based on the VSMC-PRC bound presented below.
The complete sampling distribution of Algorithm (1) is as follows.
1: Required : N , K , and M 2: for t ∈ { 1 , 2 , . .
, T } do 3: for i ∈ { 1 , 2 , . . .
, N } do 4:
z it , c it , (cid:101) w it ∼ PRC ( q, p, M ( i, t − 1)) 5:
z i 1: t = ( z A it − 1 1: t − 1 , z it ) 6: end for 7: for i ∈ { 1 , 2 , . . .
, N } do 8:
A it = DICE-ENT (cid:0) { c it , z i 1: t } Ni =1 (cid:1) 9: end for 10: end for 11: return log (cid:81)
Tt =1 (cid:16) 1 N (cid:80)
Ni =1 (cid:101)
w it (cid:17) 12: 13: PRC ( q, p, M ( i, t − 1)) 14: while sample not accepted do 15:
Generate z it ∼ q φ ( z t | x 1: t , z
A it − 1 1: t − 1 ) 16:
Accept z it with probability a θ,φ ( z it | z
A it − 1 1: t − 1 , x 1: t ) 17: end while 18: Sample { δ i,kt }
Kk =1 ∼ q φ ( z t | x 1: t , z
A it − 1 1: t − 1 ) 19: Calculate (cid:101) w it from (6) 20:
Calculate c it from (4) 21: return (cid:0)
z it , c it , (cid:101) w it (cid:1) 22: 23: DICE-ENT (cid:0) { c it , z i 1: t } Ni =1 (cid:1) 24: Sample C ∼ Multinoulli (cid:18) c
it (cid:80) Nj =
c jt (cid:19) N i =1 25: if C ==
i then 26:
[Equation] [Equation]
28: end if 29: if U i < a θ,φ ( z it | z
A it − 1 1: t − 1 , x 1: t ) then 30: return ( i ) 31: else 32: return DICE-ENT (cid:0) { c it , z i 1: t [Equation]
Ni [Equation] 1 (cid:1) 33: end if [Equation]
The normalization constants Z ( . ) in Eq.
9 are intractable and have to be estimated while calculating the weights.
Therefore, we introduce an extra parameter K , denoting the number of Monte-Carlo samples used to estimate Z ( . ) .
The Monte-Carlo estimator of VSMC-PRC bound is
[Equation]
We maximize the VSMC-PRC bound with respect to model parameters θ and variational parameters φ .
This requires estimating the gradient the details of which are provided in Section (3.2).
3. T HEORETICAL P ROPERTIES
We now present properties of the Monte-Carlo estimator ˆ L VSMC-PRC .
The key variables that affect this bound are N (number of samples), hyper-parameter M , and the number of Monte-Carlo samples used to compute the normalization constant Z ( . ) , i.e., K .
As discussed by Bérard et al.
Naesseth et al.
(2017), as N increases, we expect the VSMC-PRC bound to get tighter.
Hence, we will focus our attention on M and K .
All the proofs can be found in the appendix.
Proposition 1.
Dice-enterprise produces unbiased ancestor variables.
Further, let Λ t be the number of iterations required for generating one ancestor variable, then Λ t ∼ Geom (cid:0) E [Λ t ] − 1 (cid:1) where
[Equation]
As evident from Proposition (1), the computational efﬁciency of the dice-enterprise clearly relies on the normalization constant Z ( . ) .
Note that the value of Z ( . ) could be interpreted as the average acceptance rate of PRC which depends on the hyper-parameter M ( i, t − 1) .
If the average acceptance rate for PRC for all particles is γ , then we can express the expected number of iterations as E [Λ it ]
= γ − 1 .
Therefore, the computational efﬁciency of dice-enterprise is similar to the PRC step and depends crucially on the hyper-parameter M .
Proposition 2.
For all K , exp( ˆ L VSMC-PRC ) is unbiased, i.e., E (cid:104)
exp
( ˆ L VSMC-PRC ) (cid:105) =
p θ ( x 1: T ) .
Further, E [ ˆ L VSMC-PRC ] is non-decreasing in K .
The use of Monte-Carlo estimator in place of the true value of Z ( . ) creates an inefﬁciency, as depicted by Proposition (2).
The bound monotonically increases as we increase K despite the use of resampling operation.
It is important to note that Algorithm (1) produces an unbiased estimator of the marginal likelihood for all values of K .
Proposition 3.
Let the sampling distribution of the i th particle (generated via PRC) at time t be r θ,φ ( z t
| z
A it − 1 1: t − 1 , x 1: t ) , then
[Equation]
Proposition (3) implies that the use of the accept-reject mechanism within SMC reﬁnes the sampling distribution.
Instead of accepting all samples, the PRC step ensures that only high-quality samples are accepted, leading to a tighter bound for VSMC in general (not always).
We show in the appendix that when M ( i, t − 1) → ∞ , the PRC step reduces to pure rejection sampling (Robert & Casella, 2013).
On the other hand, M ( i, t − 1) → 0 implies that all samples are accepted from the proposal.
Recall, M ( i, t − 1) is a hyperparameter that can be tuned to control the acceptance rate.
For more details on tuning M , see Section (3.3).
3. G RADIENT E STIMATION
For tuning the variational parameters, we use stochastic optimization.
Algorithm (1) produces the marginal likelihood estimator by sequentially sampling the particles, ancestor variables, and particles for the normalization constant ( z 1: N 1: T , A 1: N 1: T − 1 , δ 1: N, 1: K 1: T ) .
When the variational distribution q φ ( . ) is reparameterizable, we can make the sampling of δ i,kt independent of the model and variational parameters.
However, the generated particles z it are not reparametrizable due to the PRC step.
Finally, the ancestor variables are discrete and, therefore, cannot be reparameterized.
The complete gradient can be divided into three core components (assuming q φ ( . ) is reparametrizable):
[Equation]
Note that g PRC and g RSAMP denote the score gradient of PRC and resampling step, respectively.
Due to high variance, we have ignored these terms for the optimization.
We have derived the full gradient and explored the gradient variance issues in the appendix.
Please see Figure (2) (
3. L EARNING THE M MATRIX
We use M as a hyperparameter for the PRC step which controls the acceptance rate of the sampler.
The basic scheme of tuning M is as follows:
• Deﬁne a new random variable F ( z t +1 | z
A it 1:
[Equation]
x
[Equation]
• Draw z jt +1 ∼ q φ ( z t +1 | x 1: t +1 ,
z
A it 1: t ) for j = 1 , 2 , . . .
, J . • Evaluate γ ∈ [0 , 1] quantile value of { F ( z jt +1 |
z A it t , x 1: t +1 ) }
Jj =1 .
In general for this case the acceptance rate would be around γ for all particles.
[Equation]
• If M matrix is very large then use a common { M ( ., t ) }
Tt =1 for every time-step.
In general, for this conﬁguration, the acceptance rate would be greater than equal to γ for all particles:
[Equation]
Through γ , we can directly control the acceptance rate.
For example, if γ = 0 .
5 then the acceptance rate would be around (greater than) 50% for (Eq. 13) (see Eq.
Note that a similar scheme was employed in Grover et al.
We update {{ M ( i, t − 1) } Ni =1 } Tt =1 dynamically once every F epochs to save time.
To learn more on setting hyper-parameter M , see Liu et al.
; Peters et al.
R ELATED W ORK AND S PECIAL C ASES
There is signiﬁcant recent interest in developing more expressive variational posteriors for latent variable models.
There are two basic schemes for constructing tighter bounds on the log marginal likelihood: sampling-based methods (MCMC, rejection sampling)
(Salimans et al., 2015; Ruiz &
Titsias, 2019; Hoffman, 2017; Grover et al., 2018) or multiple samples from VI distributions to increase the ﬂexibility (IS, SMC) (Burda et al., 2015; Maddison et al., 2017; Lawson et al., 2018;
Naesseth et al., 2015).
In this work, we present a uniﬁed framework for combining these two approaches, utilizing the best of both worlds.
Although applying sampling-based methods on VI is useful, the density ratio between the true posterior and the improved density is often intractable.
Therefore, we cannot take advantage of variance-reducing schemes like resampling, which is crucial for sequential models.
We solve this issue through dice-enterprise : an extension of the Bernoulli factory .
Recently, Bernoulli factory has amassed a great interest in the area of Bayesian inference (Gonçalves et al., 2017a;b; Vats et al., 2020).
Although Bernoulli factory is theoretically valuable, its applicability is severely limited due to a high rejection rate.
In this paper, we have presented an approach that combines SMC with dice-enterprise in a practically meaningful manner.
A closely related work is Schmon et al.
(2019), which also uses Bernoulli factory to implement unbiased resampling.
However, the work is mostly focused on partially observed diffusions and likely limited to lowdimensional models (unlike our approach).
Another closely related work is Kudlicka et al.
In contrast to our approach, this method samples one additional particle and keeps track of the number of steps required by PRC for every time-step to construct an unbiased estimator for SMC-PRC. The weights are tractable for Kudlicka et al. (2020) as they don’t take into account the effect of normalization constant Z ( . ) . On the other hand, we consider the effect of Z ( . ) on particle’s weight, making resampling operation unfeasible.
To ﬁx this intractability, we use dice-enterprise.
Although the comparison of VSMC-PRC with Kudlicka et al.
(2020) is interesting, it will distract from the main point of our paper, which is focused on improving the VI bound.
A key distinction of our approach is to apply Bernoulli factory to general settings like VRNN (Chung et al., 2015).
In contrast to our approach existing Bernoulli factory based algorithms are only applicable to niche one-dimensional toy examples/diffusion models (Flegal et al., 2012; Schmon et al., 2019).
To provide more clarity, we will consider some special cases of VSMC-PRC bound and relate it with existing work: For the special case of N = 1 and T = 1 , our method reduces to variational rejection [Figure] sampling (VRS) (Grover et al., 2018).
For N, T > 1 , if we remove the PRC step, our bound reduces to FIVO (Maddison et al., 2017).
Finally, if we remove both the PRC step and resampling, then our method effectively reduces to IWAE (Burda et al., 2015).
Please refer to Figure (1) for more details.
E XPERIMENTS
In this section, we evaluate our proposed algorithm on synthetic as well as real-world datasets and compare them with relevant baselines.
For the synthetic data experiment, we implement our method on a Gaussian SSM and compare our approach with VSMC (Naesseth et al., 2017).
For the real data experiment, we train a VRNN (Chung et al., 2015) on the polyphonic music dataset.
5. G AUSSIAN S TATE S PACE M ODEL
In this experiment, we study the linear Gaussian state space model.
Consider the model
[Equation]
where e z , e x ∼ N (0 , I ) and z 0 = 0 .
We are interested in learning a good proposal for the above model.
The latent variable is denoted by z t and the observed data by x t .
Let the dimension of z t be d z and dimension of x t be d x .
The matrix A has the elements ( A ) i,j = α | i − j | +1 , for α = 0 .
We explore different settings of d z , d x , and matrix C .
A sparse version of C matrix measures the ﬁrst d x components of z t , on the other hand a dense version of C is normally distributed i.e
C i,j ∼ N (0 , 1) .
We consider four different conﬁgurations for the experiment.
For more details please refer to Figure (2).
The variational distribution is a multivariate Gaussian with unknown mean vector µ = { µ d } d z d =1 and diagonal covariance matrix { log σ 2 d } d z d =1 .
We set N = 4 and T = 10 for all the cases:
[Equation]
The {{ M ( i, t − 1) } Ni =1 } Tt =1 matrix (see (13)) for approximate rejection sampling is updated once every 10 epochs with acceptance rate γ ∈ { 0 . 8 , 0 . 4 } .
For estimating the intractable normalization constants, we generate K = 3 samples.
Figure (2): ( left ) compares the convergence of biased gradient vs unbiased gradients.
Note that we get a much tighter bound as compared to VSMC (Naesseth et al., 2017).
5. V ARIATIONAL RNN
VRNN (Chung et al., 2015) comprises of three core components: the observation x t , stochastic latent state z t , and a deterministic hidden state h t ( z t − 1 , x t − 1 , h t − 1 ) , which is modeled through a RNN.
For the experiments, we use a single-layer LSTM for modeling the hidden state.
The conditional distributions p t ( z t | . ) and q t ( z t | . ) are assumed to be factorized Gaussians, parametrized by a single layer neural net.
The output distribution g t ( x t | . ) depends on the dataset.
For a fair comparison, we use the same model setting as employed in FIVO (Maddison et al., 2017).
We evaluate our model on four polyphonic music datasets: Nottingham , JSB chorales , Musedata , and
Piano-midi.de .
Each observation x t is represented as a binary vector of 88 dimensions.
Therefore, we model the observation distribution g t ( x t | . ) by a set of 88 factorized Bernoulli variables.
We split all four datasets into the standard train, validation, and test sets.
For tuning the learning rate, we use the validation test set.
For a fair comparison, we use the same learning rate and iterations for all the models.
Let the dimension of hidden state (learned by single layer LSTM) be d h and dimension of latent variable be d z .
We choose the setting d z = d h = 64 for all the data-sets except JSB.
For modeling JSB, we use d z = d h = 32 .
For VSMC-PRC we have considered N ∈ { 4 , 6 } Further, for each N , we consider four settings ( K, γ ) ∈ { (1 , 0 . 9) , (1 , 0 .
The M hyper-parameter for PRC step is learned from (14) due to large size.
We have updated M value once every 50 epoches.
Note that in this scenario, the acceptance rate for all particles would be greater than equal to γ .
For more details on experiments, please refer to the appendix.
As discussed in Section (3.1) [Table] the PRC step and dice-enterprise have time complexity O
( N/γ )
for producing N samples (assuming average acceptance rate γ ).
Therefore, we consider (cid:100)
Nγ − 1 (cid:101) particles for IWAE and FIVO to ensure effectively the same number of particles, where N ∈ { 4 , 6 } and γ = 0 . 8 .
Note, however, that the acceptance rate is ≥ γ , so this adjustment actually favors the other approaches more.
For FIVO, we perform resampling when ESS falls below N/ 2 .
Table (1) summarizes the results which show whether rejecting samples provide us with any beneﬁt or not [Table] and as the results show, our approach, even with the aforementioned adjustment, outperforms the other approaches in terms of test log-likelihoods, while still having a similar computational cost [Table]
[Table]
In Sec.
(3.1), we discussed the effect of K and PRC rejection rate on VSMC-PRC bound.
We expect a performance improvement when K and the rejection rate is increased.
Although the results for
VSMC-PRC’s different conﬁgurations are almost the same, we still get the best average ranking for ( K = 3 , γ = 0 .
Overall, for most cases, VSMC-PRC bound performs better than FIVO (Maddison et al., 2017) and IWAE (Burda et al., 2015) for a variety of conﬁgurations.
In VSMC-PRC, improvement in the bound value comes at the cost of estimating the normalization constant Z ( . ) , i.e., K .
On further inspection, we can clearly see that increasing K doesn’t provide us with any substantial beneﬁts despite the increase in computational cost.
Therefore, to maintain the computational trade-off ( K = 1 , γ > 0 . 8 ) seems to be a reasonable choice for VI practitioners.
Table (1) signiﬁes that rejecting samples with low importance weight is better instead of keeping a large number of particles (at least for a reasonably high acceptance rate γ ).
The proposed bound uses more particles (PRC step and dice-enterprise ) than existing approaches like FIVO and IWAE due to intractability.
Future work aims at designing a scalable implementation for VSMC-PRC bound that consumes fewer particles
C ONCLUSION
We introduced VSMC-PRC, a novel bound that combines SMC and partial rejection sampling with
VI in a synergistic manner.
This results in a robust VI procedure for sequential latent variable models.
Instead of using standard sampling algorithms, we have employed a partial sampling scheme suitable for high dimensional sequences.
Our experimental results clearly demonstrate that
VSMC-PRC outperforms existing bounds like IWAE (Burda et al., 2015) and standard particle ﬁlter bounds (Maddison et al., 2017; Naesseth et al., 2017; Le et al., 2017).
The future work aims to explore partial versions of powerful sampling algorithms like Hamiltonian Monte Carlo (Neal et al., 2011) instead of rejection sampling.
A P ROOF OF T HEORETICAL R ESULTS
Proof of proposition (1) : Dice-enterprise produces unbiased ancestor variables.
Let’s evaluate the probability that Dice-enterprise would output i as the ancestor index.
Assume that the algorithm terminates after r steps where r ∈ { 1 , 2 , . . .
, ∞} , then the probability of output i is given as follows:
[Equation]
It is easy to see that Λ t is geometrically distributed with success probability given by
[Equation]
Proof of proposition (2) :
Before explaining the proof
[Equation]
we will ﬁrst introduce (cid:98) Z : Monte-Carlo estimator of the unknown normalization constant Z ( . )
Since we are using K samples
[Equation] [Equation] Algorithm 1 is producing an unbiased estimator of the marginal likelihood.
We will ﬁrst integrate out δ 1: N, 1: K 1: [Equation] from the marginal likelihood estimator
[Equation]
(cid:90)
T (cid:89) t =
N
N (cid:88) i = p θ ( x t , z it | x 1: t − 1 , z
A it − 1 1: t − 1 )
Z ( x 1: t , z
A it − 1 1: t − 1 ) q φ ( z it | x 1: t ,
z A it − 1 1: t − 1 ) a θ,φ ( z it | x 1: t , z
A it − 1 1: t − 1 )
Q VSMC
[Equation]
PRC (cid:0) z 1: N 1:
[Equation]
, A 1: N 1: T − 1 (cid:1) dz 1: N 1: T = (cid:88)
A 1: N 1: T −(cid:90)
T (cid:89) t =
N
N (cid:88) i = p θ ( x t , z it | x 1: t − 1 , z
A it − 1 1: t − 1 ) r θ,φ ( z it | x 1: t , z
A it − 1 1: t − 1 )
Q VSMC-PRC (cid:0) z 1: N 1: T , A 1: N 1: T − 1 (cid:1) dz 1: N 1: T = (cid:88)
A 1: N 1: T −(cid:90)
T (cid:89) t =
N
N (cid:88) i =
w it (cid:32) N (cid:89) i =
r θ,φ ( z i 1 | x 1 )
T − 1 (cid:89) t =
N (cid:89) i =
Discrete ( A it | α t ) r θ,φ ( z it +1 | x 1: t +1 , z
A it 1: t ) (cid:33) dz 1: N 1: T = E (cid:34) T (cid:89) t =
N
N (cid:88) i =
w it (cid:35) .
Note that r θ,φ ( z it | . )
is the sampling density of PRC.
It is easy to see that equation (16) is a standard SMC estimator for the marginal likelihood p θ ( x 1: T ) .
The proof is quite standard in SMC literature and can be found in (Naesseth et al., 2017; 2019).
The key factor that makes our bound unbiased is the ability to produce unbiased ancestor samples despite the presence of intractable normalization constant.
Using Jensen inequality we can easily show that VSMC-PRC bound is smaller then log marginal likelihood. [Equation]
[Equation]
We will show that E [ ˆ L VSMC-PRC ; K ] is non-decreasing with K .
Let’s deﬁne a collection of subsets {{ I i,t } N i =1 } T t =1 ⊂ { 1 , 2 , . .
, K } having elements { i 1 , i 2 , . .
, i m } randomly drawn from the set { 1 , 2 , . . .
, K } having length m ≤ K .
We can easily show the following expectation:
[Equation]
[Equation]
Substitute the values of above expectation in equation 17.
Use Jensen’s inequality i [Equation] e E [log X ] ≤ log E [ X ] to complete the proof .
[Equation]
Now we will see what happens when the limit K → ∞ .
Using dominated convergence theorem we can write down the estimator as follows:
[Equation]
Proof of proposition (3) : We will show that PRC step reﬁnes the learned distribution
[Equation]
First
[Equation]
we will use the property of negatively correlated random variables.
Note that the following two random variables
[Equation]
are negatively correlated.
We know that for negatively correlated variables following identity holds E
[ XY ] ≤ E [ X ] E
[ Y ] .
Further we have used Jensen’s inequality to show that
[Equation]
In this situation all samples would be accepted.
Hence
[Equation]
we can express the sampling distribution as:
[Equation]
In this case
[Equation]
the acceptance probability would reduce to standard rejection sampling.
Therefore, the sampling distribution would become equal to the true posterior.
[Equation]
B G RADIENT E STIMATION
In this section, we will derive the unbiased gradients for the Monte-Carlo estimate E [ ˆ L VSMC-PRC ] .
Note that we can express the complete gradient into three core components (assuming q () is reparametriz- able):
[Equation]
Note that for both g PRC and g RSAMP unbiased score gradient estimates are not available due to intractability.
Therefore, we have used Monte-Carlo samples to estimate the log density for ﬁgure (3).
C E XPERIMENTAL S ETUP
For the real data experiment, we train a VRNN (Chung et al., 2015) on the polyphonic music dataset.
Polyphonic music comprises of four datasets: : Nottingham, JSB chorales, Musedata, and
Piano-midi.de.
Each dataset was divided into standard train, validation, and test datset.
The validation data was used to tune the learning rate: we picked the learning rate from the following set: { 3 × 10 − 4 , 1 × 10 − 4 , 3 × 10 − 5 , 1 × 10 − 5 } , instead of optimizing for each method, we picked the learning rate at which FIVO (Maddison et al., 2017) validation performance is the best.
Once the learning rate is decided, we ran every method for the same number of iterations to ensure uniformity.
We use a single-layer LSTM for modeling the hidden state having dimension d h .
For a length T sequence, the variational distribution and joint data likelihood are deﬁned as follows
[Equation]
Note that a t ( z t | . ) is the acceptance probability for the PRC step and Z t ( . ) is the intractable normalization constant.
Similarly, we can write down the joint data likelihood as
[Equation]
The conditional distributions p t ( z t | . ) and q t ( z t | . ) are assumed to be factorized Gaussians, where dimension of the latent variable z t is d z .
Note that the conditional densities are parametrized by fully connected neural networks with a single layer having size d h .
The output distribution g t ( x t | . ) is modeled by a set of 88 iid Bernoulli variables.
Please see Table (2) for more details regarding implementation details.
The unknown weights and biases are initialized using a Xavier initialization.
For setting up the optimization we used a batch size of 4 with adam optimizer.
The unknown hyperparameter M is updated once every 50 epoches through (14) to save time.
[Table]