BR-SNIS: Bias Reduced Self-Normalized Importance Sampling
Anonymous Author(s)
Abstract
Importance Sampling (IS) is a method for approximating expectations under atarget distribution using independent samples from a proposal distribution andthe associated importance weights.
In many applications, the target distributionis known only up to a normalization constant, in which case self-normalized IS(SNIS) can be used.
While the use of self-normalization can have a positive effecton the dispersion of the estimator, it introduces bias.
In this work, we propose a newmethod, BR-SNIS, whose complexity is essentially the same as that of SNIS andwhich signiﬁcantly reduces bias without increasing the variance.
This method is awrapper in the sense that it uses the same proposal samples and importance weightsas SNIS, but makes clever use of iterated sampling–importance resampling (i-SIR)to form a bias-reduced version of the estimator.
We furnish the proposed algorithmwith rigorous theoretical results, including new bias, variance and high-probabilitybounds, and these are illustrated by numerical examples.
Introduction
Background and
previous
work:
Importance sampling [15, 1] (IS) is a classical Monte Carlotechnique for estimating expectations under some given probability distribution (the target ) on thebasis of a sample of draws from a different distribution (the proposal ).
In the modern era of artiﬁcialintelligence and statistical machine learning, characterized by large computational resources and
Bayesian inference, IS technologies are enjoying a revival; see, e g. , [34, 20] and [11] for a recentsurvey.
The method is not only relevant to situations where sampling from the target is intractable; itcan also be used to achieve variance reduction [21].
When the proposal is dominating the target—in the sense that the support of the latter is contained in the support of the former—unbiasedestimation can be achieved by assigning each draw an importance weight given by the likelihoodratio between the target and the proposal.
In the very common case where the target is known onlyup to a normalizing constant, consistent estimation can still be achieved by simply normalizing eachimportance weight by the total weight of the sample; however, since such self-normalized importancesampling (SNIS) involves ratios of random variables, the procedure can only be implemented at thecost of bias, which can be signiﬁcant in some applications.
More precisely, let ( X , X ) be some state space and π (d x ) ∝ w ( x ) λ (d x ) a given target probabilitydistribution, where w and λ are a positive weight function and a proposal probability distributionon ( X , X ) , respectively, such that the normalizing constant λ ( w ) =
(cid:82) w ( x ) λ (d x ) (this will be ourgeneric notation for Lebesgue integrals) of π is ﬁnite.
The SNIS estimator is given by
[Equation]
where X 1: M = ( X 1 , . .
, X M ) are independent draws from λ , and can be used to approximateπ ( f )
= (cid:82) f
( x ) π (d x ) for any test function f such that π ( | f | ) < ∞ .
The estimator (1) can becalculated without knowledge of the normalizing constant λ ( w ) , which is intractable in general.
The SNIS estimator is known to be biased; provided that λ ( w 2 ) < ∞ , the bias and mean-squarederror (MSE) of the SNIS estimator (1) over bounded test functions f satisfying (cid:107) f (cid:107) ∞ ≤ 1 are givenrespectively (see [1, Theorem 2.1]) by
[Equation]
where κ [ π , λ ] = λ ( w 2 ) /λ 2 ( w ) .
Although IS is primarily intended to approximate integrals in theform π ( f ) , it can also be used to generate unweighted samples being approximately distributedaccording to π .
In this paper, we consider iterated sampling importance resampling (i-SIR), proposedin [42]; see [4, 24, 23, 5].
The i-SIR can be seen as an iterative application of the sampling importanceresampling (SISR) algorithm proposed by [37]; the k -th iteration is deﬁned as follows.
Given a state
Y k ∈ X , (i) set X 1 k +
=
Y k and draw X 2: N k +1 independently from the proposal distribution λ ; (ii) compute, for i ∈ { 1, . . .
, N } , the normalized importance weights ω iN , k +
=
w ( X ik +1 ) / (cid:80) N(cid:96)
=1 w ( X (cid:96)k +1 ) ;(iii) select Y k
+1 from the set X 1: N k +1 by choosing X ik +1 with probability ω iN , k +1 .
In the following,
Y k +1 and X 1: N k +1 will be referred to as the state and the candidate pool , respectively.
Following [42](see Section 2.1), i-SIR may be viewed (up to an irrelevant permutation of the samples) as a two-stage
Gibbs sampler targeting an extended probability distribution ϕ N on an enlarged state space includingthe state as well as the candidate pool.
As this extended distribution allows π as a marginal withrespect to the state, one can expect the marginal distribution of the generated states ( Y k ) k ∈ N , formingthemselves a Markov chain, to approach the target π of interest as k tends to inﬁnity.
This paper:
In i-SIR, the only function of the candidate pool is to guide the states selected at stage(iii) towards the target.
Thus, since all rejected candidates are discarded, the approach results generallyin a large waste of computational work.
Thus, in the present paper we propose to recycle all thegenerated samples by incorporating all the proposed candidates X 1: N k into the estimator rather thanonly the selected candidate Y k .
We proceed in three steps.
First, we show that under the stationarydistribution ϕ N of the process ( Y k , X 1: N k ) k ∈ N generated by i-SIR, the expectation of Π N f
( X 1: N k )(given by (1)) equals π ( f ) for every valid test function f
(see Theorem 2).
Second, we establishthat since i-SIR is nothing but a systematic-scan Gibbs sampler, the two processes ( X 1: N k ) k ∈ N and 60 ( Y k ) k ∈ N are interleaving (see Theorem 5); thus, if ( Y k ) k ∈ N is uniformly geometrically ergodic, sois ( X 1: N k )
k ∈ N with the same mixing rate κ N .
Third, as the main result of the present paper, weestablish a novel O ( κ kN /N )
bound on the bias of the estimator Π N f
( X 1: N k ) (see Theorem 3), wherethe exponentially diminishing factor κ kN indicates a drastic bias reduction vis-à-vis the standard ISestimator (1) based on i.i.d.
samples.
As a consequence, approximating π ( f ) by the average of(Π N f ( X 1: N k )) k(cid:96)
=
k 0 +1 , where the “burn-in” period k 0 should be chosen proportionally to the mixingtime of the process, yields an estimator whose bias can be furnished with a bound which is, roughly,proportional to κ k 0 N and inversely proportional to the total number M = kN of samples generatedin the algorithm (see Theorem 4).
To complete the theoretical analysis of these estimators, we alsoequip the same with variance bounds.
The procedure of recycling, as described above, all the samplesgenerated in the i-SIR and to incorporate, at negligible computational cost, the same into the ﬁnalestimator, will from now on be referred as BR-SNIS.
Finally, we test numerically the proposedestimators and illustrate how a signiﬁcant bias reduction relatively to the standard i-SIR can beobtained at basically no cost.
To sum up, our contribution is twofold, since we
– propose a new algorithm, BR-SNIS, which makes better use of the available computationalresources by recycling the candidate pool generated at each iteration of i-SIR.
– furnish the proposed algorithm with rigorous theoretical results, including novel bias, variance, andhigh-probability bounds which support our claim that sample recycling may lead to drastic biasreduction without impairing the variance.
Main results2.1 Statements
The i-SIR algorithm can be interpreted as a systematic-scan two-stage Gibbs sampler, alternatelysampling from the full conditions of an extended target ϕ N on the product space of states andcandidate pools.
Once the extended target ϕ N is properly deﬁned, these full conditionals can beretrieved from a dual representation of ϕ N presented in Theorem 1.
In order to deﬁne ϕ N , we introduce the Markov kernel (see Appendix A.1 for comments)
[Equation]
on X × X (cid:15) N , which describes probabilistically the sampling operation (i) in i-SIR.
Using the kernelΛ N we may now deﬁne properly the extended target ϕ N as the probability law
[Equation]
on ( X N +1 , X (cid:15) ( N +1) ) .
Note that since for every A ∈ X , ϕ N (1 A × X N ) =
π ( A ) , the target πcoincides with the marginal of ϕ N with respect to the state.
Moreover, it is easily seen that Λ Nprovides the conditional distribution, under ϕ N , of the candidate pool given the state.
Deﬁning thekernels
[Equation]
on X × X (cid:15) N , the marginal distribution π N of ϕ N with respect to x 1: N is given by
[Equation]
It is interesting to note that the marginal π N has a probability density function, proportional toΓ N 1 X ( x 1: N )
=
(cid:80) Ni =1 w ( x i ) /N , with respect to the product measure λ (cid:15) N .
Using (4), we immediately obtain the following result.
Theorem 1 (duality of extended target) .
For every N ∈ N ∗ ,
[Equation]
Note that the second identity of the dual representation (5) provides also the conditional distribution,under ϕ N , of the state given the candidates.
Consequently, i-SIR is a systematic scan two-stage
Gibbs sampler which generates a Markov chain ( X k , Y k )
k ∈ N with time-homogeneous Markov kernel
P
[Equation]
(( y
[Equation] [Equation]
+
[Equation]
( x 1:
[Equation]
k
[Equation]
1 , d y k +1 ) on X N +1 × X (cid:15) ( N +1) .
Note that the law P N ( y k , x 1: N k , · ) does not depend on x 1: N k , which meansthat only the state Y k needs to be stored from one iteration to the other.
Thus, ( Y k ) k ∈ N is a Markovchain with Markov transition kernel
[Equation]
(where integration is w.r.t.
x 1: N k +1 ) on X × X .
Given some probability distribution ξ on( X N +1 , X (cid:15) ( N +1) ) , we denote by P ξ the law of the canonical Markov chain ( X k , Y k ) k ∈ N withkernel P N and initial distribution ξ .
Our ﬁrst results establishes the unbiasedness of the estimatorΠ N f ( X 1: N ) under ϕ N .
Theorem 2.
For every N ∈ N ∗ and π -integrable function f ,
[Equation]
The proof of Theorem 2 is postponed to Appendix A.3.
Next, we present theoretical bounds on thediscrepancy, in terms of bias, MSE and covariance, between Π N f ( X 1: N k ) and π ( f ) , for boundedtarget functions f , when the i-SIR chain is initialized according to an arbitrary distribution ξ .
Wewill work under the following assumption.
A1.
It holds that ω = (cid:107) w (cid:107) ∞ /λ ( w ) < ∞ .
Under A 1, the state and candidate-pool Markov chains ( Y k ) k ∈ N and ( X 1: N ) k ∈ N can be shown to beuniformly geometrically ergodic with mixing rate and mixing-time upper bound
[Equation]
respectively; see Theorem 6 below for details.
Here the mixing time τ mix , N grows logarithmicallywith the sample size N .
The exact value of τ mix , N is likely to be grossly pessimistic, but we conjecturethat the logarithmic dependence in the minibatch size holds true.
In addition, under A 1 we deﬁne theconstants
[Equation]
With these deﬁnitions, the following holds true.
Theorem 3.
Assume A 1.
Then for every initial distribution ξ on ( X N +1 , X (cid:15) ( N +1) ) , boundedmeasurable function f on ( X , X ) such that (cid:107) f (cid:107) ∞ ≤ 1 , N ≥ 2 , and ( k , (cid:96) ) ∈ ( N ∗ ) 2 ,(i
[Equation]
(cid:12)(cid:12)
[Equation]
[Equation] [Equation]
where constants are given in (7) and (8) .
It is worth noting that the bias decreases inversely with the number of candidates and exponentiallywith the number of iterations (the mixing time of the chain also depends on N ).
The MSE is alsoinversely proportional to the number of candidates N .
In the light of the previous results, it isnatural to consider an estimator formed by an average across the IS estimators (Π N f
( X 1: N k ))
k ∈ N 130 associated with the candidate pools generated at the different i-SIR iterations.
To mitigate the bias,we remove a “burn-in” period whose length k 0 should be chosen proportional to the mixing timeτ mix , N of the Markov chain ( Y k ) k ∈ N (which turns out to coincide with that or the chain ( X 1: N k ) k ∈ N ; 133 see Section 2.2).
This yields the estimator
[Equation]
of π ( f ) .
The total number of samples (generated by the proposal λ ) underlying this estimator is
M = ( N − 1) k .
Importantly, all the importance weights included in the estimators are obtainedas a by-product of the i-SIR schedule; thus, it is, for a given budget of simulations ( i.e. , under theconstraint that ( k − k 0 )
N is constant), possible to compute Π ( k 0 , k ), N ( f ) for different values of k 0 , kand N with a negligible computational cost.
We denote by υ =
( k − k 0 )
/k
the ratio of the numberof candidate pools used in the estimator to the total number of sampled such pools.
Note that thistype of estimator was already suggested by [43] and also appears in [39].
Our ﬁnal main result provides bounds on the bias and the MSE of the estimator (9) as well as ahigh-probability bound for the same.
Deﬁne ζ bias = 4 τ mix , N ς bias / 3 , ζ mse
i
[Equation]
∧
[Equation]
1 { 0,2 } ( i ) +(8 / 3) τ mix , N ς cov i , i ∈ { 0, 1, 2 } , ζ mse = ζ mse+ ζ mse 1 ( N − 1) − 1 / 4 + ζ mse 2 ( N − 1) − 1 , and MSE is M =(4 /M )
κ
[ π , λ ] , see (2).
Theorem 4.
Assume A 1.
Then the following holds true for every initial distribution ξ on( X N +1 , X (cid:15) ( N +1) ) , bounded measurable function f on ( X , X ) such that (cid:107) f (cid:107) ∞ ≤ 1 , and N ≥ 2 .
[Table]
[Table]
[Table]
[Table] [Table] [Table]
[Table]
[Table]
(ii) E ξ [ { Π ( k , k ), N ( f ) − π ( f ) } 2 ] ≤ MSE is υM + ζ mse ( υM ) − 1 ( N − 1)
(iii)
For every δ ∈ (0, 1) , | Π ( k 0 , k ), N ( f ) − π ( f ) | ≤ ς hpd ( υM ) − 1 / 2 (log(4 /δ )) 1 / 2 with probability
at least 1 − δ , where ς hpd =
664 ω .
Bootstrap: As established in Theorem 4, the bias of the BR-SNIS estimator decreases exponentiallywith the burn-in period k 0 , leading to potentially signiﬁcant bias reduction with respect to SNIS.
Still,using a large k 0 is done at a price of increased overall MSE (mainly through the term MSE is υM in
Theorem 4(ii), which is directly related to k 0 via υ ).
A natural way to reduce the variance is to usebootstrap.
More precisely, we ﬁrst apply a random permutation to the samples and re-compute BRSNIS on the basis of the bootstrapped samples.
After this, we produce a ﬁnal estimator by averagingover the bootstrapped BR-SNIS replicates.
In most applications, the major computational bottleneckconsists of sampling from λ and evaluating w and f at the samples; thus, the additional operationsthat this bootstrap approach entails are computationally cheap.
Therefore, in our experiments, we usebootstrap in combination with the choice k
=
k − 1 (in order to minimize the bound in Theorem 4(i)).
2.2 Elements of proofs
Ergodic properties of i-SIR:
The systematic scan two-stage Gibbs sampler is a well-studied
MCMC algorithmic structure, and we summarize its most important properties in Theorem 5 below;see [27, 3] and [36, Chapter 9] as well as the references therein.
In particular, as shown in [27], thestate and candidate-pool Markov chains ( Y k ) k ∈ N and ( X 1: N k ) k ∈ N satisfy a duality property referredto as interleaving (Theorem 5(iii)).
Theorem 5.
Assume that for every x ∈ X , w ( x ) > 0 , λ ( w ) < ∞ and that there exists a set C ∈ Xsuch that λ ( C ) > 0 and sup x ∈ C w ( x ) /λ ( w ) < ∞ .
Then,
(i) the Markov kernel P N is Harris recurrent and ergodic with unique invariant distribution ϕ N .
(ii) the Markov kernel P N is π -reversible, Harris recurrent and ergodic.
(iii) the two Markov chains ( Y k ) k ∈ N and ( X 1: N k ) k ∈ N are conjugate of each other with the interleaving property,
i
e., for every initial distribution ξ and k ∈ N , under P ξ ,(a) X 1: N k and X 1: N k +1 are conditionally independent given Y k ,
(b) Y k and Y k +1 are conditionally independent given X 1: N k +1 ;
(c) moreover, under P ϕ N , ( Y k , X 1: N k − 1 ) and ( Y k , X 1: N k ) are identically distributed.
The ergodic behavior of the i-SIR algorithm has been studied in many works; see [23, 25, 5] inparticular.
The analysis is particularly simple under the assumption that the importance weightfunction w is bounded, as imposed by A 1.
Recall that the total variation-distance between twoprobability measures ξ and ξ (cid:48) on ( X , X ) is given by d TV ( ξ , ξ (cid:48) )
=
sup g :osc( g ) ≤ 1 { ξ ( g ) − ξ (cid:48) ( g ) } , whereosc( g ) =
sup ( x , x (cid:48) )
∈
X 2 | g ( x ) − g ( x (cid:48) ) | denotes the oscillator norm of a measurable function g .
Thefollowing result establishes the uniform geometric ergodicity of the state chain ( Y k ) k ∈ N .
Theorem 6.
Assume A 1.
Then for every N ≥ 2 , y ∈ X and k ∈ N , d TV ( P kN ( y , · ), π ) ≤ κ kN , whereκ N is given in (7) .
The proof is given in [25, 5], but we provide it in Appendix A.5 for completeness.
For uniformlyergodic Markov chains, it is often more appropriate to work with the mixing timemin { k ∈ N : sup y ∈
X d TV ( P kN ( y , · ), π ) ≤ 1 / 4 } ≤ τ mix , N (where τ mix , N is given in (7)), i.e. , the number of time steps required for the distribution of the chainto be within a certain total variation distance from its stationary distribution [2, 14].
An interestingconsequence of the interleaving property is that if the Markov chain ( Y k ) k ∈ N is (geometrically)ergodic, then the Markov chain ( X 1: N k ) k ∈ N is (geometrically) ergodic as well with the same mixingtime; see [36, Corollary 9.14]).
Bias of the BR-SNIS estimator:
As the BR-SNIS estimator Π N f ( X 1: N k ) (where Π N is deﬁnedin (3)) is made up by a ratio of the two unnormalized estimators Γ N f ( X 1: N k ) and Γ N 1 X
( X 1: N k ) , akey ingredient in the proof of Theorem 3 is to bound the bias and the p th order moments of statisticsdeﬁned as ratios of sums of random variables that are not necessarily independent.
The basic ideais to reduce the study of these relations to the analysis of the moments of the numerator and thedenominator of these statistics and to exploit their concentration around the respective (conditionaland unconditional) means.
The main results that we will use in the rest of the paper are summarizedin Appendix B.
Lemma 7.
For every initial distribution ξ on ( X N +1 , X (cid:15) ( N +1) ) , k ∈ N ∗ , and bounded measurablefunction f : X → R , it holds that
(i) for every y ∈ X , Λ N Γ N f ( y ) =
(1 − 1 /N )
λ ( wf )
+ (1 /N )
w ( y ) f
( y ) ,
(ii) E ξ (cid:2) Γ N f
( X 1: N k ) (cid:12)(cid:12)
[Equation]
[Equation]
We now have all the elements that allow us to determine the ﬁrst important result of this work, namelythe bias and the MSE of the estimator Π
N f
( X 1: N k ) of π ( f ) .
Proof of Theorem 3.
We establish the bias bound in (i) and postpone the proof of the bounds on the
MSE and the covariance in (ii) and (iii) to the supplement.
Deﬁne the measure ξ ( A ) =
ξ ( A × X ) , A ∈
X , and the kernel P N = Λ N Π N on X × X .
Consequently, P N f ( Y k − 1 ) =
E ξ [Π N f
( X 1: N k ) | Y k − 1 ] and Λ N Γ N f ( Y k − 1 ) =
E ξ [Γ N f ( X 1: N k ) | Y k − 1 ] , P ξ -a.s.
Since ( Y k ) k ∈ N is, under P ξ , a Markov chain with initial distribution ξ and Markov kernel P N (see (6)), it holds that
[Equation]
Consequently, the proof is concluded by establishing that for every k ∈ N ∗ ,
[Equation]
On the other hand, since by Theorem 2, π ( P N f ) =
π ( f ) , we may use Theorem 6 to obtain the bound
[Equation]
Finally, we establish (10) by bounding osc( P N f ) .
Note that
[Equation]
where, for every y ∈ X , using Theorem 11,
[Equation]
Now, since Λ N Γ N 1 X ( y ) ≥ (1 − 1 /N ) λ ( w )
[Equation]
we get, using Lemma 7,
[Equation]
On the other hand, using the elementary inequality a/b − c/d = a
[Equation]
d −
[Equation]
/bd
[Equation]
, we get, as 212 π ( f )
=
λ ( wf ) /λ
( w ) ,
[Equation]
wf ) } /λ ( w ).
Finally, the bound (10) is established by noting that
[Equation] [Equation]
215 2.3 Related works
The ﬁrst use of the IS method, then as a variance reduction technique, dates back to the ’50s; see[12, 19] and the references therein.
Today, the renewed interest in IS parallels the ﬂurry of activity inthe probabilistic ML community and its ever-increasing computational demands; thus, it is impossibleto fully present the literature.
We therefore limit ourselves to describing results that have inspired ourwork, and refer the readers to the recent reviews [1, 11] for additional references.
There is clearly a plethora of modern ML applications where the standard SNIS estimator may besubstantially improved using the BR-SNIS method.
To mention just a selection of examples, SNISplays a key role for a robust off-policy selection strategy BY [20] (extending [40, 29]),
Bayesianproblems (see, e.g. , [1, Section 3]), Bayesian transfer learning [16, 28], variational autoencoders [9],inference of energy-based models [22], patch-based image restoration [34] and many more.
Despite long-standing interest in SNIS, there are only few theoretical results.
For example, [1,
Theorem 2.1] provides bounds on the bias and variance of SNIS, results that we extend to BR-SNISin Theorem 3.
Moreover, [29, Proposition D.3] provides a suboptimal variance bound based on abound for the second-order moment.
This result can be compared to the sophisticated sub-Gaussianconcentration bound for BR-SNIS obtained in Theorem 4 (a result that can be obtained for SNIS usingthe same proof mechanism; see Appendix A.8).
Finally, [20] obtains a semi-empirical sub-Gaussianconcentration inequality using the Efron-Stein estimate of variance and the Harris inequality.
As an MCMC sampling method, the i-SIR algorithm that has been applied successfully in manysituations.
It was recently used—under the alternative name conditional importance sampling —in[31] for Markovian score climbing .
In the same work, it is mentioned that it is possible to “RaoBlackwellize” the gradient of the score using the proposed candidates, which is in line with therecycling argument underpinning the estimator suggested by us, but without theoretical justiﬁcations.
In its most basic form, the i-SIR algorithm appeared in the pioneering work of [42].
The same ideaplayed a key role in the development of the particle Gibbs sampler [4, 5, 32], which extends i-SIRprinciples to sequential Monte Carlo methods .
An approach very similar to BR-SNIS can be takenalso in this context; however, casting BR-SNIS into the framework of particle Gibbs methods is anon-trivial problem which is the subject of ongoing work.
Experimental results
In this section we compare numerically the performances of BR-SNIS and SNIS in three differentsettings: mixture of Gaussians, Bayesian logistic regression and variational autoencoders (VAE).
Weleave to the supplementary material (Appendix C.1)
the detailed numerical veriﬁcation of the boundsestablished in Section 2.
Mixture of Gaussian distributions: We start with an example where the target distribution π isa mixture of two Gaussian distributions of dimension d = 7 , as shown in Figure 2a.
The proposaldistribution is a Student distribution with ν = 3 degrees of freedom.
The test function is f = 1 A − 1 B ,where A and B are a d -dimensional rectangle intersecting each of the modes of π (see Appendix C.1for precise deﬁnitions).
We verify the positive effect of bootstrap in Figures 1a and 1b by computingthe bias and the MSE over 1000 chains for N = 129 for several k .
The purple, green, and red curvescorrespond to a number of bootstrap rounds of 1, 21 , and 201 , respectively.
We illustrate the decayof the mean Sliced Wasserstein distance (according to [7]) with k for different values of N ( N = 8purple, N = 32 green, N = 64 orange, and N = 128 red) in Figure 1c.
The decay of the Wasserteindistance is directly linked to the mixing time of the i-SIR kernel (see (7)), and hence allows us torepresent the effective mixing time of the chain.
Moreover, we represent the theoretical slopes asdashed lines.
This illustrates that the effective value of τ mix , N is smaller than its theoretical bound.
The bias and MSE for SNIS with M = 25600 are shown in black dashed lines.
We compare the bias (Figure 2b) and MSE (Figure 2c) of BR-SNIS and SNIS for a ﬁxed budgetwith a total number of M = 16384 samples.
We run the experiments 10 6 times; we compute thebias and MSE over batches of 10 4 replications using the true value of π ( f ) computed above (theboxplots in Figure 2 are therefore obtained over 100 replications).
For the algorithm BR-SNIS, weused N ∈ { 129, 513 } , k
= k max − 1 and k max = M/
( N − 1) bootstrap rounds.
As can be seen from
[Figure]
Figure
[Figure]
266 Figure 2b, BR-SNIS signiﬁcantly reduces bias (by a factor of almost 10) w.r.t.
standard SNIS for bothconﬁgurations, while MSE increases only slightly (at around 20% ), as can be seen in Figure 2c.
Wealso show in Appendix C.1 that k
= (cid:98) 0.
k max (cid:99) can lead to about 3 times less bias w.r.t.
standard
SNIS while only augmenting the MSE of 10% .
We have also compared in BR-SNIS to zero biasestimators based on SNIS such as the one described in [30].
Bayesian Logistic regression: We consider posterior inference in a Bayesian logistic regressionmodel.
Let D train =
( x i , y i )
Ti =1 be a dataset, where each x i ∈ R
d is a vector of covariates andy i ∈ {− 1, 1 } is a binary response.
Let p ( y i | x i ; θ ) = { 1 + exp
− y i x (cid:62) i θ ) } − 1 be the probability of
(a) Heart     0.
(b) Breast      0.
[Figure]
the i th observation at θ ∈ Θ ⊆ R d and π 0 (d θ ) be a prior distribution for θ .
The Bayesian posterior isgiven
[Equation]
For numerical illustration, we use the heart failure clinical records ( d = 13 , T = 299 ), breastcancer detection ( d = 30 , T = 569 ), and Covertype ( d = 55 , T = 4 · 10 4 ) datasets from the UCImachine learning repository.
For Covertype, we use Cover type 1 (Spruce/Fir) and Cover type 2(Lodgepole Pine) classes to deﬁne a binary classiﬁcation problem.
As a prior, we use a Gaussiandistribution N(0, τ − 2 I ) with τ 2 = 5 · 10 − 2 .
The importance distribution λ is Gaussian with meanand diagonal covariance learned by variational inference; see Appendix C.2 for details.
The boxplotsfor bias in Figure 3 were constructed in the same way as those in Figure 2.
We compare two test .
CoverType
Breast
Heart
[Equation]
0.0028 +/- 0.
0.00011 +/-
6.04e0.00023 +/- 7.24eBR-SNIS, M=0.0014 +/- 0.
7.9e-5 +/-
5.5e0.
+/-
6.7eSNIS, M =0.0026 +/- 0.
4.3e-5 +/-
3.3e7.8e-
+/-
6.8eBR-SNIS, M=0.0013 +/- 0.
[Table]
5e
[Table]
+/
[Table]
2e
[Table] [Table]
283 functions, f ( θ ) = θ , corresponding to evaluation of the posterior mean, and f ( θ ) =
p ( y | x , θ ) ,where ( x , y ) ∈ D test .
This last function allows us to compute a TV distance for the predictivedistribution.
Indeed, in a classiﬁcation context, one can compute the TV distance between any twopredictive distributions p and ˆ p as
[Equation]
where we compare the predictive distribution p ( y | x , D train )
= (cid:82) p ( y | x , θ ) π ( θ )
d θ and ˆ p is theestimation of this quantity, provided in the experiments by SNIS or BR-SNIS.
From Figure 3 we cansee that for each dataset we have a constant decrease in bias, while the variance increases only slightly.
We plot the bias in other components of θ and provide further numerical details in Appendix C.2.
Generative Model:
We now extend our methodology to the more complex deep latent generativemodels (DLGM).
A DLGM deﬁnes a family of probability densities p θ ( x ) over an observationspace x ∈ R P by introducing a latent variable z ∈ R d , deﬁning the joint density function p θ ( x , z )(with respect to Lebesgue measure) and aiming to ﬁnd a parameter θ maximizing the marginallog-likelihood of the model p θ ( x ) = (cid:82) p θ ( x , z )
[Equation]
z .
Under simple technical assumptions, by Fisher’sidentity,
[Equation] [Equation]
In most cases, the conditional density p θ ( z | x ) =
p θ ( x , z ) /p
θ ( x ) is intractable and can only besampled.
The variational autoencoder [18] is based on the introduction of an additional parameter φand a family of variational distributions q φ ( z | x ) .
The joint parameters { θ , φ } are then inferred bymaximizing the evidence lower bound (ELBO) deﬁned by
[Equation]
d
VAE
IWAE
BR-IWAE ( k = 4 )
BR-IWAE ( k = 8 ) -115.
This basic setup has been further developed and improved in many directions.
Here we consider theimportance weighted autoencoder (IWAE) [8], which relies on SNIS to design a tighter ELBO on thelog-likelihood.
The objective of the IWAE is given by
[Equation]
where w θ , φ , x ( z ) =
p θ ( x , z ) /q
φ ( z | x ) denote the importance weights.
However, writing, following[8, Eq.
[Equation]
where ω ( i ) θ , φ , x = w θ , φ , x ( z i ) / (cid:80) Mj
=
1 w θ , φ , x ( z j ) are normalized importance weights, yields an expression of the gradient that corresponds exactly to the biased SNIS approximation of (13).
Thus, theoptimization problem will suffer from bias.
We hence propose to use BR-SNIS for learning IWAE.
The proposed algorithm proceeds in two steps, which are repeated during the optimization (detailsare given in Appendix C.3)
• First, update the parameter φ as in the IWAE algorithm (using the reparameterization trick andfollowing the methodology of [8]) according to φ ( t +1)
=
φ ( t ) − η ∇ φ L M ( θ ( t ) , φ ( t ) ) .
• Second, update the parameter θ by estimating (13) using BR-SNIS for π ( z )
=
p θ ( x , z ) , f ( z ) =∇ θ log p θ ( x , z ) and λ ( z ) =
q φ ( z | x ) .
We refer to this model as BR-IWAE.
As an illustration, we train the model using the binarized MNISTdataset [38], where x ∈ { 0, 1 } 784 are binarized digits images in dimension 784.
For both for theencoder q φ and the decoder p θ , we use a simple fully connected architecture with 2 hidden layers(more details are given in Appendix C.3).
For a comparison, we estimate the log-likelihood using the
VAE, IWAE and BR-IWAE approaches, and the result is reported in Table 2.
All models are run for100 epochs, using the Adam optimizer [17] and a learning rate of 10 − 3 .
The complete experimentaldetails are given in Appendix C.3.
Conclusion
In this paper, we have introduced a novel method, BR-SNIS, which improves over SNIS when itcomes to producing close to unbiased estimates of expectations taken w.r.t.
to distributions knownonly up to a normalizing constant, a ubiquitous problem in machine learning and statistics.
Thehigh performance of BR-SNIS is supported theoretically by non-asymptotic bias, variance andhigh-probability bounds.
We illustrate our method on various examples, which show the practicaladvantages of BR-SNIS over SNIS.
Finally, BR-SNIS is naturally adapted to other IS based methods,for example [41], which use a Hamiltonian (gradient-based) transform [33] as part of the IS proposal.
The extension of BR-SNIS to [41] would produce an Hamiltonian based sampler able to recycleall samples, contrarily to other classical Hamiltonian-based methods [33, 13].
BR-SNIS can alsobe extended to Particle Markov chain Monte Carlo methods such as Particle Gibbs with Ancestorsampling [26].
A ProofsA.1 i-SIR Algorithm
We analyze a slightly modiﬁed version of the i-SIR algorithm, with an extra randomization of thestate position.
The k -th iteration is deﬁned as follows.
Given a state Y k ∈ X ,
(i) draw I k +1 ∈ { 1, . . .
, N } uniformly at random and set
X I k
+1 k
+
=
Y k ;
(ii) draw X 1: N \{ I k +1 } k
+ independently from the proposal distribution λ ;
(iii) compute, for i ∈ { 1, . .
. , N }
, the normalized importance weights
[Equation]
(iv) select Y k
+1 from the set X 1: N k +1 by choosing X ik +1 with probability ω iN , k +
Thus, compared to the simpliﬁed i-SIR algorithm given in the introduction, the state is inserteduniformly at random into the list of candidates instead of being inserted at the ﬁrst position.
Of course,this change has no impact as long as we are interested in integrating functions that are permutationinvariant with respect to candidates, which is the case throughout our work.
Still, this randomizationmakes the analysis much more transparent.
A.2 Proof of Theorem
We writeϕ N (d( y , x 1: N ))
=
1 N
N (cid:88) i =
π (d y )
δ y (d x i ) (cid:89)
j (cid:54)
=
i λ (d x j ) =
Nλ ( w )
N (cid:88) i =
w ( x i ) λ
(d x i )
δ x i (d y ) (cid:89)
j (cid:54)
=
i λ (d x j ) =
λ ( w )
N (cid:89) j
= λ (d x j )Γ
[Equation]
where we recognize, and after having recalled deﬁnitions (3) and (4) of π N and Π N , respectively,the right-hand side as π N (d x 1: N )Π N ( x 1: N , d y ) .
This completes the proof.
A.3 Proof of Theorem 2
Using (4) we get
[Equation]
and the proof is complete.
A.4 Proof of Theorem 5
Proof.
We ﬁrst check that ϕ N is an invariant distribution for P N .
For every A ∈ X (cid:15) ( N +1) , usingthat π is the marginal of ϕ N with respect to the state and applying Theorem 1 yields
[Equation]
which establishes invariance
We now show that P N is reversible with respect to π .
For this purpose,let g and h be two nonnegative measurable functions and write, using Theorem 1 twice,
[Equation]
A.5 Proof of Theorem 6
For completeness, we repeat the arguments in [25, 5].
Under A 1, we have, for ( x , A ) ∈
X × X ,
[Equation]
Finally, since the function f : z (cid:55)→ ( z + a ) − 1 is convex on R + and a > 0 , we get for i ∈ { 2, . . .
, N } ,
[Equation]
We ﬁnally obtain the inequality
[Equation]
This means that the whole space X is (1, (cid:15) N π ) -small
(see [10, Deﬁnition 9.3.5]).
Since P N ( x , · ) andπ are probability measures, (14) implies
[Equation]
Now the statement follows from [10, Theorem 18.2.4] applied with m = 1 .
A.6 Proof of Theorem 3
Proof of (ii).
Using the identity ( a + b ) 2 ≤ (1 + (cid:15) 2 ) a 2 + (1 + (cid:15) − 2 ) b 2 we obtain the decomposition{ Π N f
( X 1: N k )
− π ( f ) } 2 ≤ (1 + ( N − 1) − 1 / 2 )
I (1) + ( [Equation] ( N − 1) 1 / 2 )I (2) , with
[Equation] where a N ( Y k − 1 ) = Λ N Γ N f ( Y k − 1 ) and b N ( Y k − 1 ) =
Λ N Γ N 1 X
( Y k − 1 ) .
Using the identity a/b − c/d = (1 /d ) [( a/b )( d − b ) − ( c − a )] , we obtainΠ N f
( X 1: N k ) − a N
( Y k −
[Equation] ) /b N ( Y k −
[Equation]
Therefore, using the trivial bound ( a + b ) 2 ≤ 2( a 2 + b 2 ) , we get I (
[Equation]
Since Π N f
( X 1: N k ) 2 ≤ 1 , P ξ -a.s., and b N ( y ) ≥ ( N − 1) /Nλ ( w ) , it holds, P ξ -a.s., I
[Equation]
[Equation]
≤
[Equation]
w
[Equation]
[Equation]
Therefore, using Lemma 7,
E ξ [ { Π N f ( X 1: N k ) − a N ( Y k − 1 ) /b
N ( Y k − 1 ) } 2 ] =
[Equation]
≤ 4( N − 1) − 1 κ [ π , λ ].
We turn to I (2) and note that (12) implies that I (2) ≤ 4 N − 2 (1 + ω ) 2 , which completes the proof.
Proof of (iii).
Note that
[Equation]
As E ξ (cid:2) Π N f ( X 1: N k +
[Equation]
[Equation]
Y
[Equation]
+
[Equation]
= Φ N ( Y k + (cid:96) − 1 ) P ξ -a.s. , it holds that I ( [Equation]
[Equation]
[Equation]
( f
[Equation]
By the Markov property,
[Equation]
which, combined with (10), implies that
[Equation]
Combining the results above, we ﬁnally establish that
[Equation]
A.7 Proof of Theorem 4
We ﬁrst consider the bias term, which can be bounded according to
[Equation]
Thus, the claimed bias bound can be established by noting that
[Equation]
We now turn to the MSE, and make the decomposition
[Equation]
Using the MSE bound in Theorem
[Equation]
we obtain that
[Equation]
In addition, the covariance bound of Theorem 11 yields
[Equation]
[-( j − (cid:96) ) − 1 N   . As (cid:80) K(cid:96) = K 0 +
-]

[Equation]
As
[Equation]
≤ ( K − K 0 )(4 / 3) τ mix , N , we may write
[Equation]
The MSE bound may now be established by noting that ( K − K 0 )( N − 1) = υM .
[-ς mse 
 M − 1 . The-]Establishing the high-probability bound requires more complex derivations.
More precisely, we willapply the decomposition
[Equation]
), where we used that π ( f )
=
π (Φ N ) .
Therefore, for every t ≥ 0 it holds that
P ξ ( | Π ( K 0 , K ), N ( f ) − π ( f ) | ≥ t ) ≤ P ξ (cid:32) ( K
[Equation]
) −(cid:12) (cid:12) (cid:12) (cid:12)(cid:12) K (cid:88)
[Equation]
We will show that for all t > 0 , and for some absolute constants ζ (1) and ζ (2) ,
I
[Equation]
We ﬁrst consider I (1) and note that
[Equation]
By Theorem
[Equation]
the random elements ( X 1: N k ) Kk = K [Equation] +1 are independent conditionally to ( Y k ) K − 1 k = K 0 .
Thus, using the generalized Hoeffding inequality (see [44, Theorem 2.6.2] or [45, Proposition 2.1])we get, with ∆ N , k = Π
N f
( X 1: N k ) − Φ N ( Y k − 1 ) , that, P ξ -a.s., P
[Equation]
where ψ 2 : x (cid:55)→ exp( x 2 ) − 1 and
[Equation]
In order to bound (cid:107) ∆ N , k (cid:107) ψ 2 , Y k − 1 we use the decomposition ∆ N , k = ∆ (1) N , k + ∆ (2) N , k , where
[Equation]
combined with Lemma 9 with φ = χ = ψ 2 and [44, Proposition 2.6.1].
By (11) and by [44,
Equation 2.17] it holds that, P ξ -a.s.,
[Equation]
log [Equation] −
[Equation]
− 1 κ [ λ , π ]. Using Lemma 9 with φ = χ = ψ 2 and the fact that b N ( y ) ≥ (1 − 1 /N ) λ ( w ) we obtain, P ξ -a.s.,(cid:107) ∆ (1) N , k (cid:107) ψ 2 , Y k −≤
[Equation]
Furthermore, using [44, Proposition 2.6.1, Eq 2.17] we get, P ξ -a.s.,
[Equation]
The same bound applies to (cid:107) Γ N 1 X
X 1: N k ) − b N
( Y k − 1 ) (cid:107) 2 ψ 2 , Y k − 1 , and we may write(cid:107)
∆ (1) N , k (cid:107) ψ 2 , Y k − 1 ≤ 96e
[Equation]
We can now ﬁnalize the bound on I (1) by writing
[Equation]
where ζ (1,1) = 18432e(log 2) − 2 and ζ (1,2) = 8(log 2) − 1 are universal constants, which implies that
[Equation]
with ζ (1) = 1.1 · 10 5 ω 2 .
This ﬁnally yields that I (1) ≤ 2 exp( − t 2 υM/ 4 ζ (1) ) .
We treat I (2) using Lemma 12 with g i =
Φ N ( Y K
+ i − 1 ) − π (Φ N ) .
As (cid:107) g i (cid:107) ∞ ≤ osc(Φ N ) ≤( N − 1) − 1 ς bias , we obtain
[Equation]
where ζ (2) = 2 / (3 ς bias ) 2 .
Finally, we obtain
P [Equation]
We conclude by noting that for every δ ∈ (0, 1) and N − 1 ≥ τ mix , N (4 ζ (1) ζ (2) ) − 1 it holds that
P ξ ( | Π ( K 0 , K ), N ( f ) − π ( f ) | ≥ t ) ≤ 4 exp (cid:16) − t [Equation] / 4 ζ (1) (cid:17) ≤ δ for all t ≥ 2 ζ 1 / 2 I ( υM ) − 1 / 2 log(4 /δ ) 1 / 2 .
Letting ς hpd = 2 ζ 1 / {+
+} I concludes the proof.
A.8 High-probability inequality for SNIS
Theorem 8.
Assume that ω = (cid:107) w (cid:107) ∞ /λ ( w ) < ∞ .
For all bounded measurable functions f on( X , X ) such that (cid:107) f (cid:107) ∞ ≤ 1 , it holds that for every M ∈ N ∗ and δ ∈ (0, 1) ,
[Equation]
with probability larger than 1 − δ .
Proof.
Let α M = M − 1 (cid:80) Mi
=1 w ( X i ) f
( X i ) , β M = M − 1 (cid:80) Mi
=1 w ( X i ) , a = E [ α M ] = λ ( wf ) , andb =
E [ β M ] = λ ( w ) .
Note that (cid:98) π M ( f ) =
α M / β M and π ( f ) = a/b .
Using Lemma 9 with φ and χequal to the mapping x (cid:55)→ exp( x 2 ) − 1 we obtain that
[-( f ) − π ( f ) (cid:107) ψ 2 ≤ 2 λ ( w ) − 1 [ (cid:107) A M − a (cid:107) ψ 
 + 2 (cid:107) B M − b (cid:107) ψ 2 ].-][Equation]
Moreover, using [44, Eq 2.17] yields, P ξ -a.s.,
[Equation]
In the same way, (cid:107) β M − b (cid:107) 2 ψ 2 ≤ 4( M log 2) − 1 (cid:107) w (cid:107) 2 ∞ .
Therefore, we may conclude that
[Equation]
Combining the previous bound with [44, Proposition 2.5.2] provides
[Equation]
where ζ snis = (12 ω ) − 2 log 2 .
The high-probability inequality of the theorem follows directly.
B Moments and high-probability bounds for ratio statistics
Let ( U i , V i ) i ∈{ 1, ... , n } be (possibly dependent) random variables deﬁned on some probability space(Ω, F , P ) .
Assume that U i ≥ 0 P -a.s.
Moreover, let α n = n − 1 (cid:80) ni
=1 U
i V i , β n = n − 1 (cid:80) ni
=1 U i , andρ n = α n / β n as well as a = E [ α n ] , b = E [ β n ] , and r = a/b .
A continuous, even, convex function φ : R + → [0, + ∞ ] is a Young function if φ is monotonicallyincreasing for x > 0 , φ (0) = 0 , lim x →∞ φ ( x ) /x
= ∞ , and lim x → 0 + φ ( x ) /x
= 0 .
We denote by φ ∗the Fenchel-Legendre conjugate of φ .
Let X be a random variable and φ a Young function.
Then the Orlicz norm of X is
[Equation]
with the convention that inf ∅ = ∞ .
The Orlicz space L φ (Ω) of random variables is the family ofequivalence classes of random variables X such that (cid:107) X (cid:107) φ < ∞ .
L φ (Ω) is a Banach space.
Ifφ p ( x ) =
| x | p for p ≥ 1 , then L φ (Ω) =
L p (Ω) and we denote (cid:107) · (cid:107) p = (cid:107) · (cid:107) φ p .
If X ∈ L φ (Ω) , then,for every x >
P ( | X | ≥ x ) ≤ 1 /φ ( x/ (cid:107) X (cid:107) φ ), and (cid:107) 1 {| X |≥ x } (cid:107) φ = 1 /φ − 1 (1 / P ( | X | ≥ x )).
Lemma 9.
Let φ and χ be Young functions.
If max i (cid:107) V i (cid:107) ∞ ≤ c | r | , then
[Equation]
Proof.
We decompose the computation in two parts: ﬁrst, when β n > b/ 2 , we have
[Equation]
Then, when β n ≤ b/ 2 ,
[Equation]
where the second inequality follows from | β n − b | ≥ b/ 2 .
Combining the two previous inequalitiesyields
[Equation]
Recall that if | X | ≤ | Y | P -a.s. , then (cid:107) X (cid:107) φ ≤ (cid:107) Y (cid:107) φ ; hence, we may proceed like(cid:107)
ρ n − r (cid:107) φ ≤
[Equation]
Finally, we obtain the desired result by noting that for any Young function χ , P ( β n ≤ b/ 2) = 580 P
[Equation]
Theorem 10.
Let p ≥ 1 .
If max i (cid:107) V i (cid:107) ∞ ≤ c | r | , then
[Equation]
ρ n −
[Equation]
b .
Proof.
Apply Lemma 9 with χ ( x ) =
φ ( x ) = x p .
Theorem 11.
If | α n / β n | ≤ 1 P -a.s. , then
[Equation]
[Equation]
Proof.
Using the identity
[Equation]
yields
[Equation]
which completes the proof.
We conclude with a lemma that gives the concentration of a uniformly ergodic Markov chain.
Wethink that this Lemma is of independent interest, and we give it under general conditions.
Lemma 12.
Let ( Z , Z ) be a state-space and Q a Markov kernel on ( Z , Z ) which is uniformlyergodic with mixing time t mix and stationary distribution π .
Let ( g i ) ni
=1 be a family of R d -valuedmeasurable functions on Z such that (cid:107) g (cid:107) ∞ =
max i ∈{ 1, ... , n } (cid:107) g
i (cid:107) ∞ < ∞ and π ( g i ) = 0 for alli ∈ { 1, . . . , n } .
Then for every initial probability ξ on ( Z , Z ) , n ∈ N , and t ≥ 0 ,
[Equation]
where u n = 3 (cid:107) g (cid:107) ∞ √ nt mix .
Proof.
The function ϕ ( x 1: N 1 , . .
, x 1: Nn ) =
[Equation]
ni =
[Equation]
g i ( x 1: N i ) (cid:107) on Z n satisﬁes the bounded differencesproperty.
Applying [35, Corollary 2.10], we get, for t ≥
E ξ [ (cid:107) (cid:80) ni
=
g i ( Z i ) (cid:107) ] ,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
It remains to bound E ξ [ (cid:107) (cid:80) ni
[-= 
-]{+=
+}
g i ( Z i ) (cid:107) ] from above. For this purpose, note that [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Equation]
where, using that π [Equation] [Equation] which implies that
[-=
-][Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
Combining the bounds above
[Equation]
we obtain the upper bound
[Equation]
By plugging this result into (15), we obtain that
[Equation]
Now, since the right-hand side of (16) is, for every t ≥ 0 , upper bounded by 2 exp( − 2 t 2 / (9 v 2 n )) , thestatement of the lemma follows.
C Experiments
C.1 Toy Problem
[Figure]
Bias–MSE trade-off: Figures 4a and 4b display the bias and the MSE of the BR-SNIS estimatorswith the same conﬁgurations as in Figure 2 but with k [-
-]0 = (cid:98) 0.
k max (cid:99) .
In the case where N = 129 ,we observe 3 times less bias than for the SNIS estimators but only with a 10% increase of the MSE.
This can be also seen in Figure 4c, where we display, in the case N = 129 , the ratio between BR-SNIS and SNIS for bias and MSE.
Parameters Gaussian mixture: The π in Section 3 is a Mixture of two Gaussians in dimension 7with mean vectors µ 1 = (1, . .
, 1) (cid:124) and µ 2 = ( − 2, 0, . . . , 0) (cid:124) and covariance matrices Σ
= d − 1 Iand Σ 2 =
d − 1 I , where p = 1 /
and I is the identity matrix
In this setting, the quantities κ [ π , λ ] andω can be estimated by Monte Carlo and Gradient ascent respectively.
Their values are approximately7 · 10 2 and 1 · 10 4 , respectively.
The sets A and B used to deﬁne the function f are the following: A
[Equation]
6] ×
[Equation]
[Equation]
[Equation]
[Equation]
[0.75, 1.25] ×
[Equation]
] ×
[Equation] [Equation] [Equation] We used this example to illustrate numerically the bounds in Theorems 3 and 4, where each expectationwas calculated by Monte Carlo using 2 · 10 4 samples.
We displayed in each ﬁgure the equivalent
SNIS estimation in a green dashed line.
For all the bias related bounds(Theorem 3(i) in Figure 5a,
Theorem 4(i) in Figure 5c), we ﬁxed a total budget of M = 6 · 10 3 .
For Figure 5a we added a ﬁt ofthe type y = exp( ak + b ) to illustrate the exponential decay w.r.t.
k .
[Figure] We then increased the budget to M = 8 · 10 4 for the MSE and covariance bounds, in order to fullyobserve the stabilisation of the MSE in Figure 5b for all the minibatch sizes N .
For the true valueof π ( f ) needed for calculating the MSE [Figure] we use an estimation obtained by Monte Carlo (samplingdirectly from π ) with 4 · 10 7 samples.
In Figure 5d we added dashed lines with the theoretical valueof the MSE is υM with the same color as υ
[Figure]
Comparison with unbiased estimators: There exist estimators based on SNIS that have no bias,such as the estimator proposed in [30] which we will refer to as the Unbiased-PIMH estimator .
One of the main differences between this estimator and BR-SNIS is that BR-SNIS operates undera pre-established budget of samples, whereas the number of samples required in the UnbiasedPIMH estimator varies randomly.
In the BR-SNIS algorithm, the conﬁguration N = 32 and k = 4corresponds to 128 samples.
On the other hand, applying the Unbiased-PIMH estimator with N = 32requires cN samples, where c ∈ { 2, 3, . . . } is random.
Still, even though the two estimators havedifferent goals, it can be of interest to compare both of them in the case where there is a restriction inthe total number of available samples.
Table 3 displays approximate 95% intervals for both estimators under a strict budget of 8.192 · 10 7samples for several conﬁgurations of each estimator.
The pseudocode for the "Rao-Blackwellized"version of the Unbiased-PIMH estimator used in Table 3 is shown in Algorithm 1.
All the BR-SNISestimates were obtained with k 0 = (cid:98) 0.7 k max (cid:99) .
In this setting, all the conﬁgurations of the BR-SNISalgorithm implied more narrow conﬁdence intervals than the those of the Unbiased-PIMH algorithm.
C.2 Bayesian Logistic regression
The importance distribution used in the Bayesian logistic regression example is given by the meanﬁeld variational distribution [6].
More precisely, given the target π given in Section 3, the proposal λis a Gaussian distribution with mean µ and diagonal covariance diag( σ ) , where µ , σ are learnt by Data
← 11 while τ = ∞ do
; end
; end if
τ = t ; end 24 end
N k algorithm lower bound center upper bound interval length
BR-SNIS -0. BR-SNIS -0.
BR-SNIS -0. BR-SNIS -0.
N/A
Unbiased-PIMH -0.
N/A
Unbiased
[Table]
[Table]
maximization of the Evidence Lower Bound (ELBO):
[Equation]
In both Figures 3 and 6, the optimal k for a given budget M was chosen by grid search over all thefactorizations of M .
The ﬁnal settings are shown in Table 4.
C.3 Importance Weighted Auto-Encoders
The details of training procedure are given in Table 5.
The train ELBO for each latent dimension isshown in Figure 9.
For each network we display generated images in Figures 7 and 8.
The generationis done by sampling from the prior p θ .
Dataset component
M k max
N breast    breast    breast    breast    breast    breast    breast    breast    breast    breast    breast    breast    breast    breast    breast    heart    heart    heart    heart    heart    heart    heart    heart    heart    heart    heart    heart    heart    heart    heart    covertype    covertype    covertype    covertype    covertype    covertype    covertype    covertype    covertype    covertype    covertype    covertype    covertype    covertype    covertype
[Figure] For the log likelihood comparison in Table 2, we use SNIS with the posterior as importance distribution and a total of 10 4 samples.
Therefore, the estimation of the log likelihood is:ˆ L
[Equation]
T (cid:88)
j
[Equation]
[Equation]
(cid:88)
i [Equation] [Equation] | z ji ) with ω θ , φ , x ( z ) =
p θ ( x ) /q
φ ( z | x ) where z ji is sampled from q φ ( · | x j ) .
Name Value
Name
Value Batch size Epochs IS samples
(IWAE, Un IWAE Encoder inner layer size
Learning rate
[Figure]
Decoder inner layer size
[Figure] [Figure]
C.4 Resources
[Figure] All the simulations were done using a server with the following conﬁguration: [Figure] [Figure]
[Figure]
[Figure]
(c) Un-IWAE ( k = 4 )
[Figure] (d) Un-IWAE ( k = 8 ) [Figure] • GPUs: two Tesla V100-PCIE (32Gb RAM)
[Figure]
• CPU: 71 Intel(R)
Xeon(R)
Gold 6154 CPU @
3.00GHz
[Figure]
• RAM: 377Gb
locally hosted.
We estimate the total number of computing hours for the results presented in thispaper to be inferior to 200 hours of GPU usage (All the calculations were done in the GPU).