Variational Reﬁnement for Importance Sampling Using the Forward Kullback-Leibler Divergence
Anonymous Authors
Anonymous Institution
Abstract
Variational Inference (VI) is a popular alternative to asymptotically exact sampling in
Bayesian inference.
Its main workhorse is optimization over a reverse Kullback-Leibler divergence (RKL), which typically underestimates the tail of the posterior and causes miscalibration and potential degeneracy (over-pruning).
Importance sampling (IS), on the other hand, is often used to ﬁne-tune and debias the estimates of approximate Bayesian inference procedures.
The quality of IS crucially depends on the choice of the proposal distribution.
Ideally, the proposal distribution has heavier tails than the target, which is unachievable by minimizing the RKL.
We thus propose a novel combination of optimization and sampling techniques for approximate Bayesian inference by constructing an IS proposal distribution through the minimization of a forward KL (FKL) divergence.
This approach guarantees asymptotic consistency and a fast convergence towards both the optimal IS estimator and the optimal variational approximation.
1. Introduction
Bayesian analysis provides a powerful framework to encode complex hierarchical structures and prior beliefs and capture posterior uncertainty about latent variables θ given observed data x via the posterior p ( θ | x ).
The inferential goal often involves computing expectations over this posterior distribution, E θ ∼ p
( θ | x ) [ f ( θ )], for some function f which is generally accomplished by sampling.
Unfortunately, the posterior distribution is often diﬃcult to sample from directly, and approximate inference methods are necessary to estimate posterior functionals.
These methods include optimization-based techniques such as variational inference (VI) (Jordan et al., 1999; Wainwright and Jordan, 2008), and sampling-based techniques such as Markov Chain Monte Carlo (MCMC) (Brooks et al., 2011; Andrieu et al., 2003) and importance sampling (IS) (Gelman and Meng, 1998).
Variational inference has grown in popularity among these methods due to the computational convenience of minimizing the reverse (exclusive) Kullback-Leibler divergence (RKL or KL ( q || p )) over a tractable family of distributions.
Furthermore, formulating inference as divergence minimization enables the application of recent advances in stochastic optimization (Bottou, 2010; Hoﬀman et al., 2013) and automatic diﬀerentiation (Maclaurin et al., 2015).
However, the quality of VI approximations, when the variational family does not exactly match the target, is of major concern, especially when the approximation is used to estimate posterior expectations (Yao et al., 2018; Campbell and Li, 2019; Rainforth et al., 2018; Huggins et al., 2020).
With importance sampling (Gelman and Meng, 1998; Owen, 2013), one can re-weigh posterior summaries from a variational approximation q ( θ ) in order to correct for the error in q as compared to the target at the cost of additional variance relative to a simple Monte
Carlo estimate.
The performance of q when used as an IS proposal can also be a practical diagnostic for whether p ( θ | x ) should be approximated by q ( θ ) (Yao et al., 2018).
Importance sampling is thus still a relevant tool especially in the modern era of approximate inference where the family of approximate distributions is often mis-speciﬁcation and reﬁnement might be needed to de-bias posterior summaries.
However, practical applications of importance sampling are challenged by the choice of proposal distribution, which is particularly diﬃcult in multimodal and high-dimensional settings.
One potential way out of this conundrum is to use of variational inference to construct a tractable proposal distribution for IS and thereby capitalize on the complementary strengths of IS and VI.
Unfortunately, the VI approximations obtained by RKL minimization are not ideal proposal distributions for IS due to the light tails resulting from entropy penalization (Minka et al., 2005).
This typically leads to heavy-right tails in the distribution of importance weights, which can lead to unstable IS estimates, sometimes with inﬁnite variance (Yao et al., 2018).
On the other hand, forward (inclusive) KL divergence (FKL) (Minka et al., 2005) controls the worst-case estimation error of importance sampling (Chatterjee and Diaconis, 2018).
Furthermore, forward KL does not suﬀer from issues of covariance underestimation (Minka et al., 2005; Campbell and Li, 2019) or zero forcing (Hoﬀman, 2017).
We propose to replace reverse KL with forward KL as the variational objective that supplies an IS proposal distribution.
We make the following contributions in this vein:
We derive a self-normalized importance sampling estimate for the FKL divergence.
We show how FKL-based boosting can be used to better harness the combination of IS and VI to address the challenge of estimating multimodal target distributions.
We show that FKL boosting is guaranteed to converge at a rate of O ( 1 K ), where K is the number of boosting iterations, to the best approximation from a family of mixture distributions.
This implies convergence to the optimal proposal distribution
as per the results of Chatterjee and Diaconis (2018).
Our proposed algorithm is a principled inference technique, with a well-deﬁned computation-quality trade-oﬀ, that can be used independently or as a reﬁning step to correct for a given approximation’s error.
2. Background
Let θ denote the variable of interest with probability density p , and let f denote a function of θ .
Our goal is to estimate an expectation E θ ∼ p ( θ | x ) [ f ( θ )].
In Bayesian inference, θ generally represents a latent variable to be integrated over in the posterior distribution, p ( θ | x ), conditioned on the observed data x .
If one can draw S samples { θ s } Ss =1 from the target distribution p ( θ | x ) then the expectation over the target distribution can be estimated by simple Monte Carlo integration.
However, p ( θ | x ) is typically only known up to a normalization constant and thus cannot be readily sampled from.
Instead, given samples from an approximation q ( θ ) of the posterior, we can estimate the expectation as:
[Equation]
If the samples are weighted equally (i.e., w s = 1), Eq.
(1) is equivalent to the VI estimate, which has low variance but can be biased and inconsistent (Owen, 2013).
If instead we weigh by the importance ratios, w s = p ( θ s | x ) q ( θ s ) , we recover the IS estimate:
[Equation]
which is asymptotically consistent ( O (1 /S ) bias) but with possibly inﬁnite variance.
2.1. Variational Inference with Reverse KL Minimization
Variational inference posits a family Q of distributions that are easy to evaluate or sample from, and deﬁnes the variational approximation q ∗ ∈ Q as the distribution that minimizes the reverse Kullback-Leibler (RKL) divergence to the posterior p : q ∗ = argmin
[Equation]
Unless the choice of Q exactly includes the target distribution, minimizing the reverse KL divergences leads to an q ∗ that underestimates the target co-variance.
The decomposition of this divergence sheds light on the cause:
[Equation]
The ﬁrst term in Eq.
(3) is the entropy of q whose penalization is known to cause light tails.
Furthermore, the second term is minimized when q = 0 for p > 0, leading to zero forcing or over-pruning (Higgins et al., 2016).
The RKL divergence thus favors a single mode ( mode seeking ) and is biased towards avoiding false positives, which can cause severe under-estimation of the covariance (Blei et al., 2016; Murphy, 2012) and poses diﬃculties when approximating heavy-tailed (Guo et al., 2016; Li and Turner, 2016;
Dieng et al., 2017) or multimodal targets (Miller et al., 2017).
This eﬀect is illustrated in Figure F.
2.2. Importance Sampling and Forward KL
Since the the variational family Q rarely includes the true posterior, reﬁning posterior approximations through importance weighting is often recommended to de-bias expectations of interest (Yao et al., 2018; Vehtari et al., 2015).
However, the light tails of VI approximations using RKL can lead to high or even inﬁnite variance for the IS estimator.
As an alternative to RKL, minimizing the forward KL divergence can mitigate the issue of covariance underestimation and light tails.
Consider the decomposition of the FKL:
[Equation]
In Eq.
(4), the cross-entropy is minimized by setting q >
whenever p > 0, which leads to mass covering as illustrated in Figure F. With tails that are more likely to cover the target distribution p , an approximation q ∗ generated by minimizing the FKL can lead to an IS estimate (Eq.
(2)) with lower variance.
Chatterjee and Diaconis (2018) show that the forward KL divergence controls the downstream worst-case estimation error of IS.
3. Methodology
We develop our novel approach to integrate variational inference and importance sampling using the forward KL divergence with a focus on multimodal targets.
Note that we assume the target density p and the approximation q share the same support which can be R d or a subset thereof.
This guarantees that p is absolutely-continuous with respect to q (noted as p (cid:28) q ) and
vice-versa which is necessary for the deﬁnition of the KL divergence.
3.1. Forward KL-based Variational Approximation for IS
By Theorem 1 . 1 of Chatterjee and Diaconis (2018), we know that the variance of an importance sampling estimate scales as O (cid:16) e KL( p (cid:107) q ) √ S (cid:17) , where the number of samples S required for importance sampling to provide accurate mean estimates scales exponentially with the KL divergence KL( p (cid:107) q ).
Therefore, an optimal proposal distribution should minimize the forward KL divergence which is computationally diﬃcult given the need for samples from an un-normalized target p .
Alternatively, the reverse KL divergence, which utilizes samples from q instead, can be regarded as a tractable approximation with unknown bias (Yao et al., 2018).
However, for tractable choices of Q (i.e. when KL ( q || p ) (cid:54) = 0) it has been demonstrated that minimizing the RKL is not guaranteed to minimize the FKL divergence (Campbell and Li, 2019).
Furthermore, the light tails of the RKL minimization solution can lead to unstable IS estimates (Yao et al., 2018).
Therefore, we seek to directly minimize the forward KL objective instead of the RKL to estimate the optimal IS proposal with improved covariance and tail estimation.
From the IS point of view, this combination with VI enables a computationally eﬃcient and principled construction of optimal proposal distributions.
3.2. Self-Normalized IS Approximation of Forward KL
We cannot compute the forward KL divergence exactly for unnormalized target distributions.
This stems from the expectation under the target p in Eq.
By contrast, the
RKL takes the expectation in Eq.
(3)) under a normalized approximation q .
Therefore, to approximate FKL, we can rearrange densities inside the expectation as follows:
[Equation]
Eq.
(5) can then be approximated through self-normalized importance sampling (SNIS) (Murphy, 2012) which is known to generally be consistent:
[Equation]
The SNIS estimation can have arbitrarily high variance depending on q .
Gradients of
Eq.
(5) with respect to the parameters of q i can also have high variance since we optimize over the same distribution from which samples are drawn.
For certain distributions, the reparametrization trick can reduce this variance (Kingma and Welling, 2013).
Regardless, gradient variance is less of a concern in the following sequential approximation framework.
3.3. Forward KL-Based Boosting
We seek to sequentially construct a proposal mixture distribution q that is both easy to sample from and minimizes the FKL divergence to ensure eﬃcient IS estimation.
The design of a variational family Q can be challenging a priori which often leads to model misspeciﬁcation in VI estimates.
Our experiments in App.
D rely on the family of Gaussian mixtures due to the low variance of reparametrization gradients (Kingma and
Welling, 2013).
Our framework also supports mixtures of heavy-tailed distributions which are desirable for adaptive IS (Geweke, 1989).
Our proposal distribution at the K th boosting iteration is q K ( θ ) :
= (cid:80)
Ki =1 λ
i f i ( θ ; φ i ).
At each boosting iteration, we minimize the forward KL between the mixture q i and the target p while holding the parameters of previously-learned mixture components ﬁxed: argmin
[Equation]
This is known to be more eﬃcient and more stable than the joint optimization of all mixture components at each iteration (Locatello et al., 2018a).
The mixture weight for the new component is then set to λ i = γ , and all other weights { λ j } i − 1 j =
are re-scaled by (1 − γ ).
Mixture weights can be further adjusted with a fully-corrective weight search using the gradient derived in Eq.
(20) (Guo et al., 2016; Locatello et al., 2018b).
We provide an alternative approach to building q K using the remainder in App.
B.
3.4. Lower Self-Normalized IS Variance with Boosting
The main computational concern in estimating the FKL divergence with SNIS is the variance of IS in Eq.
However, the sequential construction described above enables further rearrangements of the SNIS approximation which can lead to a lower variance than the naive approximation of Eq.
For the ﬁrst approach, the global objective can be re-written as: argmin
[Equation]
Eq.
(8), which is only possible in a sequential setting such as ours, reduces the gradient variance since the component being estimated, f i , is independent of the distribution q i −from which samples are drawn to approximate the FKL.
Furthermore, we can draw a large number of samples from q i − 1 a single time at the start of each boosting iteration.
The SNIS approximation of Eq.
(8) given samples θ s from q i − 1 can be computed as:
[Equation]
where w s are the self-normalized importance weights computed as in Eq.
3.5. Initialize with Reverse KL, Reﬁne with Forward L
Our sequential construction of the proposal can reduce the variance of the SNIS approximation of the FKL divergence by drawing samples from the previously-estimated and ﬁxed mixture distribution q i − 1 .
However, in the ﬁrst boosting iteration, we do not have an existing approximation to sample from.
Instead, Eq.
(9) can be re-written as follows:
[Equation]
This exacerbates the sensitivity to the initialization of f i .
In fact, if f i were initialized to be sharply peaked at zero, for example, then our SNIS estimate of FKL cannot properly capture the tails.
Therefore, a diﬀuse initialization is recommended.
However, an even more practical initialization would use RKL-based VI to identify the mode of the target and provide a computationally eﬃcient approximation that can be reﬁned by FKL boosting in later iterations.
As such, this FKL-boosting workﬂow can be considered as general framework for the iterative reﬁnement of any given posterior approximation to be used for the estimation of expectations of interest.
This best combines the strengths of
VI and IS as the ﬁrst iteration of RKL minimization can reduce the variance of the SNIS approximation of FKL for the following iterations.
4. Analysis
We provide theoretical analysis of the proposed method of performing IS with a proposal distribution constructed from FKL-based boosting (further analysis in App. C).
4.1. Forward KL Controls Moment Estimation Error
Minimizing the FKL implicitly minimizes the error in posterior probabilities and moments via its control on total variational (through Pinsker’s inequality(Tsybakov, 2008)) and nWasserstein as follows (Bolley and Villani, 2005):
Assume that the probability density q is n -exponentially integrable.
Then for target distribution p such that p (cid:28) q :
[Equation]
[Equation] where C EIn ( q ) = inf x 0 ∈ R d ,α> 0 (cid:0) 3 α + 2 α log
[Equation]
This implies convergence in up to n th moments (Bolley and Villani, 2005).
Note that because of the symmetry of Wasserstein distances, we can switch probability densities p and q in the above inequalities and obtain bounds in terms of RKL.
However, that would incur the n -exponential integrability condition on the target probability p , which boils down to generalized sub-gaussianity of its tail.
The n -exponential integrability assumption does not need to be satisﬁed by the target density p in the case of Eq.
Instead, it is only required of the family of variational approximations q which is easier to enforce and verify (and is automatically satisﬁed by the mixture of Gaussians).
A smaller constant C EIn can also be achieved by the same reasoning.
4.2. Forward KL Boosting Converges at O(1/K)
Assuming p (cid:28) q , which can be veriﬁed by the design of Q , the functional gradient of the forward KL divergence is derived in App.
E.5 as δKL ( p || q ) δq [Equation]
The convexity of
[Equation] q [Equation] in q is well established in the literature (proven with the log-sum inequality).
Furthermore, we show in App.
4.2 that the FKL functional is also β -smooth in q where β depends on the range of the values that the density q can take.
This requires bounding q away from zero and from above which is typical in prior theoretical work (Guo et al., 2016) and aligns with practice as it can translate to a bounded parameter space for a given family of distributions.
For a convex and strongly smooth functional, the greedy sequential approximation framework of Zhang (2003) provides an asymptotic guarantee for the convergence to a target distribution in the convex hull of a given base family such that the approximation error at the K th iteration is : KL( p
[Equation]
= O
[Equation]
1 /K ) .
This framework does not require each iteration to exactly solve for the optimal mixture component which can be diﬃcult for the non-convex optimization sub-problems.
Appendix A. Related Work
We present related work on improving estimates of the expectation over intractable target distributions that are high dimensional, heavy tailed, and/or multimodal.
A.1. Variational Inference Divergences
Prior work has addressed the covariance underestimation and light tails pathologies of reverse KL minimization while seeking to improve the quality of the approximation as an
IS proposal through the minimization of alternative divergences such as (reversed) Renyi-
α (Li and Turner, 2016), χ 2 ( α = 2) (Dieng et al., 2017), or Hellinger ( α = 1 / 2) (Campbell and Li, 2019) divergences.
FKL can be seen as a special case of α divergences when α →which is not considered in any of these prior VI works.
While the forward KL’s computational inconvenience has limited its use for variational inference, inference techniques such as Belief Propagation (BP) and Expectation Propagation (EP) can be regarded as performing FKL minimization locally (Minka et al., 2005). This is diﬀerent from how a KL divergence is typically used in VI since EP/BP do not optimize a global objective that is related to the KL divergence and are thus not guaranteed to converge. Furthermore, EP/BP does not provide an easy estimate or bound on the marginals.
Another set of techniques that utilize variants of the FKL divergence includes reweighted wake-sleep (Bornschein and Bengio, 2014) which alternates minimizing an approximation of FKL during the sleep phase while minimizing an approximation of RKL during the wake phase leading to systemic bias.
A.3. Variational Boosting
Variational boosting (VB)(Jerfel, 2017; Miller et al., 2017; Guo et al., 2016; Locatello et al., 2018a) has been suggested in various forms to address the multimodality challenge for variational inference.
Variational boosting posits a family of mixture distributions Q k :
Q k = (cid:40) q : q ( θ ) =
k (cid:88)
[Equation]
and iteratively reﬁnes a variational mixture approximation by adding and re-weighting one (typically Gaussian) mixture component at a time in order to minimize the global objective:
[Equation]
This form of reverse KL-based boosting is known to struggle with degeneracy where the optimization at certain boosting iterations can lead to point-mass components (Campbell and Li, 2019).
Ad-hoc regularization techniques are often deployed in practice (Locatello et al., 2018a), but are not necessarily suﬃcient (Campbell and Li, 2019).
To address this pathology of RKL-based boosting, Campbell and Li (2019) proposed a boosting algorithm, based on the Hellinger divergence, which is still not guaranteed to scale well with dimension.
A.4. Adaptive Importance Sampling
The adaptive IS (AIS) literature presents a range of techniques that involve adapting proposal distributions for multimodal targets.
Adaptive multiple IS (Cornuet et al., 2012) and incremental mixture IS (Raftery and Bao, 2010) are two of the most popular methods.
However, none of the existing works directly optimize for the FKL divergence which controls the worst case IS estimation error.
For example, Capp´e et al.
(2008) minimize an entropy criterion whereas Douc et al.
(2007) minimize the empirical variance of the importance weights, which does not necessarily correlate with the quality of IS estimation (Vehtari et al., 2015).
A.5. Combining IS and VI
As outlined in Table 1, VI and VB suﬀer from covariance underestimation, and can struggle to approximate heavy-tailed distributions.
AIS, on the other hand, can approximate heavytailed targets but cannot scale eﬃciently in dimensionality.
A combination of these two lines of research may beneﬁt from their complementary strengths while sidestepping shared weaknesses (e.g., multimodality).
However, it is often diﬃcult to combine optimizationbased and sampling-based inference techniques.
This is because sampling methods such as MCMC deﬁne the approximate distribution implicitly such that its density cannot be evaluated.
This has driven the development of alternatives to the KL divergence such as the variational contrastive divergence (Ruiz and Titsias, 2019).
However, we are not aware of similar work for IS that leverages the computational eﬃciency of VI through a unifying loss such as the FKL divergence.
Other prior proposals combining importance weighting and VI (Domke and Sheldon, 2018) have still focused on minimizing the reverse KL, and thus do not inherently capture heavier tails of the target distribution.
Prangle (2019) recently presented concurrent work on combining IS and VI.
However, their method relies on normalizing ﬂows for constructing the proposal distribution such that it does not guarantee a multimodal approximation.
[Table]
Appendix B. An Alternative approach to FKL-Based Boosting: Minimizing the Remainder
As we seek to construct an optimal proposal through the minimization of an SNIS approximation of FKL [Table] a trade-oﬀ arises: “should we make the distribution easier to sample from in order to minimize the SNIS variance or should we bring it closer to the target in order to improve the worst-case IS estimation error?”
In particular, the closer the proposal gets to a multimodal target, the harder it may be to sample from.
Therefore, this trade-oﬀ translates to two distinct approaches for the greedy additive construction of an optimal proposal mixture distribution
[Table] The ﬁrst approach described in Section 3.3 is the most straightforward as it minimizes the forward KL between the mixture q i and the target p while holding the parameters of previously-learned mixture components ﬁxed.
Alternatively, deﬁne the remainder distribution at iteration i as r i ( θ ) =
p ( θ | x ) q i ( θ ) .
A second approach is to minimize the FKL between each new component and r i , which may be simpler with fewer modes than p :
[Equation]
The mixture weight can be estimated in this scenario for each mixture component by gradient descent using the gradient with respect to FKL (see App. E.4).
This second approach is appealing because, at each boosting iteration, r i becomes shallower with fewer modes which makes it easier to sample from than the multimodal proposal q i .
This approach can also be motivated by gradient boosting (Friedman, 2001) or matching pursuit (Mallat and Zhang, 1993) where one seeks to identify the mixture component that best ﬁts the functional residual.
Furthermore, this approach might be less prone to degeneracy.
In fact, a derivation of this approach for the reverse KL, in App.
E.2, identiﬁes intrinsic entropy regularization which is often incorporated ad-hoc in similar objectives.
B.1. Lower Variance SNIS with Boosting
The remainder-based objective can also be re-written:
[Equation]
This objective also enjoys lower gradient variance since the distribution being sampled from is a single Gaussian component and not a multimodal proposal.
However, the light tails of a single Gaussian component can lead to high variance SNIS estimates in the case of severe mismatch with the target distribution.
Therefore, the choice of either should be guided by assumptions about the tails of the target.
However, given comparable empirical results of the two approaches, we limit the analysis of Section 4 and the experiments of
Section D to the ﬁrst and more straightforward approach.
Appendix C. Additional Analysis C.1. Sample Complexity of SNIS
By Theorem 1 . 1 of Chatterjee and Diaconis (2018), we know that the variance of an importance sampling estimate scales as O (cid:16) e KL( p (cid:107) q ) √ S (cid:17) , where the number of samples S required for importance sampling to provide accurate mean estimates scales exponentially with the KL divergence KL( p (cid:107) q ).
C.2. Computation-Quality Trade-Oﬀ
While our our iterative algorithm which is guaranteed to converge asymptotically to the optimal proposal distribution, we can identify three sources of approximation error: the variational inference error, the SNIS approximation bias which depends on the number of samples, and the greedy sequential approximation error which depend on the number of VI iterations, IS samples, and boosting iterations, respectively.
These three hyperparameters thus ﬁnely control the compute-quality trade-oﬀ of our framework.
From previous sections, we observe a trade-oﬀ between sample complexity of SNIS versus the optimization complexity of variational boosting.
Inclusion of more mixture components
K and more accurate optimization in the variational boosting steps can save exponentially many samples in SNIS.
However, there is a diminishing gain in increasing K .
We demonstrate in Fig.
4 this eﬀect: both FKL boosting and RKL boosting decreases forward KL divergence as more variational components are added.
The decrease slows down signiﬁcantly after inclusion of 3 mixture components.
We therefore introduce in the experiments up to 3 mixture components and select the best performance on validation data set.
From
Fig.
4 and the experimental results, we observe uniform improvements of FKL over RKL methods.
Appendix D. Experiments
We evaluate the performance of the proposed method when applied to Bayesian linear regression (BLR) and Bayesian neural networks (BNNs) using a Gaussian prior and a heavy tailed prior.
We use four datasets from UCI Dua and Graﬀ (2017).
We split each dataset into ﬁve randomly drawn 90%/10% train/test splits, which we denote D train =
{ x i , y i } N train i = and D test =
{ x i , y i } N test i =1 , with input x i and output y
i .
We report the mean and std.
dev.
of results over all splits.
We demonstrate our proposed method with K = 1, 2, and 3 boosting iterations.
We refer to ﬁtting a single Gaussian, or K = 1, as FKL VI.
For runs with more than one boosting iteration, we initialize the ﬁrst component using the RKL, and optimize subsequent iterations using the FKL, as described in Section 3.5.
Optimization details: We use the ADAM optimizer Kingma and Ba (2014) for each boosting iteration with a ﬁxed learning rate and compute gradients based on a ﬁxed number of samples using Autograd Maclaurin et al.
At the end of each boosting iteration, mixture weights are fully re-optimized using simplex-projected gradient descent Bubeck (2014) based on the analytical gradient in Eq.
Details about practical considerations and hyperparameters can be found in App.
D, and we include our code with the submission.
Comparisons: We compare our approach to variational boosting using reverse KL
Miller et al.
First, we compare to ﬁtting a single Gaussian distribution to the target to minimize the RKL (RKL VI).
We also compare to variational boosting with two and three components, also minimizing RKL (RKL VB).
For the comparison to RKL VI and
RKL VB, we use the same parametrization, initialization, and optimization techniques as for FKL VI and FKL VB.
This might lead to discrepancies compared to the published results Miller et al.
(2017); however, keeping these details consistent better disentangles the eﬀect of the RKL vs. FKL optimization.
We also compare to directly sampling from the target distribution using Hamiltonian
Monte Carlo (HMC) Neal (2011), implemented using the TensorFlow Probability library
Dillon et al.
We additionally ran 3 HMC chains in parallel and averaged the results, similar to Hoﬀman and Ma (2020).
Results were comparable between 1 and 3 HMC chains, and we include the results for 3 chains in App.
D.
D.1. Toy Data
D.2. BLR and BNNs with Gaussian Priors
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
We ﬁrst apply the same Gaussian prior as Miller et al.
(2017) for both BNNs and BLR.
We place a Gaussian prior over each weight in the model, and an inverse Gamma prior on [Table] the variances
[Table]
α ∼ Gamma(1 , 0 . 1); τ ∼ Gamma(1 , 0 . 1); w i ∼ N (0 , 1 /α ); y |
[Equation]
where w is the set of weights, and φ ( x, w ) is either a linear function of x (BLR) or a multi-layer perception (BNN).
For our BNNs, we set φ to be a one-hidden layer neural network with 50 hidden units and ReLU activation function, as done by Miller et al.
Hern´andez-Lobato and Adams (2015).
The full set of parameters that we sample is θ =
( w, α, τ ).
We use the posterior predictive distribution to compute the distribution for a given new input x :
[Equation]
We use importance sampling to estimate this posterior predictive distribution given S samples θ s ∼ q ( θ ):
[Equation]
where q ( θ ) is the proposal distribution ﬁt to the posterior p ( θ |D train ) using either forward
KL reﬁnement (our method, FKL VB) or reverse KL reﬁnement (RKL VB, Miller et al.
Note that Miller et al.
(2017) does not use importance sampling to estimate the posterior predictive distribution.
We add importance weights here as an ablation to limit our analysis to the diﬀerence between FKL and RKL optimization.
For completness, we report the estimates without IS using RKL VB in App.
D.
For the comparison with HMC, we do not use importance sampling due to lack of an explicit density function, and instead compute (13) by averaging over direct HMC samples from the posterior distribution p ( θ |D train ):
[Equation]
As our ﬁnal evaluation metric
[Table]
we report the average predictive log probabilities on held-out test data:
[Equation]
log p ( y | x, D train ) .
D.3. BLR with Heavy Tailed Priors
[Table]
In addition to the Gaussian prior, we also perform Bayesian linear regression with a heavy tailed prior.
Following Campbell and Li (2019) we place a T 2 prior on the weights.
We use the same inverse Gamma prior on the variance:
[Equation]
where A is ﬁxed, and each entry is drawn i.i.d.
before the optimization process: A ij ∼
N (0 , 1).
For these BLR experiments, φ ( x, w ) is a linear function of x with weight parameters w .
The full set of parameters that we sample is θ = ( w, τ ).
We again estimate the same posterior predictive distribution in Eq.
(13) using importance sampling in Eq.
(14), and report the average predictive log probabilities from Eq.
Appendix E. Derivations E.1. Connecting Forward KL to other metrics used for VI By
the monotonicity of Renyi- α divergences, given that lim α → 1 D α ( p, q ) =
KL ( p (cid:107) q ), and from Dieng et al.
[Equation]
[Equation]
E.2. Reverse KL remainder: intrinsic entropy regularization
Computing the remainder-reverse KL objective using our approach leads to the well-known although usually ad-hoc entropy regularization (e.g. Locatello et al.
(2018a)).
[Equation]
As we can see, while the ﬁrst term is the mean of the log-residual under the new component f i , the typical objective for gradient boosting, the second term is the entropy of f i .
E.3. SNIS derivation
Since we do not assume to know the normalization constant of p , we shall approximate the above quantities by self-normalized importance sampling while making the distinction between the normalized p and the un-normalized ˆ p :
[Equation]
E.4. Gradients of mixture weights
For forward KL:
[Equation]
For reverse KL:
[Equation]
E.5. The functional gradient of the forward KL divergence
We assume supp p ⊆ supp q : that is, p is absolutely continuous with respect to the variational approximation q i which can be ensured by the design of the variational family Q .
Let D ( q ) =
KL( p (cid:107) q ).
Functional gradient δDδq can be computed from the Taylor expansion of the KL functional Friedman (2001) as follows:
[Equation]
As detailed in the App.
E.
[Equation]
We have the logarithmic inequality xx +1 ≤ log (1 + x ) ≤ x ∀ x > − 1 where we can substitute (cid:15) hq > 0 for x and arrive at
[Equation]
By the monotone convergence theorem we can take the limit inside the integral and arrive at
[Equation]
E.6. Boosting convergence analysis
For a convex and strongly smooth functional, the greedy sequential approximation framework of Zhang (2003) provides an asymptotic guarantee for the convergence to a target distribution in the convex hull of the base family at a rate of O (1 /K ) where K is the number of boosting iterations.
This framework does not require each iteration to exactly solve for the optimal mixture component which can be diﬃcult in variational inference.
While the convexity of KL( p (cid:107) q ) in q is well established in the literature (proven with the log-sum inequality) for the forward KL divergence functional, we can show that FKL is also β -smooth in q where β depends on the maximum and minimum values that the density q can take.
To establish strong smoothness, on the other hand, stricter assumptions about the densities are necessary.
If we assume that all densities are bounded away from 0 and from above q 1 then for any pair of densities q 1 and q 2 there exists a β = sup p q 1 ∗ q 2 ≥ 0 such that the functional gradient δDδq is β - Lipschitz, that is (cid:12) (cid:12)(cid:12) δDδq ( q 2 ) − δDδq ( q 1 ) (cid:12) (cid:12)(cid:12)
≤ β | q 2 − q 1 | .
We can verify this choice of β :
[Equation]
Note that the boundedness assumptions are not unrealistic in practice and can translate to a bounded parameter space for a given family of distributions.
[Equation]
Appendix F. Supporting Figures
Minimize Reverse KL: KL( q (cid:107) p ) =
E q [log qp ] − − − −
[Figure]
Zero [Figure] p q
Minimize Forward KL: KL( p (cid:107) q ) =
E p [log pq ] − − − −
[Figure]