R EFINING THE VARIATIONAL POSTERIOR THROUGH ITERATIVE OPTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Variational inference (VI) is a popular approach for approximate Bayesian inference that is particularly promising for highly parameterized models such as deep neural networks.
A key challenge of variational inference is to approximate the posterior over model parameters with a distribution that is simpler and tractable yet sufﬁciently expressive.
In this work, we propose a method for training highly ﬂexible variational distributions by starting with a coarse approximation and iteratively reﬁning it.
Each reﬁnement step makes cheap, local adjustments and only requires optimization of simple variational families.
We demonstrate theoretically that our method always improves a bound on the approximation (the Evidence
Lower BOund) and observe this empirically across a variety of benchmark tasks.
In experiments, our method consistently outperforms recent variational inference methods for deep learning in terms of log-likelihood and the ELBO.
We see that the gains are further ampliﬁed on larger scale models, signiﬁcantly outperforming standard VI and deep ensembles on residual networks on CIFAR10.
I NTRODUCTION
Uncertainty plays a crucial role in a multitude of machine learning applications, ranging from weather prediction to drug discovery.
Poor predictive uncertainty risks potentially poor outcomes, especially in domains such as medical diagnosis or autonomous vehicles where some forms of high conﬁdence errors may be especially costly (Amodei et al., 2016).
Thus, it is becoming increasingly important that the underlying model provides high quality uncertainty estimates along with its predictions.
Yet, possibly the most widely used models, deep neural networks (LeCun et al., 2015), are unable to accurately quantify model uncertainty.
They are often overconﬁdent in their predictions, even when their predictions are incorrect (Guo et al., 2017; Ovadia et al., 2019).
By marginalizing over a posterior distribution over the parameters given the training data, Bayesian inference provides a principled approach to capturing uncertainty.
In contrast, standard training of neural networks employs a point estimate of the parameters, which cannot account for model uncertainty.
Unfortunately, exact Bayesian inference is intractable in general for neural networks.
To model epistemic uncertainty, variational inference (VI) instead approximates the true posterior with a simpler distribution.
The most widely used one for neural networks is the mean-ﬁeld approximation, where the posterior is represented using an independent Gaussian distribution over all the weights.
Variational inference is appealing since it reduces the problem of inference to an optimization problem, minimizing the discrepancy between the true posterior and the variational posterior.
The key challenge, however, is the task of training expressive posterior approximations that can capture the true posterior without signiﬁcantly increasing the computational costs.
This paper describes a novel method for training highly ﬂexible posterior approximations.
The idea is to start with a coarse, mean-ﬁeld approximation q ( w ) and make iterative, local reﬁnements to it.
The regions of the local reﬁnements are determined by sampling the values of additive auxiliary variables.
The model parameters w are expressed using a number of auxiliary variables a k (Figureleft) for k = 1 , . .
, K that leave the marginal distribution unchanged.
In each iteration, we sample the value of an auxiliary variable according to the current variational approximation q ( a k ) and reﬁne the approximation by conditioning on the newly sampled value q ( w )
≈ p ( w | x, y, a 1: k ) (Figureright illustrates the process).
Each reﬁnement step makes cheap, local adjustments to the variational posterior in the region of the sampled auxiliary variables.
At the end, we draw one sample from
N x w y
N x y aaw w=a 1 +a 2 +aaw aa 1 +a the reﬁned q ( w ) .
The reﬁnement iterations have to be repeated for each posterior sample.
The algorithm results in samples from a highly complex distribution, starting from a simple mean-ﬁeld approximation.
While the distribution of the samples is difﬁcult to quantify, we show that it is not limited to factorized, uni-modal forms, and that the procedure is guaranteed to improve the resulting
ELBO without posing a signiﬁcant computational overhead.
Summary of contributions:
• A novel algorithm for reﬁning a variational distribution, increasing its ﬂexibility.
• We show that the reﬁnement steps are guaranteed to improve the quality of the variational distribution under mild conditions.
• We showcase the effectiveness of the method on Bayesian neural networks using a set of UCI regression and image classiﬁcation benchmarks.
We set a new state-of-the-art in uncertainty estimation using variational inference at ResNet scale (ResNet-20, (He et al., 2016)) scale on CIFAR10.
M ETHODS
In this section, we ﬁrst describe standard variational inference (VI), followed by a detailed description of the iterative reﬁnement algorithm.
While VI and our proposed methodology are generally applicable to latent variable models, in this work, we consider the application to Bayesian neural networks (Figure 1), where x are inputs, y outputs, and w the weights of network.
2. V ARIATIONAL I NFERENCE IN B AYESIAN N EURAL N ETWORKS
Exact Bayesian inference in Bayesian neural networks (BNN) (Figure 1) is often intractable and is NP-hard in the general case.
Variational inference attempts to approximate the true posterior p ( w | x, y ) with an approximate posterior q φ ( w ) , typically from a simple family of distributions, for example independent Gaussians over the weights, i.e. the mean-ﬁeld approximation.
To ensure that the approximate posterior is close to the true posterior, the parameters of q φ ( w ) , φ are optimized to minimize their Kullback-Leibler divergence: D KL ( q φ ( w ) (cid:107) p ( w | x, y )) .
At the limit of D KL ( q φ ( w ) (cid:107) p ( w | x, y ))
= 0 , the approximate posterior exactly captures the true posterior, although this might not be achievable if p ( w | x, y ) is outside of the distribution family of q φ ( w ) .
In order to minimize the KL-divergence, variational inference optimizes the Evidence Lower Bound (ELBO) w.r.t.
φ (denoted as L ( φ ) ), which is a lower bound to the log marginal likelihood log p ( y | x ) .
Since the the marginal likelihood can be expressed as the sum of the KL-divergence and the ELBO, maximizing the ELBO is equivalent to minimizing the KL divergence:
[Equation]
due to non-negativity of the KL-divergence.
1: procedure R EFINE
A ND S AMPLE ( φ ) 2: for m = 1 , . . .
, M do 3: φ 0 ← φ 4: for k = 1 , . . .
, K do 5: a k ∼ q φ k − 1 ( a k ) (cid:46)
Sample a k 6: q φ k ( w ) ← q φ k − 1 ( w | a k ) (cid:46) Initialize φ k 7: φ k ← arg max φ k L | a 1: k ( φ k ) (cid:46)
Optimize φ k 8: end for 9: w m ∼ q φ K ( w ) (cid:46)
Sample the reﬁned posterior 10: end for 11: return w 1: M 12: end procedure
[Figure]
Following the optimization of φ , the model can be used to make predictions on unseen data.
For a new input x (cid:48) , the predictive distribution p ( y (cid:48) | x (cid:48) , y, x ) can be approximated by stochastically drawing a small number (around M ≤ 10 ) of sample model parameters and averaging their prediction in an ensemble model:
[Equation]
2. R EFINING THE POSTERIOR
The main issue with variational inference is the inﬂexibility of the posterior approximation.
The most widely used variant of variational inference, mean-ﬁeld variational inference, approximates the posterior with independent Gaussians across all dimensions.
This approximation is too simplistic to capture the complexities of the posterior for complicated models such as BNNs.
Our idea is to reﬁne the posterior approximation through iterative optimization.
Using the reﬁnement procedure, it is feasible to train a detailed posterior in the regions of the posterior samples used for prediction while relying on a coarse-grained approximation further away from these samples.
The graphical model is augmented with a ﬁnite number of auxiliary variables a 1: K as shown in
Figure 1.
The constraints are that ( x, y ) must be conditionally independent of the auxiliary variables given w and that they must not affect the prior distribution p ( w ) .
This is important in justifying the use of the initial variational approximation.
A signiﬁcant way in which we distinguish ourselves from hierarchical variational models (Ranganath et al., 2016) is that the model is unaffected by the presence of the auxiliary variables .
Their purpose is solely to aid the inference procedure.
Given a
Gaussian prior N (0 , σ 2 w ) over w , we express w as a sum of independent auxiliary variablesw =
K (cid:88) k = a k , with p ( a k ) =
N (0 , σ 2 a k ) for k = 1 , . . .
, K , while ensuring that (cid:80) Kk
=1 σ 2 a k = σ 2 w so that the prior p ( w ) =
N (0 , σ 2 w ) is unchanged.
Locally reﬁning the approximate posterior refers to iteratively sampling the values of the auxiliary variables a 1: K and then approximating the posterior conditional on the sampled values, i.e. q φ k ( w ) approximates p ( w | x, y, a 1: k ) for iterations k = 1 , . .
, K (Algorithm 1).
Starting from the initial mean ﬁeld approximation q φ ( w ) , we sample the value of a 1 from q φ ( a 1 ) =
(cid:82) p ( a 1 | w ) q φ ( w ) d w ,
While we are focusing on one speciﬁc deﬁnition of the auxiliary variables, additive auxiliary variables, note that all of our results straight-forwardly generalize to arbitrary joint distributions p ( w, a 1: K ) that meet the constraints.
The true posterior p ( w ) p ( w | x,y ) (a)  1  3 0.
The approximate posterior p ( w | x ) q φ ( w ) (b)  1  3 0.
a
Variational posterior conditional on a 1 q φ 1 ( w ) p ( w | x,y,a 1 ) (c)  1  3 0.
1.0 Histogram of the weight samples p ( w | x,y ) w 1: M ( [Figure] then optimize the approximation to the conditional posterior: q φ 1 ( w ) ≈ p ( w | x, y, a 1 ) .
This procedure is then iteratively repeated for a 2: K .
In iteration k ,
[Equation]
k ∼
[Equation]
Analogously to variational inference, the KL divergence is minimized through the optimization of the conditional ELBO in each iteration:
L | a 1: k ( φ k ) =
E q φk (cid:2) log p ( y | x, w ) (cid:3) −
D KL ( q φ k ( w ) (cid:107) p ( w | a 1: k )) .
In order to get independent samples from the variational posterior, we have to repeat the iterative reﬁnement for each ensemble member w 1: M .
Toy example We use a toy example to demonstrate the procedure.
In this toy example, we have a single weight w with prior p ( w ) =
N (0 , 1) and a complicated posterior with four modes (synthetically generated data).
We express w as the sum of two auxiliary variables w = a 1 + a 2 with p ( a 1 ) =
N (0 , 0 .
8) and p ( a 2 ) =
N (0 , 0 . 2) (which recovers p ( w ) as per the constraint).
As Figure 3b shows, a Gaussian approximation to the posterior fails to capture the multimodal nature of the true posterior.
The ﬁrst step of the reﬁnement is to sample using q φ : a 1 ∼ q φ ( a 1 ) =
(cid:82) p ( a 1 | w ) q φ ( w ) d w .
Next, we condition on the value of a 1 and initialize the parameters of the variational posterior accordingly: q φ 1 ( w ) =
N (cid:0) w | µ φ 1 , σ 2 φ 1 (cid:1) ← q φ ( w | a 1 ) .
Both q φ ( a 1 ) and q φ ( w | a 1 ) can be computed analytically for Gaussian distributions.
After optimizing φ 1 , the approximate posterior q φ 1 ( w ) is able to capture a good, local approximation to the posterior p ( w | a 1 , x, y )
(Figure 3c).
In Figure 3d, we can see the histogram of the reﬁned posterior, that is, the distribution we are generating samples from (for each sample from w , we drew a sample from q φ ( a 1 ) and optimized q φ 1 ( w ) ).
Clearly, it is a much better approximation to the true posterior than the Gaussian approximation we started with, although it is important to note that the true posterior is not recovered exactly.
2. T HEORETICAL JUSTIFICATION
Our theoretical claims are twofold.
Firstly, that through this procedure, we are optimizing a lower bound to the ELBO and secondly, that the reﬁnement cannot result in a worse posterior approximation than the initial mean-ﬁeld approximation that we start with (in the ELBO sense).
That is,
[Equation]
where ELBO ref denotes the ELBO of the reﬁned posterior q ref , ELBO aux refers to the objective that the reﬁnement process is optimizing, and ELBO init is the ELBO of the initial mean-ﬁeld approximation.
Lower bound to the ELBO Consider the case with two auxiliary variables a 1 and a 2 .
The initial training optimizes the ELBO init =
E q φ (cid:2) log p ( y | x, w ) (cid:3) − D KL (cid:0) q φ ( w ) (cid:13)(cid:13) p ( w ) (cid:1) and the reﬁnement step optimizes the conditional ELBO, L | a 1 ( φ 1 ) =
E q φ 1 (cid:2) log p ( y | x, w ) (cid:3) − D KL (cid:0) q φ 1 ( w ) (cid:13)(cid:13) p ( w | a 1 ) (cid:1) .
The key observation is that we can deﬁne ELBO aux that is a lower bound to ELBO ref and is increased both by the initial training and the reﬁnement steps:
[Equation]
since the KL divergence of the joint distribution is greater than or equal to that of the marginals.
Guarantee of improvement Improvement in the ELBO ( ELBO ref ≥ ELBO init ) is guaranteed under two assumptions.
First, that the conditional variational posterior, q φ ( w | a 1 ) , is within the variational family of q φ 1 .
Second, that the process that optimizes φ 1 does not make it worse than the value it was initialized with.
The ﬁrst assumption holds for Gaussian families: q φ ( w | a 1 ) is
Gaussian and can be computed in closed form.
The second assumption is reasonable to assume for most optimizers and, in addition, it can be ensured by comparing the initial value to the ﬁnal value and choosing the one with the more desirable objective value.
The argument goes as follows.
By initializing q φ 1 such that it coincides with q φ ( w | a 1 ) , we can ensure that ELBO aux ≥ ELBO init , since they are equal at initialization time and the optimization process does not decrease ELBO aux .
From this combined with our previous result, it follows that
[Equation]
therefore ensuring that the ELBO improves through the reﬁning steps.
Note that this also implies that it is not necessary to optimize until convergence: any amount of optimization increases the ELBO.
Figure 2 shows that the ELBO improvement occurs on real world datasets.
With the sampling of each auxiliary variable, the ELBO improves forming a staircase pattern.
Table 1 and 2 serve as further empirical evidence that the ELBO improves as a result of the reﬁnement steps.
Extending to multiple auxiliary variables For simplicity, we stated the arguments with two auxiliary variables, but they straight-forwardly extend to any ﬁnite number of auxiliary variables.
In this scenario, there are K ELBO aux -s, one for each auxiliary variable, upper bounded by ELBO ref and lower bounded by ELBO init .
R ELATED WORKS
While in theory, the Bayesian approach can accurately capture uncertainty, in practice we ﬁnd that exact inference is computationally infeasible in most scenarios and thus we have to resort to approximate inference methods.
There is a wealth of research on approximate inference methods, here we focus on works closely related to this paper.
Variational inference (Hinton & Van Camp, 1993) tries to approximate the true posterior distribution over parameters with a variational posterior from a simple family of distributions.
Mean-ﬁeld VI, which for neural networks traces back to Peterson (1987), uses independent Gaussian distributions over the parameters to try to capture the posterior.
The advantage of the mean-ﬁeld approximation is that the network can be efﬁciently trained using the reparameterization trick (Kingma & Welling, 2013) and the variational posterior has a proper density over the parameter space which then can be used across tasks like continual learning (Osawa et al., 2019; Nguyen et al., 2017) and contextual bandits (Riquelme et al., 2018).
Recently, Louizos & Welling (2017) showed that normalizing ﬂows can be employed to further increase the ﬂexibility of the variational posterior.
Zhang et al.
(2018a) provide a detailed survey of recent advances in VI.
Our method is a novel variant of the auxiliary variable approaches to VI (Agakov & Barber, 2004;
Ranganath et al., 2016) that increase the ﬂexibility of the variational posterior through the use of auxiliary variables.
The key distinction, however, is that instead of trying to train a complex variational approximation over the joint distribution, we iteratively train simple, mean-ﬁeld approximations at the sampled values of the auxiliary variables.
While this poses an O ( MK ) ( K is the number of auxiliary variables and M is the number of posterior samples) overhead over mean-ﬁeld VI, the training itself is kept straightforward and efﬁcient.
The introduction of every new auxiliary variable increases the ﬂexibility of the posterior approximation.
In contrast to MCMC methods, it is unclear whether the algorithm approaches the true posterior in the limit of inﬁnitely many auxiliary variables.
There are also numerous methods that start with an initial variational approximation and reﬁne it through a few MCMC steps (Salimans et al., 2015; Zhang et al., 2018b; Ruiz & Titsias, 2019).
The distinction from our algorithm is that we reﬁne the posterior starting at large scale and iteratively move towards smaller scale reﬁnements whereas these methods only reﬁne the posterior at the scale of the MCMC steps. Guo et al.
; Miller et al.
(2017) and Locatello et al.
(2018) used boosting to reﬁne the variational posterior, iteratively adding parameters such as mixture components to minimize the residual of the ELBO.
Our method does not add parameters at training time but instead iteratively reﬁnes the samples through the introduction of auxiliary variables.
We did not include these models among our baselines because they have yet to be applied to Bayesian multi-layer neural networks.
Further related works include methods that iteratively reﬁne the posterior in latent variable models (Hjelm et al., 2016;
Cremer et al., 2018;
Kim et al., 2018; Marino et al., 2018).
These methods, however, focus on reducing the amortization gap and they do not increase the ﬂexibility of the variational approximation.
Lastly, there are non-Bayesian strategies for estimating epistemic uncertainty in deep learning.
Bootstrapping (Breiman, 1996) and deep ensembles (Lakshminarayanan et al., 2017) are perhaps the most promising.
Deep ensembles, in particular, have been demonstrated to achieve strong performance on benchmark regression and classiﬁcation problems and uncertainty benchmarks including out-of-distribution detection (Lakshminarayanan et al., 2017) and prediction under distribution shift (Ovadia et al., 2019).
Both methods rely on constructing a set of independently trained models to estimate the uncertainty.
Intuitively, the amount of disagreement across models reﬂects the uncertainty in the ensemble prediction.
To induce diversity among the ensemble members, bootstrapping subsamples the training set for each member while deep ensembles use the randomness in weight initialization and mini-batch sampling.
E XPERIMENTS
The goal of the experiments is twofold.
First, we empirically conﬁrm the improvement in the ELBO as claimed in Section 2.3.
Second, we quantify the improvement in the uncertainty estimates due to the reﬁnement.
We conducted experiments on a selection of regression and classiﬁcation benchmarks using Bayesian neural networks as the underlying model.
We looked at the marginal loglikelihood of the predictions as well as accuracy in classiﬁcation tasks.
Reﬁnement (Reﬁned VI) After training the initial mean-ﬁeld approximation, we reﬁne M =ensemble members, each with K = 5 auxiliary variables.
The means on their prior distributions are ﬁxed at 0., and their variances form a geometric series (each auxiliary variable has variance 0.3 times the previous one, which roughly halves the standard deviation of the prior each time): σ 2 a 1 = 0 .
7 σ 2 w , σ 2 a 2 = 0 .
21 σ 2 w , σ
a 3 = 0 .
063 σ 2 w , σ
a 4 = 0 .
σ 2 w , and σ 2 a 5 = 0 .
σ 2 w .
In each reﬁnement iteration, we optimized the posterior with Adam (Kingma & Ba, 2014) for 200 iterations.
To keep the training stable, we scaled the learning rate according to the standard deviation of the conditional posterior: in iteration k , lr= 0 .
3 k 2 0 .
4. B ASELINES
We used three baselines.
First, mean-ﬁeld variational inference in order to quantify the improvement provided by reﬁning.
Second, multiplicative normalizing ﬂows (MNF) to compare against a more ﬂexible posterior approximation and third, deep ensemble models to compare against a state-of-theart non-Bayesian approach.
For all methods we used a batch size of 256 and the Adam optimizer with the default learning rate of 0.001 3 .
of each baseline were tuned using a Bayesian optimization package, however we found batch size and learning rate to be consistent across methods.
Variational inference (VI) (Graves, 2011; Blundell et al., 2015)
Naturally, we investigate the improvement over variational inference with a mean-ﬁeld Gaussian posterior approximation.
We do inference over all weights and biases with a Gaussian prior centered at 0.
The variance of the prior is tuned through empirical Bayes.
This model is trained for 30000 iterations.
Multiplicative Normalizing Flows (MNF), (Louizos & Welling, 2017)
To measure the performance against a more ﬂexible class of posterior approximations, we look at Multiplicative Normalizing Flows.
In this work, the posterior means are augmented with a multiplier from a ﬂexible distribution parameterized by the masked RealNVP.
This model is trained with the default ﬂow parameters for 30000 iterations.
Deep ensemble models, (Lakshminarayanan et al., 2017)
Deep ensemble models are shown to be surprisingly effective at quantifying uncertainty.
While they are less principled than Bayesian methods, they are certainly a competitive baseline.
For the regression datasets, we used adversarial training ( (cid:15) = 0 . 01 ) whereas in classiﬁcation we did not use adversarial training (adversarial training did not give an improvement on the classiﬁcation benchmarks).
For each dataset, we trainedensemble members for 5000 iterations each.
4. R EGRESSION BENCHMARKS
Following Hern ´ andez-Lobato & Adams (2015), we evaluate the methods on a set of UCI regression benchmarks (Table 1).
The datasets used a random 80-20 split for training and testing.
The model used for these datasets is a feed forward neural network with a single hidden layer containing 50 units with a ReLU activation function.
As it is common practice, we utilize the local reparameterization trick (Kingma et al., 2015).
On these benchmarks, we are able to conﬁrm that the reﬁnement step consistently improves both the
ELBO and the uncertainty estimates over VI.
On 7 out of the 9 datasets, Reﬁned VI is one of the best performing approaches. [Figure]
[Table]
Metrics: marginal log-likelihood (MLL, higher is better), and lower bound (ELBO higher is better).
The mean values and standard deviations are in the table.
4. C LASSIFICATION BENCHMARKS
We examine the performance on commonly used image classiﬁcation benchmarks (Table 2).
The architecture used for this experiment is the LeNet5 (LeCun et al., 1995) architecture containing three convolutional layers with 6, 16 and 120 channels respectively followed by a feed-forward layer with 84 units and an output layer with ReLu activations throughout the network.
We use the local reparameterization trick (Kingma et al., 2015) for the dense layers and
Flipout (Wen et al., 2018) for the convolutional layers to reduce the gradient noise.
On the classiﬁcation benchmarks, we again are able to conﬁrm that the reﬁnement step consistently improves both the ELBO and the uncertainty estimates over VI.
While Reﬁned VI is unable to outperform Deep Ensembles in classiﬁcation accuracy, it does outperform them in MLL on the largest dataset, CIFAR10. [Table]
4. L ARGE SCALE MODELS
To demonstrate the performance on larger scale models, we apply the reﬁning algorithm to Residual
Networks (He et al., 2016) with 20 layers (based on Keras’s ResNet implementation).
We look at two models.
Firstly, a model where we do inference over all of the residual blocks and secondly, following Ovadia et al.
(2019), a hybrid model (ResNet Hybrid) where inference is only done over the ﬁnal layer of each residual block and every other layer is treated as a regular layer.
For this model, we used a batch-size of 256 and we decayed the learning rate starting from 0.001 over 200 epochs.
We used 10 auxiliary variables each reducing the prior variance by a factor of 0.5.
Furthermore we investigate the effect of Batch Normalization (Ioffe & Szegedy, 2015).
While it is difﬁcult to incorporate batch normalization into the Bayesian framework, its positive effects are undeniable.
Regarding batch normalization, we can conﬁrm the ﬁndings of Osawa et al.
(2019), that it provides a substantial improvement for VI, although interestingly, this improvement disappears in the case of the hybrid model.
To our knowledge, the reﬁned hybrid model is state-of-the-art in terms of MLL.
It outperforms Osawa et al.
(2019) in both MLL and accuracy.
[Table]
4. C OMPUTATIONAL COSTS
When introducing a novel algorithm for variational inference, we have to discuss the computational costs.
Table 4 shows the wall-time required to train each model including the possibility of parallel training.
Deep ensembles require fewer training iterations and parallelize very well, although it is important to note that the system we tested on is heavily optimized towards training these models both in hardware and software.
For Reﬁned VI, training the initial mean-ﬁeld approximation cannot be parallelized, but the following reﬁnement iterations can be straight-forwardly split into M threads.
[Table]
4. C ONCLUSIONS
In this paper we describe a novel algorithm for reﬁning a coarse variational approximation to the
Bayesian posterior.
We show, both theoretically and empirically, that the reﬁned posterior is a better approximation to the posterior than the initial variational distribution.
Our method outperforms the baseline variational approximations in both uncertainty estimation as well as computational requirements.
It sets a new state-of-the-art in uncertainty estimation using variational inference at ResNet scale (ResNet-20) on CIFAR10.
A C LOSED FORMS OF THE SAMPLING DISTRIBUTIONS
In Section 2.2, we claim that q φ ( a 1 ) (used to sample a 1 ) and q φ ( w | a 1 ) (used to initialize q φ 1 ) have analytic solutions for additive Gaussians.
For p ( w ) =
N (0 , σ 2 w ) , q ( w ) = N ( µ φ , σ 2 φ ) , p ( a 1 ) =
N (0 , σ 2 a 1 ) and p ( a 2 ) =
N (0 , σ 2 a 2 ) ,
[Equation]
These are derived by applying Bayes rule to the Gaussian probability density functions.
B N OTE ON ELBO AUX ≥ ELBO INIT
We formally show that ELBO aux ≥ ELBO init under the assumption that the conditional variational posterior, q φ ( w | a 1 ) , is within the variational family of q φ 1 for all values of a 1 .
For a given a 1 , let φ 1 0 | a 1 be a set of variational parameters such that ∀ w, q φ 10 | a 1 ( w )
=
q φ ( w | a 1 ) .
Such φ 1 0 | a 1 must exist as a result of the initial assumption.
The formula for Gaussian distributions is given in Appendix A.
Set φ 1 | a 1 4 such that
[Equation]

[Equation]
if
[Equation]
φ
[Equation]
opt |
[Equation]
otherwise (2) where φ 1 opt | a 1 is the result of the stochastic optimizer that attempts to maximize
E q
[Equation]
1 |
[Equation]
[Equation]
By construction, for all a 1 ,
[Equation]
Substituting this into ELBO aux gives
[Equation]
concluding the proof.