T HE D EEP W EIGHT P RIOR
Anonymous authors
Paper under double-blind review
A BSTRACT
Bayesian inference is known to provide a general framework for incorporating prior knowledge or speciﬁc properties into machine learning models via carefully choosing a prior distribution.
In this work, we propose a new type of prior distributions for convolutional neural networks, deep weight prior, that in contrast to previously published techniques, favors empirically estimated structure of convolutional ﬁlters e.g., spatial correlations of weights.
We deﬁne deep weight prior as an implicit distribution and propose a method for variational inference with such type of implicit priors.
In experiments, we show that deep weight priors can improve the performance of Bayesian neural networks on several problems when training data is limited.
Also, we found that initialization of weights of conventional networks with samples from deep weight prior leads to faster training.
I NTRODUCTION
Bayesian inference is a tool that, after observing training data, allows to transform a prior distribution over parameters of a machine learning model to a posterior distribution.
Recently, stochastic variational inference (Hoffman et al., 2013) – a method for approximate Bayesian inference – has been successfully adopted to obtain a variational approximation of a posterior distribution over weights of a deep neural network (Kingma et al., 2015).
Currently, there are two major directions for the development of Bayesian deep learning.
The ﬁrst direction can be summarized as the improvement of approximate inference with richer variational approximations and tighter variational bounds (Dikmen et al., 2015).
The second direction is the design of probabilistic models, in particular, prior distributions, that widen the scope of applicability of the Bayesian approach.
Prior distributions play an important role for sparsiﬁcation (Molchanov et al., 2017; Neklyudov et al., 2017), quantization (Ullrich et al., 2017) and compression (Louizos et al., 2017) of deep learning models.
Although these prior distributions proved to be helpful, they are limited to fullyfactorized distributions.
Thus, the often observed spatial structure of convolutional ﬁlters cannot be enforced with such priors.
Convolutional neural networks are an example of the model family, where a correlation of the weights plays an important role, thus it may beneﬁt from more ﬂexible prior distributions.
In this work, based on the fact that convolutional neural networks tend to learn similar convolutional kernels on different datasets from similar domains (Sharif Razavian et al., 2014; Yosinski et al., 2014), we propose a method for building prior distributions for convolutional neural networks.
For a speciﬁc data domain, we consider the distribution of kernels of trained convolutional networks.
In the rest of the paper, we refer to this distribution as the source kernel distribution .
Our main assumption is that within a speciﬁc domain the source kernel distribution can be efﬁciently approximated with kernels of models trained on a small subset of tasks from this domain.
For example, we expect that a good approximation for the source kernel distribution of models trained on NotMNIST – a dataset of grayscale images, will be a good approximation for source kernel distribution of models trained on MNIST dataset.
However, some difﬁculties appear due to an inability to access the density function of the source kernel distribution.
In Section 3 we propose a technique to overcome this problem.
Our contributions can be summarized as follows:
We propose deep weight prior , a framework that approximates the source kernel distribution and incorporates prior knowledge about the structure of convolutional ﬁlters into the prior distribution.
We also propose to use an implicit form of this prior (Section 3.1).
We develop a method for variational inference with the proposed type of implicit priors (Section 3.2).
In experiments (Section 4), we show that variational inference with deep weight prior signiﬁcantly improves classiﬁcation performance upon a number of popular prior distributions in the case of limited training data.
We also ﬁnd that just initialization of the conventional convolution networks with samples from a deep weight prior leads to faster convergence and better random feature extraction.
D EEP B AYES
In a Bayesian setting, after observing a dataset D = { x 1 , . . .
, x N } of N points, the goal is to transform our prior knowledge p ( ω ) of the unobserved distribution parameters ω to the posterior distribution p ( ω | D ) .
However, computing a posterior distribution through Bayes rule p ( ω | D ) =
p (
D | ω ) p ( ω ) /p
( D ) may involve computationally intractable integrals.
This problem, nonetheless, can be solved approximately.
Variational Inference (Jordan et al., 1999) is one of such approximation methods.
It reduces the inference to an optimization problem, where we optimize parameters θ of a variational approximation q θ ( ω ) , so that KL-divergence between q θ ( ω ) and p ( ω | D ) is minimized.
This divergence in practice is minimized by maximizing so-called variational lower bound L ( θ ) of the marginal log-likelihood of the data w.r.t parameters θ of the variational approximation q θ ( W ) .
[Equation]
(1) where L D = E q θ ( ω ) log p ( D | ω ) (2)
The variational lower bound L ( θ ) consists of two terms: 1) the (conditional) expected log likelihood
L D , and 2) the regularizer D KL ( q θ ( ω ) (cid:107) p ( ω )) .
Since log p ( D )
=
L ( θ )+ D KL ( q θ ( ω ) (cid:107) p ( ω | D )) and p ( D ) does not depend on q θ ( w ) maximizing of L ( θ ) minimizes D KL ( q θ ( ω ) (cid:107) p ( ω | D )) .
However, in case of intractable expectations in equation 1 neither the variational lower bound
L ( θ ) nor its gradients can be computed in close form.
Recently, Kingma & Welling (2013) and Rezende et al.
(2014) proposed an efﬁcient mini-batch based approach to stochastic variational inference, so-called stochastic gradient variational Bayes or doubly stochastic variational inference .
The idea behind this framework is reparamtetrization, that represents samples from a parametric distribution q θ ( ω ) as a deterministic differentiable function ω
= f
( θ, (cid:15) ) of parameters θ and an (auxiliary) noise variable (cid:15) ∼ p ( (cid:15) ) .
Using this trick we can efﬁciently compute an unbiased stochastic gradient ∇ θ L of the variational lower bound w.r.t the parameters of the variational approximation.
Bayesian Neural Networks.
The stochastic gradient variational Bayes framework has been applied to approximate posterior distributions over parameters of deep neural networks (Kingma et al., 2015).
We consider a discriminative problem, where dataset D consists of N object-label pairs
D = { ( x i , y i ) } Ni =1 .
For this problem we maximize the variational lower bound L ( θ ) with respect to parameters θ of the variational approximation:
[Equation]
where W denotes weights of a neural network, q θ ( W ) is a variational distribution, that allows reparametrization (Kingma & Welling, 2013;
Figurnov et al., 2018) and p ( W ) is a prior distribution.
In a simplest case q θ ( W ) can be a fully-factorized normal distribution.
However, more expressive variational approximations may lead to better quality of variational inference (Louizos &
Welling, 2017; Yin & Zhou, 2018).
Typically, Bayesian neural networks use fully-factorized normal or log-uniform priors (Kingma et al., 2015; Molchanov et al., 2017; Louizos & Welling, 2017).
Variational Auto-encoder.
Stochastic gradient variational Bayes has also been applied for building generative models.
The variational auto-encoder proposed by Kingma & Welling (2013) maximizes a variational lower bound L ( θ, φ ) on the marginal log-likelihood by amortized variational inference:
[Equation]
where inference model q θ ( z i | x i ) approximates posterior distribution over local latent variables
z i , reconstruction model p φ ( x i
| z i ) transforms the distribution over latent variables to a conditional distribution in object space and a prior distribution over latent variables p
( z i ) .
The vanilla VAE deﬁnes q θ ( z | x ) , p φ ( x | z ) , p ( z ) as fully-factorized distributions, however, a number of richer variational approximations and prior distributions have been proposed (Rezende & Mohamed, 2015; Kingma et al., 2016; Tomczak & Welling, 2017).
The approximation of the data distribution can then be deﬁned as an intractable integral p ( x ) ≈ (cid:82) p φ ( x | z ) p ( z ) dz which we will refer to as an implicit distribution.
D EEP W EIGHT P RIOR
We propose the deep weight prior – an expressive prior distribution which allows us to encode the information about the interdependence of the learned convolutional ﬁlters.
In this section, we consider a neural network with L convolutional layers.
We denote parameters of l -th convolutional layer as w l ∈ R
I l
× O l × H l × W l , where I l is the number of input channels, O l is the number of output channels, H l and W l are spatial dimensions of kernels.
Parameters of the neural network are denoted as W = ( w 1 , . . .
w L ) .
We consider a variational approximation q θ ( W ) and a prior distribution p ( W ) with the following factorization over layers, ﬁlters and channels:
[Equation]
where w lij ∈ R H l × W l is a kernel of j -th channel in i -th ﬁlter of l -th convolutional layer.
We also assume that q θ ( W ) allows reparametrization.
For a speciﬁc domain, we deﬁne the source kernel distribution – a distribution of trained convolutional kernels of the l -th convolutional layer.
The source kernel distribution favors learned kernels, and thus it is a very natural candidate to be a prior distribution p l ( w lij ) for convolutional kernels of the l -th layer.
Unfortunately, we do not have access to its probability density function.
We assume that the p.d.f.
of the source kernel distribution can be approximated using kernels of models trained on external datasets from the same domain.
The external datasets do not overlap with the dataset under consideration.
For example, the approximation of the source kernel distribution obtained on models that were learned on NotMNIST dataset is a good approximation for the source kernel distribution of models learned on MNIST dataset.
In the next subsection, we propose to approximate this intractable probability density function of the source kernel distribution using the framework of generative models.
3. M ODEL OF P RIOR D ISTRIBUTION
In this section, we discuss explicit and implicit approximations ˆ p l ( w ) of the probability density function p l ( w ) of the source kernel distribution of l -th layer.
We assume to have a trained convolutional neural network, and treat kernels from the l -th layer of this network
w lij ∈ R
H l × W l as samples from the source kernel distribution of l -th layer p l ( w ) .
Explicit models.
A number of approximations allow us to evaluate probability density functions explicitly.
Such families include but are not limited to Kernel Density Estimation (Silverman, 1986),
Normalizing Flows (Rezende & Mohamed, 2015; Dinh et al., 2017) and PixelCNN (van den Oord et al., 2016).
For these families, we can estimate the KL-divergence D KL ( q ( w | θ lij ) (cid:107) ˆ p l ( w )) and its gradients without a systematic bias, and then use them for variational inference.
Despite the fact that these methods provide ﬂexible approximations, they usually demand high memory or computational cost (Louizos & Welling, 2017).
Implicit models.
Implicit models, in contrast, can be more computationally efﬁcient, however, they do not provide access to an explicit form of probability density function ˆ p l ( w ) .
We consider an approximation of the prior distribution p l ( w ) in the following implicit form:
[Equation]
where the conditional distribution p ( w | z ; φ l ) is an explicit parametric distribution and p l ( z ) is an explicit prior distribution that does not depend on trainable parameters.
Parameters of the conditional
Algorithm
Require: dataset D
= { ( x i , y i ) } Ni =1 Require: variational approximations q ( w | θ lij ) and reverse models r
( z | w ; ψ l )
Require: reconstruction models p ( w | z ; φ l ) , priors for auxiliary variables p l
( z ) while not converged do ˆ M ← mini-batch of objects form dataset D ˆ w lij ← sample weights from q ( w | θ lij ) with reparametrization ˆ z lij ← sample auxiliary variables from r ( z | ˆ w lij ; ψ l ) with reparametrization ˆ L aux ← L ˆ M + (cid:80) l,i,j − log q ( ˆ w lij | θ lij ) − log r (ˆ z lij | ˆ w lij ; ψ l )+log p l (ˆ z lij )
+log p ( ˆ w lij | ˆ z lij ; φ l )
Obtain unbiased estimate ˆ g with E [ˆ g ] =
∇L aux by differentiating ˆ L aux
Update parameters θ and ψ using gradient ˆ g and a stochastic optimization algorithm end while return Parameters θ , ψ distribution p ( w | z ; φ l ) can be modeled by a differentiable function g ( z ; φ l ) e.g. neural network.
Note, that while the conditional distribution p ( w | z ; φ l ) usually is a simple explicit distribution, e.g. fully-factorized Gaussian, the marginal distribution ˆ p l ( w ) is generally a more complex intractable distribution.
Parameters φ l of the conditional distribution p ( w | z ; φ l ) can be ﬁtted using the variational autoencoder framework.
In contrast to the methods with explicit access to the probability density, variational auto-encoders combine low memory cost and fast sampling.
However, we cannot compute the probability density function ˆ p l ( w ) and therefore cannot build an unbiased estimator of the variational lower bound (equation 3).
In order to overcome this limitation we propose a modiﬁcation of variational inference for implicit prior distributions.
V ARIATIONAL I NFERENCE W ITH I MPLICIT P RIOR D ISTRIBUTION
Stochastic variational inference approximates the true posterior distribution by maximizing the variational lower bound L ( θ ) (equation 1), which includes the KL-divergence D KL ( q ( W ) (cid:107) p ( W )) between a variational approximation q θ ( W ) and a prior distribution p ( W ) .
In case of simple prior and variational distributions (e
[Equation]
g. Gaussian), this KL-divergence can be computed in closed form or estimated without systematic bias.
Unfortunately, it does not hold anymore in case of an implicit prior distribution ˆ p ( W ) =
(cid:81) l,i,j ˆ p l ( w lij ) .
In this case, the KL-divergence becomes intractable and cannot be estimated without bias since the prior ˆ p ( W ) is intractable.
However, we can make the variational lower bound tractable again by upper bounding the KL-divergence:
[Equation]
where r ( z | w ; ψ l ) is an auxiliary inference model for the prior of l -th layer ˆ p l ( w )
, The ﬁnal auxiliary variational lower bound has a following form:
[Equation]
The lower bound L aux is tight if and only if the KL-divergence between the auxiliary reverse model and the intractable posterior distribution over latent variables z given w is zero (Appendix A).
In case when q θ ( w ) , p ( w | z ; φ l ) and r ( z | w ; ψ l ) are explicit parametric distributions which can be reparametrized, we can perform an unbiased estimation of a gradient of the auxiliary variational lower bound L aux ( θ, ψ ) (equation 8) w.r.t.
parameters θ of the variational approximation q θ ( W ) and parameters ψ of reverse models r ( z | w ; ψ l ) .
Then we can maximize the auxiliary lower bound w.r.t.
parameters of the variational approximation and reversed models L aux ( θ, ψ ) → max θ,ψ .
Note, that parameters φ of the prior distribution ˆ p ( W ) are ﬁxed during variational inference, in contrast to the
Empirical Bayesian framework (MacKay, 1992).
Algorithm 1 describes stochastic variational inference with an implicit prior distribution.
In case when we can calculate an entropy H ( q ) or the divergence D KL ( r ( z | w ; ψ l ) (cid:107) p l ( z )) explicitly, the
[Figure]
variance of the estimation of the gradient ∇ ˆ L aux ( θ, ψ ) can be reduced.
This algorithm can also be applied to an implicit prior that is deﬁned in the form of Markov chain:
[Equation]
where p ( z t +1 | z t ) is a transition operator (Salimans et al., 2015), see Appendix A. We provide more details related to the form of p ( w | z ; φ l ) , r ( z | w ; ψ l ) and p l ( z ) distributions in Section 4.
L EARNING D EEP W EIGHT P RIOR
In this subsection we explain how to train deep weight prior models for a particular problem.
Examples of samples from learned prior distribution are presented at Figure 1(c).
Source datasets of kernels.
For kernels of a particular convolutional layer l , we train an individual prior distribution ˆ p l ( w ) =
(cid:82) p ( w | z ; φ l ) p l ( z ) dz .
First, we collect source dataset of the kernels of the l -th layer of convolutional networks (source networks) trained on a dataset from a similar domain.
Then, we train reconstruction models p ( w | z ; φ l ) on these collected source datasets for each layer, using the framework of variational auto-encoder (Section 2).
Finally, we use reconstruction models to construct priors ˆ p l ( w ) as shown at Figure 1(a).
In our experiments, we found that regularization is crucial for learning of source kernels.
It helps to learn more structured and less noisy kernels.
Thus, source models were learned with L 2 regularization.
We removed kernels of small norm as they have no inﬂuence to predictions (Molchanov et al., 2017), but they make learning of a generative model more challenging.
Reconstruction and inference models for prior distribution.
In our experiments, inference models r ( z | w ; ψ l ) are fully-factorized normal-distributions N ( z | µ ψ l ( w ) , diag ( σ 2 ψ l ( w ))) , where parameters µ ψ l ( w ) and σ ψ l ( w ) are modeled by convolutional neural network.
The convolutional part of the network is constructed from several convolutional layers that are alternated with ELU (Clevert et al., 2015) and max-pooling layers.
Convolution layers are followed by a fully-connected layer with 2 · z ldim output neurons, where z ldim is a dimension of the hidden representation.
Dimension z ldim of the hidden space is speciﬁc for a particular layer.
Reconstruction models p ( w | z ; φ l ) are also modeled by a fully-factorized normal-distribution N ( w | µ φ l ( z ) , diag ( σ 2 φ l ( z ))) and networks µ φ l and σ 2 φ l have the similar architectures, but use transposed convolutions.
We use the same architectures for all prior models, but with slightly different hyperparameters, due to different sizes of kernels.
The prior distribution in space of latent variables was chosen to be fully-factorized standard
Gaussian p l ( z i ) =
N ( z i | 0 , 1) .
We provide a more detailed description at Appendix D.
E XPERIMENTS
We apply deep weight prior to variational inference, random feature extraction and initialization of convolutional neural networks.
In our experiments we used MNIST (LeCun et al., 1998), NotM-
[Figure]
NIST (Bulatov, 2011), CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009) datasets.
Experiments were implemented using PyTorch (Paszke et al., 2017).
For optimization we used Adam (Kingma & Ba, 2014) with default hyperparameters.
We trained prior distributions on 10 source networks which were learned from different initial points on NotMNIST and CIFAR-100 datasets for MNIST and CIFAR-10 experiments respectively.
4. C LASSIFICATION
In this experiment, we perform variational inference over weights of a discriminative convolutional neural network (Section 3) with three different prior distributions for the weights of the convolutional layers: deep weight prior (dwp), standard normal and log-uniform (Kingma et al., 2015).
We do not perform variational inference over the parameters of fully connected layers.
We used a fullyfactorized variational approximation with additive parameterization proposed by Molchanov et al.
(2017) and local reparametrization trick proposed by Kingma et al. (2015).
Note, that our method can be combined with more complex variational approximations, in order to improve quality of variational inference.
On MNIST dataset we used a neural network with two convolutional layers with 32 , 128 ﬁlters of shape 7 × 7 , 5 × 5 respectively, followed by one linear layer with 10 neurons.
On the CIFAR dataset we used a neural network with four convolutional layers with 128 , 256 , 256 ﬁlters of shape 7 × 7 , 5 × 5 , 5 × 5 respectively, followed by two fully connected layers with 512 and 10 neurons.
After the ﬁrst convolutional layer a max-pooling layer (Nagi et al., 2011) was applied.
All layers were divided with leaky ReLU nonlinearities (Nair & Hinton, 2010).
At ﬁgure 2 we report accuracy for variational inference with different sizes of training datasets and prior distributions.
Variational inference with deep weight prior leads to better mean test accuracy, in comparison to log-uniform and standard normal prior distributions.
Note that the difference gets more signiﬁcant as the training set gets smaller.
4. R ANDOM F EATURE E XTRACTION
Convolutional neural networks produce useful features even if they are initialized randomly (Saxe et al., 2011;
He et al., 2016; Ulyanov et al., 2017).
In this experiment, we study an inﬂuence of different random initializations of convolutional layers – that is ﬁxed during training – on the performance of convolutional networks of different size, where we train only fully-connected layers.
We use three initializations for weights of convolutional layers: learned kernels, samples from deep weight prior, samples from Xavier distribution (Glorot & Bengio, 2010) and the same architectures as in Section 4.1.
We found that initializations with samples from deep weight prior and learned kernels signiﬁcantly outperform the standard Xavier initialization when the size of the network is
[Figure]
a) Results for MNIST
[Figure] [Figure]
for
[Figure] [Figure]
i a t i o
[Figure]
a l
[Figure]
[Figure] [Figure]
small.
Initializations with samples form deep weight prior and learned ﬁlters perform similarly, but with deep weight prior we can avoid storing all learned kernels.
At Figure 3, we show results on
MNIST and CIFAR-10 for different network sizes, which are obtained by scaling the number of ﬁlters by k .
4. C ONVERGENCE
Deep learning models are sensitive to initialization of model weights.
In particular, it may inﬂuence the speed of convergence or even the local minimum the model converges to.
In this experiment, we study the inﬂuence of initialization on the convergence speed of two problems: a variational autoencoder on MNIST, and a convolutional network on MNIST and CIFAR-10.
We compare three different initializations of weights of conventional convolutional layers: learned ﬁlters, samples from deep weight prior and samples form Xavier distribution.
Figure 4 provides the results for a convolutional variational auto-encoder trained on MNIST and for a convolutional classiﬁcation network trained on CIFAR-10.
We found that deep weight prior and learned ﬁlters initializations perform similarly and lead to signiﬁcantly faster convergence compar- ing to standard Xavier initialization.
Deep weight prior initialization however does not require us to store a possibly large set of ﬁlters.
Also, we plot samples from variational auto-encoders at a different training steps Appendix C.
R ELATED W ORK
The recent success of transfer learning (Yosinski et al., 2014) shows that convolutional networks produce similar weights of convolutional ﬁlters while being trained on different datasets from the same domain e.g. photo-realistic images.
In contrast to Bayesian techniques (Kingma et al., 2015;
Kochurov et al., 2018), these methods do not allow to obtain a posterior distribution over parameters of the model, and in most cases, they require to store convolutional weights of pre-trained models and careful tuning of hyperparameters.
The Bayesian approach provides a framework that incorporates prior knowledge about the weights of the machine learning model by choosing or leaning a prior distribution p ( w ) .
There is a long line of research on prior distributions for Bayesian inference (MacKay, 1992; Williams, 1995), where empirical Bayes – an approach that tunes parameters of the prior distribution on the training data – plays an important role (MacKay, 1992).
These methods are widely used for regularization and sparsiﬁcation of linear models (Bishop & Tipping, 2003), however, applied to deep neural networks (Kingma et al., 2015; Ullrich et al., 2017), they do not take into account the structure of the model weights, e.g. spatial correlations, which does matter in case of convolutional networks.
Our approach allows to perform variational inference with implicit prior distribution, that is based on previously observed convolutional kernels.
In contrast to an empirical Bayes approach, parameters φ of a deep weight prior (equation 6) are adjusted before the variational inference and then remain ﬁxed.
Prior to our work implicit models have been applied to variational inference.
That type of models includes a number of ﬂexible variational distributions e.g., semi-implicit (Yin & Zhou, 2018) and Markov chain (Salimans et al., 2015; Lamb et al., 2017) approximations.
Implicit priors have been used for introducing invariance properties (Nalisnick & Smyth, 2018), improving uncertainty estimation (Ma et al., 2018) and learning meta-representations within an empirical Bayes approach (Karaletsos et al., 2018).
In this work, we propose to use an implicit prior distribution for stohastic variationa inference (Kingma et al., 2015) and develop a method for variational inference with the speciﬁc type of implicit priors.
The approach also can be generalized to prior distributions in the form of a Markov chain.
We show how to use this framework to learn a ﬂexible prior distribution over kernels of
Bayesian convolutional neural networks.
D ISCUSSION & C ONCLUSION
In this work we propose deep weight prior – a framework for designing a prior distribution for convolutional neural networks, that exploits prior knowledge about the structure of learned convolutional ﬁlters.
This framework opens a new direction for applications of Bayesian deep learning, in particular to transfer learning.
Factorization.
The lottery ticket hypothesis , proposed by Frankle & Carbin (2018), argues that to converge to a good solution we need just a few ﬁlters or neurons in the network which a lucky good initialization.
The factorization of deep weight prior, however, does not take into account inter-layer dependencies of the weights, although these dependencies might be useful.
Accounting inter-layer dependencies may give us an opportunity to recover a distribution in the space of trained networks rather then in the space of trained kernels.
Inference.
An alternative to variational inference with auxiliary variables is semi-implicit variational inference (Yin & Zhou, 2018).
This algorithm might provide a better way for variational inference with a deep weight prior, however it was developed only for semi-implicit variational approximations, so it might be interesting to adapt it for implicit prior distributions.
A V ARIATIONAL I NFERENCE WITH I MPLICIT P RIOR D ISTRIBUTION
We consider a variational lower bound L with variational approximation q ( w ) and prior distribution deﬁned in a form of Markov chain p ( w )
=
(cid:82) dz 0 . . .
dz T p ( w | z T ) (cid:81) Tt =0 p ( z t +1 | z t ) p ( z 0 ) and joint distribution p ( w, z ) =
p ( w | z T ) (cid:81) Tt =
p ( z t +1 | z t ) p ( z 0 ) .
Where p ( z t +1 | z t ) is a transition operator, and
z = ( z 0 , . .
, z T ) (Salimans et al., 2015).
Unfortunately, gradients of L cannot be efﬁciently estimated, but we construct a tractable lower bound L aux for L :
[Equation]
Inequality 13 has a very natural interpretation.
The lower bound L aux is tight if and only if the KLdivergence between the auxiliary reverse model and the posterior intractable distribution p ( z | w ) is zero.
The deep weight prior (Section 3) is a special of Markov chain prior for T = 0 and p ( w )
= (cid:82) p ( w | z ) p ( z ) dz .
The auxiliary variational bound has the following form:
[Equation]
15) where the gradients in equation 14 can be efﬁciently estimated in case q ( w ) , for explicit distributions q ( w ) , p φ ( w | z ) , r ( z | w ) that can be reparametrized.
B V ISUALIZATION OF D EEP W EIGHT P RIOR L ATENT S PACE
C S AMPLES FORM V ARIATIONAL A UTO - ENCODERS
[Figure]
D P RIOR A RCHITECTURES
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
E N ETWORK A RCHITECTURES
[Table]
F
T HE E STIMATE OF THE A PPROXIMATION G AP WITH IWAE E STIMATES
[Equation]
[Table]
During variational inference with deep weight prior (Algorithm 1) we optimize a new auxiliary lower bound L aux ( θ, ψ ) on the evidence lower bound L ( θ ) .
However, the quality of such inference depends on the gap G ( θ, ψ ) between the original variational lower bound L ( θ ) and the variational lower bound in auxiliary space L aux ( θ, ψ ) :
[Equation]
[Equation]
The gap G ( θ, ψ ) cannot be calculated exactly but can be estimated by using tighter but less computationally efﬁcient lower bound.
We follow Burda et al.
and construct tighter lower bound
L IWAEK ( θ, ψ ) :
[Equation]
The estimate L IWAEK ( θ, ψ ) converges to L ( θ ) with K goes to inﬁnity (Burda et al., 2015).
We estimate the gap with K = 10000 as follows:
[Equation] [Equation]
The results are presented at the Table 3.
In order to show the range of the estimate and the gain from learning of q ( z | w lij ψ l ) we compare results to the value of auxiliary lower bound L aux ( θ, ψ p l ) computed at the point ψ p l where q ( z | w lij ψ p l ) ≡ p l ( z ) .
The estimate of the gap, however, may be not very accurate and we consider it as a sanity check.