C OMPOUND D ENSITY N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite the huge success of deep neural networks (NNs), ﬁnding good mechanisms for quantifying their prediction uncertainty is still an open problem.
It was recently shown, that using an ensemble of NNs trained with a proper scoring rule leads to results competitive to those of Bayesian NNs.
This ensemble method can be understood as ﬁnite mixture model with uniform mixing weights.
We build on this mixture model approach and increase its ﬂexibility by replacing the ﬁxed mixing weights by an adaptive, input-dependent distribution (specifying the probability of each component) represented by an NN, and by considering uncountably many mixture components.
The resulting model can be seen as the continuous counterpart to mixture density networks and is therefore referred to as compound density networks .
We empirically show that the proposed model results in better uncertainty estimates and is more robust to adversarial examples than previous approaches.
I NTRODUCTION
Deep neural networks (NNs) have achieved state-of-the-art performance in many application areas, such as computer vision (Krizhevsky et al., 2012) and natural language processing (Collobert et al., 2011).
However, despite achieving impressive prediction accuracy on these supervised machine learning tasks, NNs do not provide good ways of quantifying predictive uncertainty.
This is undesirable for many mission critical applications, where taking wrong predictions with high conﬁdence could have fatal consequences (e.g. in medical diagnostics or autonomous driving).
A principled and the most explored way to quantify the uncertainty in NNs is through Bayesian inference.
In the so-called Bayesian neural networks (BNNs) (Neal, 1995), the NN parameters are treated as random variables and the goal of learning is to infer the posterior probability distribution of the parameters given the training data.
Since exact Bayesian inference in NNs is computationally intractable, different approximation techniques have been proposed (Neal, 1995; Blundell et al., 2015; Hern ´ andez-Lobato & Adams, 2015; Ritter et al., 2018, etc.).
Given the (approximate) posterior, the ﬁnal predictive distribution is obtained as the expected distribution under the posterior, where the expectation can be seen as an ensemble of an uncountably inﬁnite number of predictors, where the prediction of each model is weighted by the posterior probability of the corresponding parameters.
Based on a Bayesian interpretation of dropout (Srivastava et al., 2014), Gal & Ghahramani (2016) proposed to apply it not only during training but also when making predictions to estimate predictive uncertainty.
Interestingly, dropout has been also interpreted as ensemble model (Srivastava et al., 2014) where the predictions are averaged over the different NNs resulting from different dropoutmasks.
Inspired by this, Lakshminarayanan et al.
(2017) proposed to use an simple NN ensemble to quantify the prediction uncertainty, i.e. to train a set of independent NNs using a proper scoring rule and deﬁning the ﬁnal prediction as the arithmetic mean of the outputs of the individual models, which corresponds to deﬁning a uniformly-weighted mixture model.
It is argued, that the model is able to encode two sources of uncertainty by calibrating the prediction uncertainty in each component and capturing the “model uncertainty” by averaging over the components.
Note, that this should hold for any kind of predictive model that is deﬁned in terms of a mixture distribution.
In this paper, we therefore aim at further investigating the potential that lies in using mixture distributions for uncertainty quantiﬁcation.
The ﬂexibility of a mixture model can be increased by learning input-conditioned mixture weights, like it is done by mixture density networks (MDN) (Bishop,
Furthermore, one can consider uncountably many component distributions instead of a ﬁnite set, which turns the mixture distribution into a compound distribution.
We combine both by deriving the continuous counterpart of MDNs, which we call compound density networks (CDNs).
This model corresponds to a compound distribution in which both, component and mixing distribution, are parametrized based on NNs.
We experimentally show that CDNs allow for better uncertainty quantiﬁcation and are more robust to adversarial examples than previous approaches.
This paper is organized as follows.
In Section 2 we give a brief introduction to MDNs.
We then formally deﬁne CDNs in Section 3.
We review related work in Section 5 and present a detailed experimental analysis in Section 6.
Finally we conclude our paper in Section 7.
M IXTURE D ENSITY N ETWORKS
Let D = { x n , y n } Nn =1 be a i.i.d dataset and let us deﬁne the following conditional mixture model
[Equation]
and an NN that maps x onto both the parameters
π ( x ) of the mixing distribution and the parameters { φ k ( x ) } Kk
=1 of the K mixture components.
The complete system is called mixture density network (MDN) and was proposed by Bishop (1994).
That is, an MDN is an NN parametrizing a conditional mixture distribution, where both the mixture components and the mixture coefﬁcients depend on input x . 1 MDNs can be trained by maximizing the log-likelihood of the parameters of the NN given the training
set D using gradient-based optimizers such as stochastic gradient descent (SGD) and its variants.
MDNs belong to a broader class of models called mixture of experts (MoE) (Jacobs et al., 1991) which differ from standard mixture models by assuming that the mixture coefﬁcients depend on the input.
Because of its formulation as a mixture distribution, the predictive distribution of an MDN can handle multimodality better than a standard discriminative neural network.
Furthermore, the mixture coefﬁcients allow us to encode uncertainty about the prediction, namely by modelling the probability from which component distribution a data point was sampled.
C OMPOUND DENSITY NETWORKS
We aim at generalizing the MDN from a ﬁnite mixture distribution to a mixture of an uncountable set of components.
The continuous counterpart of a conditional mixture distribution in eq.
(1) is given by the conditional compound probability distribution
[Equation]
where φ ( x ) turns from a discrete into a continuous random variable.
We now want to follow the approach of MDNs to model the parameters of the components and the mixing distribution by NNs.
To handle the continuous state space of φ ( x ) in the case of a compound distributions, the key idea is now to let φ ( x ) be given by a stochastic NN f ( x ; θ )
=
φ ( x ) with stochastic parameters θ .
Since given x , f is a deterministic map from θ to φ ( x ) , it is possible to replace the mixing distribution p ( φ ( x ) | π ( x ))
=
p ( f ( x ; θ ) | π ( x )) by a distribution p ( θ | π ( x )) over θ .
We further assume, that the parameters π ( x ) of the mixing distribution are given by some parametrized function g ( x ; ψ )
=
π ( x ) which can also be modelled based on NNs.
In correspondence to MDNs, the complete system is called compound density network (CDN) and it is summarized by the following equation
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
As for MDNs, CDNs training corresponds to maximizing its likelihood function, which is given by
[Equation]
As we now have to deal with an integral instead of a ﬁnite sum, this log-likelihood function is intractable.
Fortunately, applying the reparametrization trick (Kingma & Welling, 2014) to θ ∼ p ( θ | g ( x ; ψ )) enables Monte Carlo integration, while still being able to optimize the log-likelihood via backpropagation in conjunction with SGD.
To avoid overﬁtting, we can regularize the mixing distribution p ( θ | g ( x ; ψ )) by penalizing its KLdivergence w.r.t.
some simple distribution p ( θ ) 3 , leading to the regularized objective function
[Equation]
where λ controls the regularization strength.
We summarize the training procedure of CDNs in the pseudo-code provided in Appendix A. Note, that CDNs correspond to an abstract framework for modelling compound distributions with NNs.
In the following section, we present a concrete example on how CDNs can be implemented.
3. P ROBABILISTIC HYPERNETWORKS
Ha et al.
(2017) proposed to (deterministically) generate the parameters of an NN by another NN, which they call the hypernetwork .
5 We would like to follow this approach for modelling a CDN, that is, we aim at modelling the mixing distribution p
( θ | g ( x ; ψ )) over network parameters by NNs.
Since now the hypernetworks maps x to a distribution over parameters instead of a speciﬁc value θ , we refer to them as probabilistic hypernetworks .
In the following we will describe this idea in more detail.
Let f be a multi-layer perceptron 6 (MLP) with L -layers, parametrized by a set of layers’ weight matrices 7 θ = { W l } Ll =1 , that computes the parameters φ ( x ) =
f ( x ; θ ) of the CDNs component distribution in eq.
Let h 1 , . .
, h l − 1 denote the states of the hidden layers, and let us deﬁne h
= x , and h L = f
( x ; θ ) .
We now assume the weight matrices { W l } Ll =1 to be random variables and to be independent from each other given the state of the previous hidden layer.
We deﬁne a series of probabilistic hypernetworks
g = { g l } Ll =1 (parametrized by ψ = { ψ l } Ll =1 ), where g l maps h l − 1 to the parameters of the distribution of W l , and let the joint distribution over θ be given by
[Equation]
An illustration of a stochastic two-layer network f ( x ; θ ) computing φ ( x ) , with parameters distributions given by probabilistic hypernetworks as deﬁned in eq.
(6) is given in Figure 1.
To make alternative implementation adaptive dropout apply hypernetwork the deﬁnition of our model complete, we need to deﬁne the concrete statistical model we pick for p ( W l | g l ( h l − 1 ; ψ )) .
This is done in the following section.
3. P ROBABILISTIC HYPERNETWORKS WITH MATRIX VARIATE NORMALS
A statistical model that was recently applied as the posterior over weight matrices in BNNs (Louizos & Welling, 2016; Sun et al., 2017; Zhang et al., 2018; Ritter et al., 2018) is the matrix variate normal (MVN) distribution (Gupta & Nagar, 1999).
An MVN is parametrized by three matrices: a mean matrix M and two covariance factor matrices A and B .
It is connected to the multivariate Gaussian by the following equivalence
[Equation]
where vec ( X ) denotes the vectorization of X .
Due to the Kronecker factorization of the covariance, an MVN requires less parameters compared to a multivariate Gaussian, which motivates us to use it as the distribution over weight matrices in this work.
Furthermore, we assume that the covariance factor matrices are diagonal matrices, following Louizos & Welling (2016).
That is, we choose the mixture distribution of the CDN to be
[Equation]
where g l maps the state h l − 1 of the previous hidden layer onto the l -th MVN’s parameters { M l , a l , b l } , deﬁning the distribution over W l . Suppose W l ∈ R r × c , then the corresponding
MVN distribution has rc + r + c parameters, which is more efﬁcient compared to rc + rc parameters when modeling W l as fully-factorized Gaussian random variable.
In practice, the procedure of sampling θ and computing f ( x ; θ ) =
h L now can be described by h l = σ ( h l − 1 W l ) , where W l ∼ MN ( W l | g l ( h l − l ; ψ l )) , (9) for l = 1 , . . .
, L , where σ is an arbitrary point-wise nonlinear activation function, which may differ for hidden and output layers.
Note that using the mixing distribution deﬁned in eq.
(8) allows us to apply the reparametrization trick (Appendix C.1).
For the KL-divergence-based regularization, a straight forward choice for the simple distribution is p ( θ ) =
(cid:81)
Ll =
MN ( W l | 0 , I , I ) , which corresponds to assuming that each layer’s weight matrix is standard matrix normal distributed.
In this case, the KL-term can be computed in closed form, as noted by Louizos & Welling (2016) and shown in Appendix C.2.
Finally, we use a vector-scaling parametrization similar to the one used by Ha et al.
(2017) and Krueger et al.
(2017) for the mean matrices { M l } Ll =1 , which we explain in detail in Appendix D.
C ONNECTION TO B AYESIAN NEURAL NETWORKS AND V ARIATIONAL I NFORMATION B OTTLENECK
Interestingly, eq.
(2) could also be interpreted as integrating over the parameters of a Bayesian neural network, where p ( φ ( x ) | π ( x )) is an amortized approximate posterior.
Performing variational inference in such a setting would lead to the following objective
[Equation] Note, that this is a lower bound of eq. (5) (by Jensen’s inequality) and that stochastic approximations of both objectives become equivalent when they are based on a single sample of θ and λ = 1 . For arbitrary value of λ , the one-sample-approximation of eq.
(5) is also equivalent to the approximated variational information bottleneck (VIB) objective (Alemi et al., 2017).
Following this objective, the proposed approach corresponds to performing VIB where the network parameters, instead of the hidden units, are considered as latent variables.
We will analyze the effects of following these different objectives experimentally in Section 6.4.
R ELATED WORK
Various approaches for quantifying predictive uncertainty in NNs have been proposed.
Applying
Bayesian inference to NNs, i.e. treating the network parameters as random variables and estimating the posterior distribution given the training data based on Bayes’ theorem, results in Bayesian neural networks (BNNs) (MacKay, 1992; Neal, 1995; Graves, 2011; Blundell et al., 2015; Louizos &
Welling, 2016; Sun et al., 2017; Louizos & Welling, 2017; Ritter et al., 2018; Zhang et al., 2018, etc).
Since the true posterior distribution is intractable, BNNs are trained based on approximate inference methods such as variational inference (VI) (Peterson, 1987; Hinton & Van Camp, 1993;
Graves, 2011; Blundell et al., 2015), Markov Chain Monte Carlo (MCMC) (Neal, 1995), or Laplace approximation (MacKay, 1992; Ritter et al., 2018).
The ﬁnal prediction is then given by the expectation of the network prediction (given the parameters) w.r.t.
the approximate posterior distribution.
Louizos & Welling (2016) proposed to train BNNs based on VI with an MVN as the approximate posterior of each weight matrix (leading to a model they refer to as variational matrix Gaussian (VMG)).
Multiplicative normalizing ﬂow (MNF) (Louizos & Welling, 2017) models the approximate posterior as a compound distribution, where the mixing density is given by a normalizing ﬂow.
Zhang et al.
(2018) also use an MVN approximate posterior and apply approximate natural gradient (Amari, 1998) based maximization on the VI objective, which results in an algorithm called noisy
K-FAC.
Meanwhile, the Kronecker-factored Laplace approximation (KFLA) (Ritter et al., 2018) extends the classical Laplace approximation by using an MVN approximate posterior with tractable and efﬁciently computed covariance factors, based on the Fisher information matrix.
There have been several concurrent works (Krueger et al., 2017; Louizos & Welling, 2017;
Pawlowski et al., 2017; Sheikh et al., 2017) applying hypernetworks (Ha et al., 2017) to model the posterior distribution over network parameters in BNNs.
In contrast to CDNs, the hypernetworks in these approaches are used to transform random noise drawn from a simple distribution into a random variable with a complicated distribution.
That is, they use hypernetworks to sample from an implicit distribution of the parameters, without explicitly specifying the statistical model (like the MVN in our case).
Krueger et al.
and Louizos & Welling (2017) use normalizing ﬂows, while Pawlowski et al.
(2017) and Sheikh et al.
(2017) use arbitrary NNs as their hypernetworks.
The approach by Sheikh et al.
(2017) is the one most closely related to ours, since they use an objective analogous to eq.
Note, that the main difference between CDNs and these hypernet-BNNs is that the approximate posterior in a Bayesian setting does not depend on the current input point, while the mixing distribution of the CDN does.
Gal & Ghahramani (2016) developed a theoretical framework that relates dropout training in NNs to approximate Bayesian inference and, as a result, proposed to approximate the predictive distribution by an average over the different networks resulting from independently sampled dropout-masks, a technique which they referred to as MC-dropout and which they applied to estimate the prediction uncertainty in NNs.
Recently, Lakshminarayanan et al.
(2017), proposed to use an ensemble of
NNs in conjunction with a proper scoring rule and adversarial training to quantify the prediction uncertainty of deep NNs, leading to a model referred to as Deep Ensemble.
The Deep Ensemble provides a non-Bayesian way to quantify prediction uncertainty, and is in this sense related to the approaches of Guo et al.
and Hendrycks & Gimpel (2017).
E XPERIMENTS
We consider several standard tasks in our experimental analysis: 1D toy regression problems inspired by Hern´andez-Lobato & Adams (2015) and Depeweg et al.
(Section 6.1), classiﬁcation under out-of-distribution data (Section 6.2), and detection of and defense against adversarial examples (Szegedy et al., 2014)
(Section 6.3).
We consider the following recent models (described in Section 5) as the baselines for our CDN model: MNF, KFLA, noisy K-FAC, MC-dropout, and Deep Ensemble.
In all of the experiments, we consider λ ∈ { 10 − n } 5 n =1 when training CDNs based on eq.
(5), and pick the highest value of λ that still allows for high predictive power (e.g. an accuracy of > 0 .on
the validation set).
We use this selection heuristic, based on our observation that in general, as λ increases, the accuracy is decreasing while the uncertainty estimate is increasing (see Appendix F. for an investigation of this behavior).
The hyperparameters for the baselines are set to the values suggested by the respective original publications as outlined in Appendix F. For BNNs and the CDN we estimate the predictive distribution p ( y | x ) , based on 100 samples of network parameters from the posterior distribution or the mixing distribution, respectively. Unless stated otherwise, we use a single sample of θ to do Monte Carlo integration during training.
For MNIST and CIFAR-10 experiments, we use mini-batches of size 200.
We use Adam (Kingma & Ba, 2015) with default hyperparameters for optimization in all experiments and the implementations provided by Louizos & Welling (2017) 10 and Zhang et al.
(2018) 11 for MNF and noisy K-FAC, respectively.
The implementation of all models and experiments will be made available to the public once the review process is concluded.
6. T OY REGRESSION
Following Hern´andez-Lobato & Adams (2015) and Depeweg et al.
(2018), we use two toy regression problems to demonstrate the capability of CDNs of quantifying predictive uncertainty.
The datasets of those probems are generated as follows. • Cubic (Hern´andez-Lobato & Adams, 2015): We sample 20 input points x ∼ U [ − 4 , 4] and their target values y = x 3 + (cid:15) , where (cid:15) ∼ N (0 , 3 2 ) .
• Mixture (Depeweg et al., 2018): We sample 750 input points x ∼ p ( x ) , where p ( x ) = 13 N ( − 4 , 25 ) + 13 N (0 , 0 . 9) + 13 N (4 , 25 ) and generate their target values by y = sin x + 3 | cos x 2 | (cid:15) , where (cid:15) ∼ N (0 , 1) .
We use a single layer MLP with 100 hidden units as the predictive network, while the hypernetworks ( g 1 and g 2 ) are modeled with single layer MLPs with 10 hidden units each. Three samples from the mixing distribution are used to do Monte Carlo integration to approximate eq.
[Figure]
The results are presented in Figure 2.
We observe that our model is able to accurately quantify the heteroscedastic noise in the Mixture dataset. Comparing the results for the CDN with those of Bayesian models on the Cubic dataset (Figure 8 in the appendix) it becomes clear that the CDN is quantifying the noise applied in the data-generating process (aleatoric uncertainty) instead of epistemic uncertainty. A visualization of the learned mixing distributions for different input points is provided in Appendix G.
6. O UT - OF - DISTRIBUTION CLASSIFICATION
Following Lakshminarayanan et al.
(2017), we train all models on the MNIST training set and investigate their performance on the MNIST test set and the notMNIST dataset 12 , which contains 10 https://github.com/AMLab-Amsterdam/MNF_VBNNhttps://github.com/gd-zhang/noisy-K-FAC
Available at http://yaroslavvb.blogspot.com/2011/09/notmnist-dataset.html .
[Figure]
We present the results in Figure 3, where we plotted the cumulative distribution function (CDF) of the empirical entropy of the predictive distribution, following Louizos & Welling (2017).
A
CDF curve close to the top-left corner of the ﬁgure implies that the model yields mostly low entropy predictions, indicating that the model is very conﬁdent.
While one wishes to observe high conﬁdence on data points similar to those seen during training, the model should express uncertainty when exposed to out-of-distribution data.
That is, we prefer a model to have a CDF curve closer to the bottom-right corner on notMNIST, as this implies it makes mostly uncertain (high entropy) predictions, and a curve closer to the upper-left corner for MNIST.
As the results show, our model has very high conﬁdence on the test set of MNIST while having the lowest conﬁdence on notMNIST compared to all baseline models.
It is surprising that even though CDNs are designed to capture aleatoric uncertainty they outperforms other models in this experiment, which is rather designed for quantifying model uncertainty (Kendall & Gal, 2017).
6. A DVERSARIAL EXAMPLES
To investigate the robustness and detection performance of the CDN w.r.t.
adversarial examples (Szegedy et al., 2014), we apply the Fast Gradient Sign Method (FGSM)
(Goodfellow et al., 2015) to a 10% fraction (i.e. 1000 samples) of the MNIST and CIFAR-10 test set.
We do so, by making use of the implementation provided by Cleverhans (Papernot et al., 2018).
We use the same
MLP (with 784-100-10 architecture) as before for experiments on MNIST, and the LeNet5 convolutional network (LeCun et al., 1998) for experiments on CIFAR-10.
Out of implementation reasons, we only model the parameters of the fully-connected layers of LeNet5 as random variables for CDN and KFLA.
The probabilistic hypernetworks are two-layer MLPs with 100 hidden units.
Note, that we do not use adversarial training when training the Deep Ensemble in this experiment to allow for a fair comparison.
MNIST Figure 4 presents the accuracy and the average empirical entropy of the predictive distribution w.r.t.
adversarial examples for MNIST with varying levels of perturbation strength (between 0 and 1).
We observe that the CDN is signiﬁcantly more robust to adversarial examples than all baseline models.
The prediction entropy for adversarial examples is also higher for the CDN than for most other models (only MC-dropout shows higher uncertainty in the beginning).
Furthermore, the prediction uncertainty of the CDN is steadily increases with increasing perturbation strength.
[Figure] [Figure] We demonstrate how the performance of our model depends on the choice of regularization strength λ in Figure 5.
It is clearly visible that λ acts as a hyperparameter that balances the trade-off between detection (uncertainty) and robustness (accuracy).
When allowing for lower uncertainty, the CDN gets surprisingly robust, showing a prediction accuracy of around 0.4 even for a perturbation strength of 1 .
Even when we increase the strength of the adversarial examples by computing them based on 10 different samples from our model, the CDN is still signiﬁcantly more robust than all baseline models w.r.t.
the weaker adversarial examples relying on a single sample.
CIFAR-10 Figure 6 shows that, leading to second best results w.r.t.
accuracy as well as uncertainty, our model is competitive to other state of the art models on CIFAR-10.
Note, that for the
CDN we do not use probabilistic hypernetworks for the convolution layers of LeNet5, i.e. we only treat the parameters of the fully-connected layers as random variables.
Treating all parameters as random variables, as it is done by the BNNs in this experiment (except from KFLA) could potentially improve the performance of the CDN.
6. C OMPARISON TO TRAINING BASED ON ELBO AND VIB OBJECTIVE
In this section, we experimentally investigate the effects of (a) optimizing our proposed objective (eq.
(5)), (b) treating our model as a BNN and train it with the ELBO objective (eq.
(10)), and (c) following the VIB approach (which corresponds to optimizing the objective which results from introducing a hyperparameter λ in front of the KL-term in eq. As stated before, (a) and (c) [Figure] [Figure]
become equivalent when approximating the objective by a single sample of θ , and both become equivalent to (b) when setting λ = 1 in addition.
To better analyze the effects of following the different objectives we therefore approximate them based on 10 samples.
Results for the OOD classiﬁcation and the robustness to adversarial examples are shown in Figure 7.
Training based on the ELBO results in unsatisfactory performance as can be seen from the orange graphs and the indicated low validation accuracy.
The CDN and the VIB objective both work well on the OOD classiﬁcation task, where no performance is gained by using 10 instead of one sample (which makes both equivalent).
This suggests that the input dependency of the distribution over θ (in addition to the weighting of the KL-term in the objective) plays a crucial role for the increased performance observed compared to baseline models for OOD classiﬁcation.
This hypothesis is also supported by the fact that the performance is increase compared to the VMG (Louizos & Welling, 2016), which is a closely related BNN using a MVN distributions as approximate (input independent!)
posterior.
While for the robustness against adversarial attacks the increased sample size improved the performance of models trained with the VIB as well as with the CDN objective, the model trained with the CDN objective clearly outperforms the others, reaching an surprisingly high accuracy about 0.
even under huge perturbations.  C ONCLUSION We introduce compound density networks (CDNs), a new framework that allows for better uncertainty quantiﬁcation in neural network (NN), and corresponds to a compound distribution (i.e. mixture with uncountable components) in which both the component distribution and the mixing distribution are parametrized by NNs.
CDNs are inspired by the success of recently proposed ensemble methods and represent a continuous generalization of mixture density networks (MDNs) (Bishop, 1994).
They can be implemented by using a hypernetwork to map the input to a distribution over the parameters of the target NN, that models a predictive distribution.
An extensive experimental analysis showed that CDNs are able to produce promising results in terms of uncertainty quantiﬁcation.
Especially, when facing FGSM-based adversarial attacks, the predictions of our model are signiﬁcantly more robust than those of previous models, while simultaneously providing a better chance of detecting the attack by showing increased uncertainty.
A PPENDIX A P SEUDOCODE FOR TRAINING CDN S
A PPENDIX B CDN S WITH LATENT VARIABLES
In Section 3, we introduce CDNs as a compound distribution where the stochasticity of f ( x ; θ ) comes from treating θ as random variables.
Another approach to achieve stochasticity on f is by introducing latent variable z and letting the parameter θ to be ﬁxed (learned with MLE).
The CDN would then be descibed by
[Equation]
The log-likelihood function is therefore
[Equation]
and the objective function is
[Equation]
The training algorithm in Algorithm 1 only needs to be trivially modiﬁed to accommodate θ , i.e. in the gradient steps.
In the following section, we present an instance of this framework.
B. A VARIANT OF ADAPTIVE DROPOUT
This model relies on injecting noise into the hidden activations of the neural network f , similar to dropout (Srivastava et al., 2014).
However, in this model, we condition the dropout’s Gaussian on the activations of the previous hidden layer, similar to adaptive Bernoulli dropout proposed by Ba &
Frey (2013).
Let z l ∼ p ( z l | g l ( h l − 1 ; ψ l )) be the noise vector of layer l .
We then multiplicatively apply this noise to the activations h l , i.e. h l (cid:12) z l , where (cid:12) denotes the Hadamard product.
Let z :=
{ z l } L − 1 l =1 and let g ( x ; ψ ) :
= { g l ( h l − 1 ; ψ l ) } Ll =1 .
Let p ( z | g ( x ; ψ ))
=: (cid:81) L − 1 l =1 p ( z l | g ( h l ; ψ l )) be the mixing distribution.
Each of the p ( z l | g ( h l ; ψ l )) can be any distribution as long as we can apply reparametrization trick.
Then, we can immediately apply latent CDNs training procedure on this model.
One example of p ( z l | g l ( h l − 1 ; ψ l )) is N ( z l | 1 , g l ( h l − 1 ; ψ l )) , which then the model resembles Gaussian dropout (Srivastava et al., 2014) with adaptive scaling.
A PPENDIX C D ETAILS ABOUT THE MATRIX VARIATE NORMAL (MVN)
Let X ∈ R r × c and p ( X ) :
=
MN ( X | M , diag ( a ) , diag ( b )) .
Taken from Louizos & Welling (2016), the procedure to sample from p ( X ) using reparametrization trick (Kingma & Welling, 2014) and the closed-form expression of the KL-divergence to MN ( 0 , I , I ) are presented in the following sections.
C. R EPARAMETRIZATION TRICK
Let E ∈ R r × c .
Sampling p ( X ) can be done by
[Equation]
C. KL- DIVERGENCE TO STANDARD MVN
Let I r ∈ R r × r , I c
∈ R c
×
c be identity matrices, the KL-divergence between p ( X ) and
MN ( X | 0 , I , I ) is given by
[Equation]
A PPENDIX D V ECTOR SCALING PARAMETRIZATION
The naive formulation of g l can be very expensive in term of number of parameters.
Suppose
W l ∈ R r × c
and g l is a two layer MLP with k hidden units.
Then g l would have rk + krc
+ kr + kc many parameters, which quickly becomes very large for a moderately sized NNs.
The majority of the parameters are needed to deﬁne the mean matrix M l .
Following the approach of Ha et al.
(2017) and Krueger et al.
(2017), we make a trade-off between expressiveness of g l on the mean matrix with the number of parameter by instead replacing M l with a matrix V l of the same size and a vector d l ∈ R r , which is the output of g l .
Thus, now g l maps h l − 1 (cid:55)−→ { d l , a l , b l } and we can get M l by [Equation]
[Equation]
That is, each element of d l is being used to scale the corresponding row of V l .
Note that although
V l is a parameter matrix with the same size of M l , it crucially is not an output of g l as in the naive parametrization.
Thus the number of parameter of g l is now rk + rc + 2 kr + kc , which is more manageable and implementable for larger weight matrices.
A PPENDIX E E XPERIMENTAL RESULTS FOR ADDITIONAL MODELS
In this section we compare the CDN descibed in the main text with the CDN proposed in Appendix B.1 (CDN-dropout), the BNN proposed by Louizos & Welling (2016) (VMG), a mixture of experts (MoE) and an MDN on the MNIST dataset.
For both MDN and MoE, we use two-layer MLP with 100 hidden units, with 5 mixture components.
Speciﬁcally for MoE, the mixing distribution is also given by another NN of the same architecture.
We use the code provided by Louizos & Welling (2016) to get VMG’s results. The results for the out-of-distribution prediction are presented in Figure 9.
Note that the analysis presented in the main text applies here as well.
[Figure]
[Figure]
A PPENDIX F H YPERPARAMETERS
For the baseline models in our experiments, we use the hyperparameter values that are suggested in the respective publications ans summarized in the following: • MNF: The KL-term is weighted by 1 /B , where B is the number of mini-batches used during optimization (see (Graves, 2011) for a justiﬁcation of this).
Moreover it is annealed with a hyperparameter initialized to 0 and increasing to 1 during training. We found that this yields better results than using no re-weighting of the KL term and is necessary to achieve the results reported by Louizos & Welling (2017). • noisy K-FAC: KL-term weight is set to λ = 0 . 5 and the prior variance to η = 0 .
• KFLA: we use the best prior precision τ ∈ { 1 , 10 , 20 , 30 , 40 , 50 } we found in an hyperparameter search, which we detail in the next section. • Deep Ensemble: The number of mixture components is 5, the adversarial perturbation strength is set to 1% of the input range, and the weight decay is 0 . • MC-dropout: The dropout probability was set to 0 .
5 and the weight decay parameter to 0 . F. H YPERPARAMETER SEARCH We perform a hyperparameter search on λ for the training of CDNs and for τ for KFLA based on the MNIST validation accuracy, and pick the highest λ (the lowest τ ) that achieve > 0 .
97 validation accuracy. [Table] [Table] [Figure]
[Figure]
To further understand the effect of conditioning the distribution over θ (i
[Figure]
e. the mixing distribution) we compute the mixing distribution p
( θ | g ( x i ; ψ )) for a set of samples x 1 , . . . x n and a CDN trained on a variant of the Cubic dataset, where we add random noise of different variance depending on x (that is, (cid:15) ∼ N (0 , 15 2 ) if x < 0 and (cid:15) ∼ N (0 , 3 2 ) , otherwise). The results for two randomly selected weights w ( l ) i ∈ θ are shown in Figure 12. We found that the mean and the variance of the distribution (which is Gaussian due to our model) varies depending on the value of the input x indicating that different mixture components get high probability for different inputs.
[Figure]
Furthermore, to show that CDNs are able to capture multimodality in weight space, we train a CDN with a 5 hidden units mixture component on a toy classiﬁcation dataset that is constructed as follows: we sample an input x from a mixture of Gaussian p ( x ) = 12 N ( − 3 , 1) + 12 N (3 , 1) , and assign a label depending whether it comes from the ﬁrst ( y = 0 ) or the second Gaussian ( y = 1 ). To evaluate the resulting distribution, we marginalize the mixing distribution p ( θ | g ( x ; ψ )) w.r.t. x , i.e. we evaluate p ( θ ) =
(cid:82) p ( θ | g ( x ; ψ )) p ( x ) d x . The resulting distribution for two randomly selected weights w ( l ) i ∈ θ are shown in the ﬁgure below. We observe that indeed our model can learn a multimodal weight distribution.
[Figure]