Bayesian Evidential Deep Learning with PAC Regularization
Abstract
We propose a novel method for closed-form predictive distribution modeling with neural nets.
In quantifying prediction uncertainty, we build on Evidential Deep Learning, which has been impactful as being both simple to implement and giving closed-form access to predictive uncertainty.
We employ it to model aleatoric uncertainty and extend it to account also for epistemic uncertainty by converting it to a Bayesian Neural Net.
While extending its uncertainty quantiﬁcation capabilities, we maintain its analytically accessible predictive distribution model by performing progressive moment matching for the ﬁrst time for approximate weight marginalization.
The eventual model introduces a prohibitively large number of hyperparameters for stable training.
We overcome this drawback by deriving a vacuous PAC bound that comprises the marginal likelihood of the predictor and a complexity penalty.
We observe on regression, classiﬁcation, and out-of-domain detection benchmarks that our method improves model ﬁt and uncertainty quantiﬁcation.
1. Introduction
As the interest of the machine learning community in data-eﬃcient and uncertainty-aware predictors increases, research on Bayesian Neural Networks (BNNs) (MacKay, 1995; Neal, 1995) gains prominence.
Diﬀerently from deterministic nets, BNNs have stochastic weights.
Thanks to their stacked structure, they propagate predictive uncertainty through the hidden layers and can characterize complex uncertainty structures.
Exact inference of such a highly nonlinear system is analytically intractable and very hard to approximate with high precision.
Consequently, most research on BNNs thus far focused on improving approximate inference techniques in terms of precision and computational cost (Hernández-Lobato and Adams, 2015; Kingma et al., 2015; Louizos and Welling, 2017).
These approaches take the posterior inference of global parameters as given and develop their approximation based on it.
A newly emerging alternative approach is direct predictive distribution modeling.
It proposes devising a highly expressive predictive distribution with several free parameters.
These parameters are then ﬁt to data via maximum likelihood estimation.
This way, observations are used to train directly the end product of interest: the predictive distribution, bypassing the need for an intractable posterior inference step.
Some existing methods model the predictive distribution via a stochastic process parameterized as a neural net (Garnelo et al., 2018a), while others introduce local priors on the likelihood activations, integrate them out and train the hyperparameters of the marginal (Sensoy et al., 2018; Malinin and Gales, 2018).
This work builds on Evidential Deep Learning (EDL) (Sensoy et al., 2018), due to its technical simplicity and observed eﬀectiveness.
EDL places a Dirichlet prior on the class assignment probabilities of a classiﬁer and parameterizes the Dirichlet strengths of this prior with a neural net.
While demonstrating substantial improvements in out-ofdomain detection and adversarial robustness, it is not capable of decomposing epistemic and aleatoric uncertainties.
We propose an eﬃcient and eﬀective method that extends EDL to BNNs, equipping it with more advanced uncertainty quantiﬁcation and decomposition capabilities.
We assume independent local weight random variables controlling the BNN for each input/target pair of data points, which share common hyperparameters.
Marginalizing these data-point speciﬁc weights of our network we perform training via type 2 maximum likelihood (ML) (Berger, 1985) on the prior hyperparameters.
This analytically intractable marginalization is approximated using the Central Limit Theorem (CLT).
Diﬀerently from other weight marginalization approaches that assign global weight distributions on inﬁnitely many neurons and recover Gaussian Processes (GP) (Neal, 1995; Lee et al., 2018; GarragiaAlonso et al., 2019), our formulation maintains ﬁnitely many hidden units per layer and assigns them individual weight distributions.
Due to per data-point treatment of the weight marginalization the BNN scales linearly with the data size.
The advantages of such a BNN with data-point speciﬁc marginalization comes at the expense of a major drawback.
The number of hyperparameters in a weight-marginalized BNN grows proportionally to the number of synaptic connections.
Maximizing the marginal likelihood w.r.t.
such a large number of hyperparameters is prone to overﬁtting (Bauer et al., 2016).
Since the weight variables are marginalized out and their hyperparameters of the weight prior are set via optimization, the model can no longer incorporate regularizing knowledge other than the parametric form of the prior distribution (e.g. normal with mean and variance as free parameters).
We address this drawback by deriving a provably vacuous Probably
Approximately Correct (PAC) (McAllester, 1999, 2003) bound that contains the marginal likelihood as its empirical risk term.
Minimization of this PAC bound automatically balances the ﬁt to the data and deviation from a prior regularizing hypothesis.
We compare our method on various standard regression, classiﬁcation, and out-of-domain detection benchmarks against state-of-the-art approximate posterior inference based BNN training approaches.
We observe that our method provides competitive prediction accuracy and better uncertainty estimation scores than those baselines.
2. Bayesian Evidential Deep Learning
Evidential Deep Learning.
Classiﬁcation with cross-entropy loss in deep learning can be interpreted as a categorical likelihood parameterized by a neural net.
EDL generalizes this setup by parameterizing a prior by a neural net f ( · ; w ) with deterministic weights w , instead of the likelihood.
For classiﬁcation, given a data set D
=
{ ( x n , y n ) Nn =1 } consisting of N pairs of input x n and target y n , a natural choice for the prior p ( λ n | w , x n ) on the categorical likelihood is a Dirichlet distribution, and the ﬁnal model becomes
[Equation]
where α n = f ( x n ; w ) + 1 .
This way, the model explicitly accounts for the distributional uncertainty which may arise due to a mismatch between the train and test data distributions.
To train, the loss is the expected sum of squares between y n and λ n with an additional regularizing Kullback Leibler divergence on the λ n .
Bayesian Local Neural Nets
Parameterizing the likelihood by a BNN f ( · ; w ) with random variables w as the weights results in the following probabilistic model
[Equation]
where p ( y n | x n , w n ) is some likelihood, and p φ ( w n ) is a prior over local weights with shared hyperparameters φ .
Diﬀerently from a canonical BNN where all data points share the same global weight latent variable (Blundell et al., 2015; Gal and Ghahramani, 2015; Kingma et al., 2015; Louizos and Welling, 2017), here the mapping between each input-output pair is determined by a separate random variable w n , giving a unique mapping constrained by sharing a common set of prior hyperparameters φ , which is required for the type 2 ML based objective we introduce below.
As this modiﬁcation implies a collection of only local latent variables, we refer to the resulting model as a Bayesian Local Neural Net (BLNN) .
It consists of two sources of uncertainty.
First, the model (epistemic) uncertainty captured by the prior over the parameters, i.e. p φ ( w n ) , which accounts for the mismatch between the model and the true functional mapping from x n to y n .
Second, the irreducible data (aleatoric) uncertainty given by p ( y n | x n , w n ) stemming from irreducible measurement noise.
BLNN Training with Type 2 ML and Prediction.
An alternative to full Bayesian model training with posterior inference is marginalizing out all latent variables and maximizing the marginal likelihood with respect to the hyperparameters, hence avoiding the posterior inference step on latent variables, referred to as type 2 ML (Berger, 1985).
The optimization objective then is: arg max φ log p φ ( y | X ) .
Introducing an independent w n for each pair ( x n , y n ) leads to a sum of N independent marginal likelihoods,
[Equation]
which is amenable to using mini-batches for further scalability.
The marginal of a training data point is identical to the posterior predictive for new test data x ∗ .
Hence, an analytic approximation developed for training is directly applicable to test time.
Analytic Marginalization of Local Weights with Moment Matching.
Marginalizing out the local weights w n in (2) is an intractable problem due to the highly nonlinear neural net appearing in the likelihood p ( y n | x n , w n ) .
However, we can marginalize the weights approximately by recursive moment matching resorting to the Central Limit Theorem (CLT).
This technique has previously been used in BNNs for other purposes, such as expectation propagation (Hernández-Lobato and Adams, 2015; Ghosh et al., 2016), fast dropout (Wang and Manning, 2013), and variational inference (Wu et al., 2019).
We employ the same technique for marginalizing out the weights of the BLNN (see appendix).
Bayesian Evidential Deep Learning.
The original formulation for EDL builds on deterministic nets, hence assumes a point estimate on the weights (consciously ignoring model uncertainty) and a sum-of-squares loss term.
We improve this framework by assigning a local prior on the EDL weights p φ ( w n ) , 1 and instead consider the optimization of the marginal likelihood, which amounts to employing a BLNN as a prior on the likelihood and marginalizing over all w n as well as λ n .
We name the eventual model that combines BLNN with EDL Bayesian Evidential Deep Learning (BEDL) .
By virtue of the localized weights, the marginal likelihood of BEDL factorizes across data points, bringing additive data point speciﬁc marginal log-likelihoods maintaining the central source of its scalability, formally
[Equation]
The marginalization of λ n on the last step can be performed analytically under conjugacy, eﬃciently approximated by Taylor expansion, or via Monte Carlo sampling, after moment matching to marginalize the weights w n .
For the C -class classiﬁcation task with one-hot encoded targets y n and Dirichlet distributed λ n , this gives us for the n -th term in (3), log
[Equation]
where f Ln is the last layer after marginalization of the others (see Appendix D), and we use α n = ( α n 1 , ..., α nC ) =
exp( f Ln ) , and α n 0 = (cid:80) c α nc .
Since the computational bottleneck on weight marginalizing is circumvented by the analytical CLT-based moment matching, the ﬁnal expectation can be eﬃciently approximated by samples.
3. A Vacuous PAC Bound to Regularize BEDL
Training the objective in (3) is eﬀective for ﬁtting a predictor on data.
It also naturally provides a learned loss attenuation mechanism.
However, it lacks a key advantage of the
Bayesian modeling paradigm.
As the hyperparameters of the weight priors are employed for model ﬁtting, they no longer contribute to training as complexity penalizers.
It is well-known from the GP literature that marginal likelihood-based training is prone to overﬁtting for models with a large number of hyperparameters (Bauer et al., 2016).
2 We address this shortcoming by complementing the marginal likelihood objective of (3) with a penalty term derived from learning-theoretic ﬁrst principles.
We tailor the eventual loss only for robust model training and keep it maximally generic across learning setups.
This comes at the expense of arriving at a generalization bound that makes a theoretically trivial statement, yet brings signiﬁcant improvements to training quality as illustrated in our experiments.
PAC bounds have been commonly used for likelihood-free and loss-driven learning settings.
A rare exception by Germain et al.
(2016) proves the theoretical equivalence of a particular sort of PAC bound to variational inference.
Similarly, we keep the notion of a likelihood in our risk deﬁnition, but diﬀerently, we correspond our bound to the marginal likelihood.
Given a predictor h chosen from a hypothesis class H as a mapping from x to y , we deﬁne the true and the empirical risks as
[Equation]
and
[Equation]
for the data set D drawn from an arbitrary and unknown data distribution ∆ .
The risks
R ( h ) and R D ( h ) are bounded below by − max p ( y | h ( x )) and above by zero.
Although this setting relaxes the common assumption that bounds risk to the [0 , 1] interval, it is substantially simpler than the one suggested in (Germain et al., 2016), which deﬁnes
R ( h ) =
E x,y ∼ ∆ [log p ( y | h ( x ))]
∈ ( −∞ , + ∞ ) .
This unboundedness brings severe technical complications, which are no longer relevant for our approach.
Denoting by Q the distributions learnable over H and by P some regularizing distribution on H , according to Theorem 2.1in (Germain et al., 2009) we have for any δ ∈ (0 , 1] and any convex function d ( · , · )
[Equation]
where B = E D∼ ∆ (cid:2) E h ∼ P (cid:2) exp (cid:0) Nd ( R D ( h ) , R ( h )) (cid:1)(cid:3)(cid:3) .
Using a quadratic distance measure d ( x, y )
=
( x − y ) 2 and suitably bounding B by exploiting the boundedness of the likelihood and in turn the risk, we get as an upper bound on the expected true risk (see the appendix)
[Equation]
which is the objective we use to train BEDL that contains the marginal likelihood in the ﬁrst term and a regularizer in the second.
The additional term resembles the KL term in the
EDL loss, and gives a theoretically-grounded mechanism to incorporate regularization.
Computation of the Bound for Classiﬁcation.
For a C -class classiﬁcation we can compute the ﬁrst term in (6) as discussed above ((4)).
In the regularization term as we use (similar to EDL) P = D ir (cid:0) λ | (1 , . . . , 1) (cid:1) , i.e. the assumption that each class is equally likely, as the regularizing distribution.
Given Q
=
D
ir ( λ | α ) , KL ( Q (cid:107) P ) is analytically tractable, as is the last term where we use the upper bound log max( B ) ≤ N .
Looseness of the Bound.
It should be noted that while we use PAC theory to derive and motivate the ﬁnal objective, it should no longer be used in its PAC interpretation, as the approximations result in a loose bound that is trivially fulﬁlled.
Its justiﬁcation lies mainly in its regularizing function.
4. Experiments
We evaluate BEDL and its PAC-regularized version BEDL+Reg on several classiﬁcation tasks.
Complete details on the training procedure can be found in the appendix.
Additionally we provide there a discussion of the computational cost as well as extensions to regression and a comparison to GP based approaches.
Classiﬁcation and out-of-domain detection.
We train LeNet-5 networks on the MNIST train split, evaluate their classiﬁcation accuracy on the MNIST test split as the in-domain task, and measure their uncertainty on the Fashion-MNIST data set as the out-of-domain task, adhering to the protocol used in prior work (Louizos and Welling, 2017; Sensoy et al., 2018).
[Table]
We expect from a perfect model to predict true classes with high accuracy on the in-domain task and always predict a uniform probability mass on the out-of-domain task,
i
[Table]
e. the area under the curve of the empirical CDF (ECDF-AUC) of its predictive distribution entropy is zero.
We perform the same experiment on CIFAR10 using the ﬁrst ﬁve classes for the in-domain task and treating the rest as out-of-domain.
We use P :=
D ir (cid:0) λ | (1 , . .
, 1) (cid:1) as the regularization prior on the class assignment parameters, which has the uniform probability mass on its mean, encouraging an OOD signal in the absence of contrary evidence.
In
Table 1, we compare BEDL+Reg against EDL (Sensoy et al., 2018), also the non-Bayesian and heuristically trained counterpart of BEDL+Reg.
We consider EDL also as a special case of Prior Networks (Malinin and Gales, 2018) that does not need to rely on OOD data during training time, commensurate for our training assumptions.
We evaluate MC Dropout,
VarOut, and DVI as baselines in this setup.
BEDL+Reg improves the state of the art in all four metrics except the CIFAR10 in-domain task, where it ranks second after the prediction time weight sampling-based (hence less scalable) MC Dropout.
Remarkably, BEDL+Reg detects the OOD samples with signiﬁcantly better calibrated ECDF-AUC scores than EDL.
5. Conclusion
We present a method for performing Bayesian inference within the framework of evidential deep learning.
Employing type 2 maximum likelihood for inference and combining it with
PAC-bounds for regularization, we achieve higher accuracy and better predictive uncertainty estimates while maintaining scalable inference.
Exact inference in a fully Bayesian model such as a GP (c.f.
Table 2) or Hamiltonian Monte Carlo inference for BNNs (Bui et al., 2016) are known to provide better error rates and test log-likelihood scores, yet their computational demand does not scale well to large networks and data-sets.
Our method, on the other hand, shows strong indicators for improvement in uncertainty quantiﬁcation and predictive performance when compared to other BNN approximate inference schemes with reasonable computational requirements.
These beneﬁts of the BEDL+Reg framework might especially be fruitful in setups such as model-based deep reinforcement learning, active learning, and data synthesis, where uncertainty quantiﬁcation is a vital ingredient of the predictor.
Appendix A. Related Work
CLT-based moment matching.
The objective for variational inference on BNNs (Kingma et al., 2015; Wu et al., 2019; Gal and Ghahramani, 2016), optimizing a global variational posterior q ( w ) , consists of a computationally intractable E q ( w ) [log p ( y | x , w )] that decomposes across data points.
Fast dropout (Wang and Manning, 2013) approximates these terms via local reparameterization with moment matching.
The same local reparameterization has been later combined with a KL term to perform mean-ﬁeld VI via MC sampling (Kingma et al., 2015) or moment matching (Wu et al., 2019).
Our BLNN formulation is akin to an amortized VI approach learning a single global set of posterior parameters φ for the variational posterior approximation q φ ( w ) .
We use the same trick to marginalize the local weights, which keeps the machinery intact until the top-most step where the order of the log( · ) and E [ · ] operations is swapped.
This small change, however, has a large impact on the quality of uncertainty estimations.
Wide neural nets as GPs.
The equivalence of a GP to a weight-marginalized BNN with a single inﬁnitely wide hidden layer has been discovered long ago (Neal, 1995).
This result has later been generalized to multiple dense layers (Lee et al., 2018; Matthews et al., 2018), as well as to convolutional layers (Garragia-Alonso et al., 2019).
The asymptotic treatment of the neuron count makes this approach exact at the expense of a lack of neuronspeciﬁc parameterization.
The eventual GP has few hyperparameters to train, however, a prohibitively expensive covariance matrix to calculate.
We employ the same training method on a middle ground where the hyperparameter count is double as many as a deterministic net and the cross-covariances across data points are not explicitly modeled.
Predictive density modeling.
Neural Processes (Garnelo et al., 2018a,b) follow a GP inspired approach of learning the predictive density using neural networks as part of the mapping from input to output space relying on an incorporation of the input data as context points for the predictive distribution of a test point.
Earlier work on prior networks (Malinin and Gales, 2018) parameterizes a prior to a classiﬁcation-speciﬁc likelihood with deterministic neural nets, hence, discards model uncertainty.
Additionally, they require samples from another domain to learn the distributional awareness.
BEDL reformulates prior networks independently from the output structure, extends them to support also model uncertainty, and introduces a principled scheme for their training.
Appendix B. Uncertainty Decomposition.
Following Depeweg et al.
(2018) one can use the law of total variance to decompose the predictive variance for one data point ( x , y ) as follows
[Equation]
where the ﬁrst term, var w [ E [ y | w , x ]] , focuses on the contribution to this predictive uncertainty by the variance over the network weights, i.e. the epistemic uncertainty, while the second, E w [var [ y | w , x ]] , represents the remaining variance in the likelihood for the average weights.
After having marginalized over λ , the mean and variance E [ y | w , x ] , var [ y | w , x ] , are analytically tractable with
E [ y c | w , x ] =
α c α and var [ y c | w , x ] =
α c α (cid:16) 1 − α c α
[Equation]
(cid:17) .
The EDL formulation allows for an analytic computation of the predictive variance, however as it considers only deterministic weights, it gets for learned parameters ˆ w
[Equation]
i.e. only a measure of the aleatoric uncertainty, lacking the epistemic.
Our extension allows for the decomposition of the predictive uncertainty maintaining analytical tractability of the approximation to a great extent
[Equation]
where the ﬁnal variance and expectation can be eﬃciently approximated with samples as discussed above.
Appendix C. Extensions C.1. BEDL with Hyperpriors
Instead of relying on the PAC-bound based could try to incorporate further hyper-priors on φ .
This would hopefully reintroduce the missing regularization BEDL faces.
The hierarchical model then has the following structure:
[Equation]
The marginal to be optimized over is then given as
[Equation]
n d w n + log p ( φ ) .
The ﬁrst term is our regular marginal likelihood, while the second serves as as a regularizer as an optimization scheme aims to choose φ such that the marginal likelihood is high, but also that the prior density is large.
The form of this hyperprior will vary depending on the problem at hand, but if we consider e.g. the i -th weight of the BNN w in to follow a normal distribution, we have
[Equation]
where
[Equation]
An obvious choice for the prior p ( φ i ) is then given as
[Equation]
The regression results summarized in Table 1 in the main paper however show that this approach tends to perform worse than both the PAC regularized BEDL as well as the unregularized BEDL.
C.2. Generalization to Regression
The EDL formulation was introduced by Sensoy et al.
(2018) only for the case of classiﬁcations.
However, the main motivation of the approach can also be extended to the case of regression.
We place a normal likelihood over the targets, treating the λ n as the mean and another normal as the distribution over λ n parameterizing both mean and variance with a BNN, giving
[Equation]
with some ﬁxed observation precision β .
For the n -th sample in (3) with f Ln = ( f Ln 1 , f Ln 2 ) , m n = ( m n 1 , m n 2 ) , and s 2 n = ( s 2 n 1 , s 2 n 2 ) (the moment matching mean and variance), the log marginal likelihood is then given as
[Equation]
The approximation is computed via a ﬁnal moment matching step approximating the result of the inner integral p ( λ n ) =
(cid:82) p ( λ n | f
Ln ) p ( f Ln )
d f Ln with a normal distribution while the ultimate equality follows directly from standard results on normal distributions.
Contrary to the case of classiﬁcation where the ﬁnal step requires samples, the regression stays sampling-free.
We bound max B in (6) by exploiting that β is ﬁxed prior to training.
Consequently, we get log max( B ) N ≤ β 2 π as a bound, which, as for the classiﬁcation case, gives only a trivial performance guarantee (exceeding the maximum possible risk) but provides a justiﬁed training scheme.
C.3. Further Extensions
Adaptations of CLT-based recursive moment matching to many other activation types and skip connections are feasible without further approximations.
Max pooling can also be incorporated using approximations, but have also been shown to be replaceable altogether by strided convolutions without a performance loss (Springenberg et al., 2015).
Deeper networks tend to require normalization procedures, which are not directly amenable to this moment matching.
However tractable moments can also be computed for activation functions such as the ELU (Clevert et al., 2015) as we show in the appendix alleviating this constraint.
The variance computations in (8) and the ReLU speciﬁc post-activation do not model any potential covariance structure between the pre-/post-activations units of a layer.
While this is in principle feasible, e.g. along the lines of Wu et al.
(2019), it leads to an explosion in the required computational cost and memory, hindering the applicability of the approach to deeper nets.
Hence, we stick to a diagonal covariance structure throughout, as Wu et al.
(2019) have also shown only little test set performance beneﬁt of modeling it.
Appendix D. Further Details on the BLNN Derivations Derivation of the Moment Matching.
For a single data point and the l -th hidden fully-connected layer 5 consisting of K units with an arbitrary activation function a ( · ) , the post-activation layer output is given as h l = a ( f l ) ,
where f l = W l h l − 1 .
The j -th preactivation output f lj is a sum of K terms f lj = (cid:80)
k w ljk h l − 1 k , which allows us to assume it to be normal distributed via the CLT due to the independence of the individual w ljk and h l − 1 k terms.
The mean and the variance of this random variable can be computed as
[Equation]
where we drop any potential covariance structure between the outputs of a layer.
The mean and the variance of the weights are readily available via the distributions p φ ( w n ) .
For common activations such as the ReLU, a ( h l − 1 k ) = max(0 , h l − 1 k ) , which we will rely on in this work, closed-form solutions to the ﬁrst two moments of h l − 1 k are tractable (Frey and
Hinton, 1999) given the moments of the pre-activations of the previous layer f l − 1 .
This gives a recursive scheme terminating at the input layer , where f 1 j = (cid:80) k w 1 hj x k .
As x k is a constant, its ﬁrst moment is itself and the second is zero.
Consequently,
E
[Equation]
=
E [ w 1 jk ] x k and var (cid:2) f 1 j (cid:3) =
K (cid:88) k
= var[ w ljk ] x 2 k , completing the full recipe of how all weights of a BNN can be recursively integrated out from bottom to top, subject to a tight approximation.
Scenarios with stochastic input x ∼ p ( x ) typically entail controllable assumptions on p ( x ) .
The equations above remain intact after adding an expectation operator around x k and x 2 k , readily available for any explicitly deﬁned p ( x ) .
Contrarily to the case in GPs, stochastic inputs can be trivially adapted into this framework, greatly simplifying the math for uncertainty-sensitive setups.
For a net with L layers, the outcome is a distribution over the ﬁnal latent hidden layer
f
Ln ∼ N
( f Ln | m n , s 2 n ) , simplifying the highly nonlinear integrals in (2)
to
[Equation]
leaving us in a much simpler situation as we can choose a suitable distribution family for y n .
First two Moments of the ReLU Activation.
Mean and variance of a normally distributed variable transformed by the ReLU activation function are analytical tractable (Frey and Hinton, 1999).
Following the notation from the main paper, we have that for h l − 1 k
[Equation]
where f l − 1 k ∼ N ( f l − 1 k | µ, σ 2 ) for some mean and variance they can be computed as
[Equation] [Equation]
where Φ( · ) and φ ( · ) are the cdf and pdf of the standard normal distribution respectively.
First two Moments of the ELU Activation.
The following derivations are an adaptation of the ReLU results to ELU in order to scale to deeper networks.
We are again interested in the ﬁrst two moments for the ELU deﬁned as
[Equation]
With f ( x ) = max(0 , x ) , i.e. the ReLU activation, we have for the expectation of g ( · ) , that
[Equation]
x
[Equation]
The ﬁrst term can be split into
[Equation]
We get for these to terms that the ﬁrst is equal to
[Equation]
and the second gives
[Equation]
Combining all of this we end up with
[Equation]
For the variance we have the general form var [ g
[Equation]
in which only the second moment is missing.
We have that
[Equation]
The ﬁrst term expands into the following monstrosity
[Equation] [Equation]
These two moments ﬁnally can be used as replacements for the ReLU moments in the main paper for deeper networks.
Derivation of the Marginalization for Regression.
We approximate the marginal distribution
[Equation]
with a normal distribution by a further moment matching step.
Dropping the indices n and
L for notational simplicity, the mean of the right hand side is given as
[Equation]
For the variance term we rely on the law of total variance and have
[Equation]
where the last integral is given as the mean of a log-normal random variable.
Altogether we end up with the desired
[Equation]
This then allows us to compute the log marginal likelihood log p ( y n | x n ) =
[Equation]
Appendix E. Derivation of the PAC-Bound
This section gives a more detailed derivation of the individual results stated in the main paper.
As stated there, given a predictor h chosen from a hypothesis class H as a mapping from x to y , we deﬁne the true and the empirical risks as
[Equation]
for the data set D drawn from an arbitrary and unknown data distribution ∆ .
R ( h ) and
R D ( h ) are bounded below by − max p ( y | h ( x )) and above by zero.
Theorem 2.1 in (Germain et al., 2009) gives us that for any δ ∈ (0 , 1] and any convex function d ( · , · )
[Equation]
where B = E D∼ ∆ (cid:2) E h ∼ P (cid:2) exp (cid:0) Nd ( R D ( h ) , R ( h )) (cid:1)(cid:3)(cid:3) .
The PAC framework necessitates a convex and non-negative distance measure for risk evaluations.
Common practice is to rescale the risk into the unit interval, deﬁne the KL divergence as the distance measure, and upper bound its intractable inverse (Germain et al., 2016) using Pinsker’s inequality (Catoni, 2007;
Dziugaite and Roy, 2017).
We follow an alternative path.
As our risk is bounded but not restricted to the unit interval, we choose our distance measure as d ( r, r (cid:48) )
=
( r − r (cid:48) ) 2 and avoid the Pinsker’s inequality step.
Adapting the standard KL inversion trick (Seeger, 2002) to the Euclidean distance, we can simply deﬁne d − 1 ( x, ε ) =
max { x (cid:48) : ( x − x (cid:48) )
2 = ε } = x + √ ε for some ε ≥ 0 .
We apply this function to both sides of the inequality and get
[Equation]
where d ( · , · ) ≥ 0 and KL ( Q || P ) ≥ 0 by deﬁnition and because δ ∈ [0 , 1] and e d ( · , · ) ≥
we have log( B/δ ) =
− log δ + log B ≥ 0 .
Since
[Equation]
directly follows from d − 1 ( · , · ) , we bound the true risk as
[Equation]
This outcome has a similar structure to application of Pinsker’s inequality to a setup with risk deﬁned on the unit interval, but without such a restriction.
Hence, the implied upper bound is no longer trivial.
To arrive at the ﬁnal bound we have to further approximate each of the two terms of the bound.
For the ﬁrst term, we have that
[Equation]
where the inequality uses that − log( u ) > − u ∀ u ∈ (0 , ∞ ) and the equality follows by the marginalisation techniques discussed in the main paper.
To get a tractable second term we process B further.
Exploiting the fact that
[Equation]
for any p ( · ) and f ( · )
, we can drop the the expectation term and get
[Equation]
For a multiclass classiﬁcation, the likelihood is bounded into the interval [0 , 1] such that with max( R D ( h ) − R ( h ))
=
(0 − 1) 2 =
we have that
[Equation]
For regression, with the likelihood N ( y | h ( x ) , β − 1 ) , we have that R D ( h ) and R ( h ) are bounded from above by 0 and from below by −N (cid:0) y
= µ | µ, β − 1 (cid:1) , i.e. by the density at the mode of a normal distribution with precision β − 1 .
Hence,
[Equation] [Equation]
Combining these relaxations we get the objectives described in the main paper.
Appendix F. Experimental Details and Further Experiments
This section contains further experiments as well as details on the hyperparameters of the experiments performed in the main paper.
F.1. Regression
We evaluate the regression performance of BEDL+Reg and the baselines on eight standard
UCI benchmark data sets.
Adopting the experiment protocol introduced in (HernándezLobato and Adams, 2015), we use 20 random train-test set splits comprising 90% and
[Table]
10% of the samples, respectively.
The nets consist of a single hidden layer with 50 units and ReLU nonlinearities.
6 The hypothesis class in this task is over the regularization parameters P :=
p ( λ ) =
(cid:81) n N ( λ n | 0 , α − 1 ) , with precision α , while Q is given as Q :=
p ( λ ) = (cid:81) n (cid:82) p ( λ n | f n ) p ( f n ) d f n .
We compare BEDL+Reg against the state of the art in BNN inference methods that do not require sampling across neural net weights, Probabilistic BackPropagation (PBP) (Hernández-Lobato and Adams, 2015) and Deterministic Variational
Inference (DVI) (Wu et al., 2019), which use the CLT-based moment matching for expectation propagation and VI, respectively.
For completeness, we also compare against the two most common sampling-based alternatives, Variational Dropout (VarOut) (Kingma et al., 2015;
Molchanov et al., 2017) and MC Dropout (Gal and Ghahramani, 2016).
In the results summarized in Table 2, BEDL+Reg outperforms all baselines in the majority of the data sets and is competitive in the others.
The PAC regularization improves over BEDL in all data sets except one.
We also report results for a sparse GP with 50 inducing points, which approximates a BNN of one inﬁnitely wide hidden layer (Neal, 1995).
As expected, the GP sets a theoretical upper bound on
BEDL+Reg as well as the baselines for one hidden layer architectures.
Lastly, we compare our tediously derived PAC regularizer to straightforward Maximum-A-Posteriori estimation on the BEDL hyperpriors (BEDL-Hyper) (see the appendix for details), which deteriorates performance on all UCI data sets.
Experimental Setup.
The neural net used consists of a single hidden layer of 50 units for all data sets except protein , which gets 100.
The results for all of the baselines except for Variational Dropout (VarOut) are quoted from the results reported by the respective papers who introduced them, while the results on the sparse GP are reported via (Bui et al., 2016).
For VarOut we rely on our own implementation as there are no oﬃcial results.
BEDL,
BEDL+Reg, and VarOut all share the same initialization scheme for the mean and variance parameters for each weight following the initialization of (Louizos and Welling, 2017), i.e.
He-Normal for the means and N ( − 9 , 0 .
001) for the log variances.
VarOut gets a Normal prior with a precision of α = 1 . 0 , and all three get an observation precision of β = 100 , to encourage them to learn as much of the predictive uncertainty instead of relying on a ﬁxed hyper-parameter.
Note that we keep these values ﬁx and data set independent, diﬀerent to many of the baselines who set them to data set speciﬁc values given cross-validations on separate validation subsets.
Each model is trained with the Adam optimizer with default parameters for 100 epochs with a learning rate of 10 − 3 , with varying minibatch sizes depending on the data set size.
F.2. Classiﬁcation and Out-of-domain Detection.
The network for this task follows the common LeNet5 architecture with the following modiﬁcations.
Instead of max-pooling layers after the two convolutional layers, the convolutional layers themselves use a larger stride to mimic the behavior.
And for the more complex
CIFAR data set the number of channels in the two convolutional layers is increased from the default 20,50 to 192 each, while the number of hidden units for the fully connected layer is increased from 500 to 1000 for that data set following (Gal and Ghahramani, 2015).
Since there are no OOD results on the BNN baselines we compare against, we rely on our own reimplementations of them, ensuring that they each share the same initialization schemes as in the regression setup.
For DVI we implement the diagonal version and use a sampling-based approximation on the intractable softmax.
Each model gets access to ﬁve samples whenever it needs to conduct an MC sampling approximation.
All models get trained via the Adam optimizer with the default hyperparameters and a learning rate of 10 − 3 .
For EDL we rely on the public implementation the authors (Sensoy et al., 2018) provide and use their hyperparameters to learn the model.
Due to the license status of the not-MNIST data conﬂicting with the aﬃliation of the authors, we have to change the setup of earlier work, e.g. (Lakshminarayanan et al., 2017; Louizos and Welling, 2017; Sensoy et al., 2018), using instead Fashion-MNIST as the closest substitute.
F.3. Comparison to GP variants.
We evaluate the impact of local weight realization on prediction performance by comparing
BEDL+Reg to GPs with kernels derived from BNNs with global weight realizations (GarragiaAlonso et al., 2019; Lee et al., 2018; Neal, 1995) on MNIST and CIFAR10 data sets.
It is technically not possible to perform this evaluation in a fully commensurate setup, as these baselines assume inﬁnitely many neurons per layer and do not have weight-speciﬁc degrees of freedom.
Furthermore, Garragia-Alonso et al.
(2019) perform neural architecture search and Lee et al.
(2018) use only part of the CIFAR10 training set reporting that the rest does not ﬁt into the memory of a powerful workstation.
We nevertheless view the performance scores reported in these papers as practical upper bounds and provide qualitative comparison.
For the choice of neural net depth, we take NNGP (Lee et al., 2018) as a reference and devise a contrarily thin two-layer convolutional BEDL+Reg network.
The results and the architectural details are summarized in Table 3.
BEDL and BEDL+Reg can reach lower error rates using signiﬁcantly less computational resources.
Experimental Setup.
The results for the baselines are taken from the respective original papers.
The nets for BEDL and BEDL+Reg consist of two convolutional layers withﬁlters of size 5 × 5 and a stride of 5.
They are trained until convergence (50 epochs) using
Adam with the default hyperparameters and a learning rate of 10 − 3 .
[Table]
BEDL reaches lower error rate than previously proposed neural net based GP constructions by two convolutional layers with 96 ﬁlters of size 5 × stride
BEDL converges in 50 epochs, amounting to circa 30 minutes of training time on a single GPU.
The GP alternatives have been reported to have signiﬁcantly larger time and memory requirements.
The GP results are cited from (Garragia-Alonso et al., 2019)
F.4. Computational cost.
Table 4 summarizes the computational cost analysis of the considered approaches.
MC
Dropout and VarOut can quantify uncertainty only by taking samples across weights, which increases the prediction cost linearly to the sample count.
DVI and BEDL+Reg perform the forward pass during both training and prediction time via analytical moment matching at double and triple costs, respectively.
Both methods have sampling costs for intractable likelihoods.
7 BEDL+Reg may also have another additive per-data-point sampling cost for calculating intractable functional mapping regularizers.
Favorably, both of these overheads are only additive to the forward pass cost, i.e. sampling time is independent of the neural net depth, hence they do not set a computational bottleneck.
The training and prediction cost of BEDL+Reg is three times EDL which builds on deterministic neural nets.
However, it provides substantial improvements in both prediction accuracy and uncertainty quantiﬁcation.
[Table]