B AYES A DAPTER : B EING B AYESIAN , I NEXPENSIVELY AND R OBUSTLY , VIA B AYESIAN F INE - TUNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite their theoretical appealingness, Bayesian neural networks (BNNs) are falling far behind in terms of adoption in real-world applications compared with deterministic NNs, mainly due to their limited scalability in training and low ﬁdelity in uncertainty estimates.
In this work, we develop a new framework, named
BayesAdapter , to address these issues and bring Bayesian deep learning to the masses.
The core notion of BayesAdapter is to adapt pre-trained deterministic
NNs to be BNNs via Bayesian ﬁne-tuning .
We implement Bayesian ﬁne-tuning with a plug-and-play instantiation of stochastic variational inference, and propose exemplar reparameterization to reduce gradient variance and stabilize the ﬁnetuning.
Together, they enable training BNNs as if one were training deterministic
NNs with minimal added overheads.
During Bayesian ﬁne-tuning , we further propose an uncertainty regularization to supervise and calibrate the uncertainty quantiﬁcation of learned BNNs at low cost.
To empirically evaluate BayesAdapter , we conduct extensive experiments on a diverse set of challenging benchmarks, and observe satisfactory training efﬁciency, competitive predictive performance, and calibrated and faithful uncertainty estimates.
I NTRODUCTION
Much effort has been devoted to developing ﬂexible and efﬁcient Bayesian deep models to make accurate, robust, and well-calibrated decisions (MacKay, 1992; Neal, 1995; Graves, 2011; Blundell et al., 2015), with Bayesian neural networks (BNNs) as popular examples.
The principled uncertainty quantiﬁcation inside BNNs is critical for realistic decision-making, well evaluated in scenarios ranging from model-based reinforcement learning (Depeweg et al., 2016) and active learning (Hern ´ andez-Lobato & Adams, 2015), to healthcare (Leibig et al., 2017) and autonomous driving (Kendall & Gal, 2017).
BNNs are also known to be capable of resisting over-ﬁtting.
However, there are fundamental obstacles posed in front of ML practitioners when trying to push the limit of BNNs to larger datasets and deeper architectures: ( i ) The scalability of the existing BNNs is generally restrictive owing to the essential difﬁculties of learning a complex, non-degenerate distribution over parameters in a high-dimensional and over-parameterized space (Liu & Wang, 2016;
Louizos & Welling, 2017; Sun et al., 2019).
( ii ) The Bayes posteriors learned from scratch are often systematically worse than their point-estimate counterparts in terms of predictive performance when “cold posterior” strategies are not applied (Wenzel et al., 2020).
( iii )
It is shown that the BNNs have the possibility to assign low ( epistemic ) uncertainty for realistic out-of-distribution (OOD) data (e.g., adversarial examples), rendering their uncertainty estimates unreliable in safety-critical scenarios (Grosse et al., 2018).
To solve these problems, we present a scalable workﬂow, named BayesAdapter , to learn more reliable BNNs.
In a holistic view, we unfold the learning of a BNN into two steps: deterministic pre-training of the deep neural network (DNN) counterpart of the BNN followed by several-round
Bayesian ﬁne-tuning .
This enables us to learn a principled BNN with slightly more efforts than training a regular DNN, and provides us with the opportunities to embrace qualiﬁed off-the-shelf pre-trained DNNs (e.g., those on PyTorch Hub).
The converged parameters of the deterministic model serve as a strong start point for Bayesian ﬁne-tuning , allowing us to bypass extensive
local
[Figure]
optimum suffered by a direct learning of BNN 1 .
To render the ﬁne-tuning in the style of training normal NNs, we resort to stochastic variational inference (VI) to update the approximate posterior.
We develop optimizers with built-in weight decay for the parameters of the variational distribution to absorb the regularization effects from the prior, and develop exemplar reparametrization to reduce the gradient variance.
Moreover, to make the uncertainty estimation of the learned models reliable, we propose to additionally, explicitly regularize the model to behave uncertainly on representative foreseeable OOD data during ﬁne-tuning.
This regularization takes the form of a margin loss, and is readily applicable to most of the existing BNNs.
Figure 1 depicts the whole framework of BayesAdapter .
Extensive empirical studies validate the efﬁciency and effectiveness of our workﬂow.
In summary, our contributions are as follows:
We propose BayesAdapter , to quickly and cheaply adapt a pre-trained DNN to be Bayesian without compromising performance when facing new tasks.
We provide an easy-to-use instantiation of stochastic VI, which allows learning a BNN as if training a deterministic NN and frees the users from tedious details of BNN.
We augment the ﬁne-tuning with a generally applicable uncertainty regularization term to rectify the predictive uncertainty according to a collection of OOD data.
Extensive studies validate that BayesAdapter is scalable; the delivered BNN models are high-quality; and the acquired uncertainty quantiﬁcation is calibrated and transferable.
B AYES A DAPTER
In this section, we ﬁrst motivate BayesAdapter by drawing a connection between maximum a posteriori (MAP) and Bayesian inference.
We then describe the proposed procedure Bayesian ﬁne-tuning , and a practical and robust implementation of stochastic VI to realize it.
Figure 1 illustrates the overall workﬂow of BayesAdapter .
2. F ROM DNN S TO BNN S
Let D =
( x i , y i ) } ni =1 be a given training set, where x i ∈ R d and y i
∈ Y denote the input data and label, respectively.
A DNN model can be ﬁt via MAP as following:
[Equation]
We use w ∈ R p to denote the high-dimensional model parameters, and p ( y | x ; w ) as the predictive distribution associated with the model.
The prior term p ( w ) , when taking the form of an isotropic
Gaussian, reduces to the common L2 weight decay regularizer in optimization.
Despite the wide adoption, DNNs are known to be prone to over-ﬁtting, generating over-conﬁdent predictions, and are unable to convey valuable information on the trustworthiness of their predictions.
Naturally,
Bayesian neural networks (BNNs) come into the picture to address these limitations.
Typically, a BNN imposes a prior p ( w ) on model parameters, which is put together with the likelihood p ( D| w ) to infer the posterior p ( w |D ) .
Among the wide spectrum of BNN algorithms (MacKay, 1992; Neal, 1995; Graves, 2011; Blundell et al., 2015; Liu & Wang, 2016; Gal & Ghahramani, 2016; Louizos & Welling, 2017), variational BNNs are particularly promising due to their ease of training compared with other BNN variants.
Formally, variational BNNs derive a θ -parameterized varitional distribution q ( w | θ ) to approximate the true posterior p ( w |D ) , by maximizing the evidence lower bound (ELBO) (scaled by 1 /n ):
[Equation]
where L ell is the expected log-likelihood and L c is the complexity loss .
By casting posterior inference into optimization, Eq.
(2) makes the training of BNNs more approachable.
However, most existing
BNNs 2 trained under such a criterion exhibit limitations in scalability and performance (Osawa et al., 2019a; Wenzel et al., 2020) compared with their deterministic counterparts, mainly attributed to the higher difﬁculty of learning high-dimensional distributions than point estimates, and challenges in ﬁnding non-degenerated optima of highly nonlinear functions characterized by NNs.
Given that MAP converges to the mode of the Bayesian posterior, it might be plausible to adapt pretrained deterministic DNNs to be Bayesian economically.
Following this hypothesis, we propose to repurpose the converged parameters w ∗ of MAP, and use it to instantiate q ( w | θ ) as a Gaussian
N ( w ; θ ) with θ = ( µ , Σ ) , where µ is initialized as w ∗ and Σ ∈ R p × p denotes the covariance.
Then, we arrive at a BNN with posterior predictive :
[Equation]
p
[Equation]
y | x ; w ( s ) ) , where w ( s ) ∼ N ( w ; µ , Σ ) , s = 1 , ..., S. (3)
Eq.
(3) is also called Bayes ensemble , where µ is perturbed, and the predictions from multiple likely models are assembled.
Σ controls the magnitude of perturbation.
A classic method to generate an informative Σ is by Laplace approximation (Bleistein & Handelsman, 1986), but it is more like a postprocessing procedure, lacking the ﬂexibility to jointly adapt the mean and covariance of the
Gaussian posterior w.r.t.
data, and its naive implementation without strong assumptions may be computationally prohibitive.
Instead, we suggest a more practical workﬂow – that ﬁne-tunes the approximate posterior N ( w ; µ , Σ ) by maximizing the ELBO with randomly initialized Σ .
2. B AYESIAN F INE - TUNING IN THE STYLE OF F INE - TUNING DNN S
We develop practical learning algorithms under the stochastic VI scheme to ﬁne-tune the imperfect variational posterior, and to cope with contemporary ML frameworks.
In the following, we discuss how to deal with each term in Eq.
Algorithm 1 gives an overview of BayesAdapter .
Complexity loss L c .
Without losing generality, we assume an isotropic Gaussian prior p ( w ) =
N ( w ; 0 , σ 20 I ) .
Then the complexity loss is derived as:
[Equation]
+
[Equation]
4) where tr and det are matrix trace and determinant, respectively.
c is a constant.
The gradients of
L c w.r.t.
µ and Σ can be estimated precisely as:
[Equation]
Eq.
(5) indicates that max µ L c amounts to applying a weight decay regularizer with coefﬁcient λ
=σ 20 n on µ , which can be conveniently optimized by leveraging the built-in weight decay modules in
ML frameworks such as TensorFlow (Abadi et al., 2016) or PyTorch (Paszke et al., 2019).
Directly computing ∇ Σ L c involves matrix inversion.
Implementing the posterior as matrix-variate
Gaussian is an alternative, while existing algorithms for matrix-variate Gaussian posterior typically exhibit high complexity in time or memory, limited compatibility with contemporary NN building block operations (e.g., convolution), and struggle to scale with data-parallel distributed training (Louizos & Welling, 2016; Sun et al., 2017; Osawa et al., 2019b).
To simplify the implementation and boost scalability, we assume a fully factorized Gaussian variational by devising Σ as diag (exp(2 ψ )) , where ψ ∈ R p is the parameter to be optimized along with µ (i.e., θ = ( µ , ψ ) ).
Injecting this into Eq.
(5) gets a more concise gradient estimator: ∇ ψ L c = 1 /n − λ exp(2 ψ ) , meaning that max ψ L c adds an exponential weight decay of ψ with coefﬁcient λ , which can be realized by modifying only two lines of code on top of de facto DL frameworks (see Figure 1).
Expected log-likelihood L ell .
With the complexity loss expressed as weight decay, we now develop efﬁcient ways for calculating the L ell at the end of forward pass, and for performing backpropagation afterwards.
In particular, we derive a Monte Carlo (MC) estimation of L ell based on reparameterization (Kingma & Welling, 2013): we sample a p -dimensional Gaussian noise (cid:15) ∼ N ( 0 , I ) , then obtain the sampled parameter for the whole mini-batch B of data via w = µ +exp( ψ ) (cid:15) , given which we approximate L ell with L (cid:48) ell = 1 |B| (cid:80) ( x i ,y i ) ∈B log p ( y i | x i ; w ) .
The gradients of µ and ψ can be derived automatically with autodiff libraries, thus the training resembles that of normal DNNs.
However, gradients derived by L (cid:48) ell might exhibit high variance, caused by sharing one set of sampled parameters w across all the training instances in B .
Local reparameterization is proposed to reduce the variance, but it requires at least 2x forward-backward FLOPS than vanilla reparameterization (refer to Kingma et al.
(2015) for more details).
Flipout (Wen et al., 2018) is an alternative solution.
But it is only suitable for perturbation based MC estimation and its modeling assumptions make Flipout unable to handle complex variational posterior like a FLOW (Louizos & Welling, 2017), or an implicit model (Shi et al., 2018b).
Besides, it is still as slow as local reparameterization .
To mitigate these issues, we propose exemplar reparametrization (ER) which samples a separate set of parameters for every exemplar in the minibatch.
Formally, for ∀ x i ∈ B , we draw w ( i ) = µ + exp( ψ ) (cid:15)
( i ) where (cid:15) ( i ) ∼ N ( 0 , I ) , and approximate the expected log-likelihood by
L ∗ ell = 1 |B| (cid:80) ( x i ,y i ) ∈B log p ( y i | x i ; w ( i ) ) .
[Figure]
Obviously, ER is distribution agnostic, and is readily applicable to various variational distributions.
While ER generates more parameters at training, they are mostly temporary, and the resultant computational FLOPS are provably identical to that of the vanilla reparameterization.
The challenge of ER is to cope with nowadays ML frameworks and maintain computing efﬁciency, because off-the-shelf computation kernels in autodiff libraries typically assume a batch of instances share a common set of parameters.
We present an example in Figure 2 on how the standard convolution op can be converted into its exemplar version without compromising computational efﬁciency.
The key insight here is that multiple exemplar convolutions can be expressed as a group convolution, which can be performed in parallel using a single group convolution kernel, leveraging the optimized implementations provided by various device-propriety kernel backends (e.g. cuDNN (Chetlur et al., 2014)).
Other common operators such as matrix multiplication are straightforward to handle (refer to Appendix A).
With this insight, BayesAdapter enables to obtain a BNN with only minor computational cost in addition to pre-training, and can immediately beneﬁt from the availability of higher-performance computational kernels (e.g., more powerful group convolution kernel).
C ALIBRATE THE U NCERTAINTY E STIMATION
So far we have developed an inexpensive ﬁne-tuning procedure to obtain BNNs from deterministic
NNs.
While BNNs can offer uncertainty estimates, these uncertainty measures are highly nonsmooth due to the non-convexity of NNs – they might exhibit high uncertainty for data from faraway out-of-distribution (OOD) regions, but become vulnerable on OOD samples close to the normal ones (Grosse et al., 2018), rendering BNNs unable to react to potentially harmful inputs.
We quantify this phenomenon in Section 4.2.
To address this problem, we next develop methods to further calibrate the uncertainty estimation of naively trained BNNs.
Inspired by recent work on OOD detection (Wang et al., 2020; Durall et al., 2020), we propose to additionally incorporate uncertainty regularization on top of the above ﬁne-tuning procedure.
The idea is to force BNNs to generate inconsistent predictions for each sample from a cheaply collected OOD sample set, so that they acquire the ability to yield high uncertainty for OOD samples with similar ﬁngerprints.
To achieve this, we start by deﬁning a differentiable uncertainty metric in terms of mutual information, following Smith & Gal (2018):
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
w where
[Equation]
H is the Shannon entropy.
I highly correlates with softmax variance (Smith & Gal, 2018), and measures the epistemic uncertainty which describes uncertainty in the model and can be used to identify OOD instances.
Then, assuming access to an OOD dataset D † = { x † i } n † i =1 , we enforce the model to behave uncertainly on each of them by optimizing a margin loss with threshold γ :
[Equation]
where B † refers to a mini-batch of OOD data.
For efﬁciency, we adopt S = 2 MC samples for estimating I ( w , y | x † i , D ) in Eq.
(7) in the training.
While this loss has a seemingly opposite form from the consistency-promoting loss in semi-supervised learning (SSL) (Laine & Aila, 2016), they share the same design philosophy: L unc maximizes the prediction inconsistency of OOD instances so as to distinguish them from in-distribution instances, while SSL minimizes the prediction inconsistency of unlabeled data so to classify them without labels.
Put it in the context of autonomous driving: if the model is trained on data only containing scenes in regular weather, we can take a small set of scene data of extreme weather, e.g., tornado and sandstorm, to regularize the training following
Eq.
Then the model will learn to identify these abnormal scenes based on predictive uncertainty, thus can refuse to make unreliable decisions in these scenes.
Constructing the OOD dataset D † is ﬂexible and application-speciﬁc.
In discriminative tasks, two types of OOD data of particular concerns are adversarial and fake samples, which can be both collected trivially following procedures described below.
Adversarial samples.
Directly generating adversary samples following methods like PGD (Madry et al., 2017) might be expensive.
We propose a more cost-effective alternative based on a key observation: given a valid perturbation space [ − δ m , δ m ] d where δ m is the maximum norm under the l ∞ threat model, we can see that uniform noises δ ∼ U ( − δ m , δ m )
d radically encompass the adversarial perturbations which usually reside at local optimas.
Thus we can add uniformly perturbed samples into uncertainty training to direct the model to behave uncertainly on randomly contaminated data, bypassing the potential cost of generating real adversary samples.
The results in Sec 4.2 surprisingly conﬁrm the effectiveness of uniform noises, and imply a strong connection between uniform noises and adversarial ones, which deserves a future investigation.
Fake samples.
Fake samples can be obtained by utilizing pretrained state-of-the-art GANs (Miyato et al., 2018; Brock et al., 2018), DeepFake (Deepfakes, 2018), and FaceSwap (Faceswap, 2018).
We use only 1000 random fake samples for Bayesian ﬁne-tuning on diverse benchmarks.
For both, we empirically ﬁnd the proposed uncertainty regularization is data efﬁcient – with access to a proxy set of adversarial samples and a small set of fake samples, the model can acquire reliable, transferable uncertainty quantiﬁcation.
E XPERIMENTS
In this section, we evaluate BayesAdapter on a diverse set of challenging benchmarks.
[Table] [Table]
Settings.
We ﬁrst conduct experiments on CIFAR-10 (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009) using wide-ResNet-28-10 (Zagoruyko & Komodakis, 2016) and ResNet-50 (He et al., 2016), respectively.
Besides, we train face recognition models on CASIA (Yi et al., 2014) with MobileNetV2 (Sandler et al., 2018), and perform comprehensive evaluation on face veriﬁcation datasets including LFW (Huang et al., 2007), CPLFW (Zheng & Deng, 2018), CALFW (Zheng et al., 2017),
CFP (Sengupta et al., 2016), VGGFace2 (Cao et al., 2018), and AgeDB-30 (Moschoglou et al., 2017).
We pre-train DNNs following standard protocols, and perform Bayesian ﬁne-tuning for 40, 12, and 16 epochs with weight decay coefﬁcients (i.e., λ ) 2e-4, 1e-4, and 5e-4 on these benchmarks respectively.
We set uncertainty threshold γ according to an observation that the normal examples usually present < 0 .
75 mutual information uncertainty, across the studied scenarios.
Then we use γ = 0 .
75 in the regularization to push both the adversarial and fake data to exhibit distinguishable uncertainty from the normal data.
We initialize ψ uniformly from [ − 6 , − 5] p and use 20-step
PGD as validation adversaries.
On the three benchmarks, the perturbation budgets δ m are set 0.031, 16/255, and 16/255, and the fake samples are obtained from SNGAN (Miyato et al., 2018), BigGAN (Brock et al., 2018), and DeepFake.
We perform intensive data augmentation for fake training data with a random strategy including Gaussian blur, JPEG compression, etc.
We defer more details to Appendix B. We run every experiment 3 times on 8 RTX 2080-TIs and report the average.
Baselines.
We compare the full BayesAdapter to extensive baselines including: (1) MAP , (2)
Laplace Approx.
: Laplace Approximation with diagonal Fisher information matrix, (3) MC dropout (detailed in Appendix B), (4) BNN :
BNNs trained from scratch by solving Eq.
(2) without uncertainty regularization, (5) BayesAdapter- : a variant of BayesAdapter without uncertainty regularization.
We also include Deep Ensemble (Lakshminarayanan et al., 2017), one of the state-of-the-art
BNNs, and SWAG (Maddox et al., 2019), whose performance is not worse than SGLD (Welling &
Teh, 2011), KFAC Laplace (Ritter et al., 2018), and temperature scaling (Guo et al., 2017), into the comparison on CIFAR-10 3 .
Metrics.
We concern ( i ) the posterior predictive performance with S = 100 MC samples; ( ii ) the average precision (AP) of directly using the uncertainty estimated by Eq.
(6) ( S = 20 ) to distinguish
OOD test samples (labeled 1) from normal test samples (labeled 0).
Eq.
(6) of the deterministic baseline MAP is 0, so we take the predictive entropy as an alternative uncertainty measure for MAP .
4. P REDICTIVE P ERFORMANCE
We compare the prediction performance, which is of central importance in practice, of various methods in Table 1 and 2.
Deep Ensemble shows outperforming classiﬁcation performance because the ensemble candidates can investigate diverse function modes, but it is orders of magnitude more expensive than BayesAdapter .
BayesAdapter- notably surpasses the MAP , especially in NLL, verifying the modeling superiority of a Bayesian formulation and highlights the practical value of our workﬂow.
Laplace Approx.
is consistently worse than MAP .
In all settings, BNN is signiﬁcantly defeated by BayesAdapter- , conﬁrming our claim that performing Bayesian ﬁne-tuning from the [Figure] n [Figure]
[Figure]
D e n s i t y [Figure] [Table] converged deterministic checkpoints is beneﬁcial to bypass the local optimas potentially encountered by direct Bayesian inference.
The popular baselines MC dropout and SWAG show weaker performance on ImageNet and CIFAR-10, respectively, revealing limited applicability.
Also of note that no method shows dominant performance on face recognition, probably due to the diversity of these validation sets.
Across these tasks, BayesAdapter is slightly worse than its regularization-free version BayesAdapter- .
This is reasonable since such a regularization enforces the model to trade partial capacity for ﬁdelity of uncertainty estimates.
Nevertheless, BayesAdapter is substantially better than its ﬁne-tuning start point MAP and the BNN trained from scratch in most settings.
Speedup.
BayesAdapter is a much more economical way to obtain BNNs.
To interpret the speedup of BayesAdapter over BNN , we assume deterministic ResNet-50 takes one unit time t for one epoch of training on ImageNet, and observe Bayesian ResNet-50 takes ≈ 2 .
1 t for one epoch.
Thus, BNN trained from scratch consumes 189 t for 90-epoch training, while BayesAdapter- need t ∗ 90 + 2 .
1 t ∗ 12 = 115 .
2 t , saving 73 .
8 t (around 40 % ) training time than BNN .
4. Q UALITY OF U NCERTAINTY E STIMATES
We study the effects of the proposed uncertainty regularization by visualizing the predictive uncertainty of BayesAdapter and BayesAdapter- on validation data in Figure 3.
On both CIFAR-10 and
ImageNet, BayesAdapter yields evidently higher uncertainty for OOD data than normal data, while
BayesAdapter- is on the contrary, showing it can effectively calibrate the predictive uncertainty.
To precisely evaluate its efﬁcacy, we quantitatively assess the quality of the predictive uncertainty of various methods by estimating AP, which reﬂects if the model knows what it knows .
As stated, we take adversarial samples crafted by PGD and fake samples from GANs and DeepFake as proxies of harmful OOD data.
We report the results in Table 3 and Table 6 in Appendix C. As shown, SWAG ,
Laplace Approx.
, MC dropout , BNN , and BayesAdapter- perform all as bad as MAP across settings, except that MC dropout is capable of partially detecting OOD data on face tasks 5 .
Despite impressive prediction accuracy, Deep Ensemble also yields unreliable uncertainty estimates on these two kinds of challenging OOD data.
These results echo our concern on the reliability of existing BNNs’ predictive uncertainty.
By contrast, BayesAdapter , which is ﬁne-tuned upon MAP for only several rounds based on low-cost supervisions, achieves near-perfect results in detecting OOD instances on CIFAR-10 and face recognition, and also detects most of the OOD instances on ImageNet (see Appendix D for some samples).
4. A BLATION S TUDY
[Table]
Model calibration.
Model calibration is another important aspect of the uncertainty estimation.
Suggested by pioneering works, we take the
Expected Calibration Error (ECE) (Guo et al., 2017) as the measure of calibration, and report the ECE of the studied methods in Table 4.
The ECE of BayesAdapter is on par with the MC dropout , Deep Ensemble , and BNN baselines, but
BayesAdapter can meanwhile offer much better uncertainty for detecting risky OOD data.
Transferability of uncertainty quantiﬁcation.
One may wonder if the uncertainty quantiﬁcation learned according to specialized OOD data can generalize to other OOD data.
To ﬁgure out this problem, we evaluate the BayesAdapter trained on CIFAR-10 on 10000 samples from PGGAN (Karras et al., 2017) whose patterns are unseen during training.
We compute their uncertainty and calculate the AP metric, obtaining 0.932 .
As comparison, the APs of MAP , Deep Ensemble , SWAG , Laplace Approx.
, MC dropout , BNN , BayesAdapter- on such data are 0.789, 0.797, 0.809, 0.800, 0.792, 0.793, 0.803 respectively.
On the other hand, we craft adversarial examples by the fast gradient sign method (FGSM)
(Goodfellow et al., 2014) against the ResNet-152 DNN model with 1000 validation images from ImageNet.
Then we estimate the AP on these instances, and obtain 0.011, 0.125, 0.027, 0.202, 0.019, and 0.882 for MAP , Laplace Approx.
, MC dropout , BNN , BayesAdapter- , and BayesAdapter respectively.
These studies validate the transferability of our uncertainty estimation.
The effectiveness of exemplar reparameterization.
We build a toy model with only a Bayesian convolutional layer, ﬁxing the model input and target output, and computing the variance of stochastic gradients across 500 runs.
We average the gradient variance of µ and ψ over all their coordinates, and observe that standard reparameterization typically introduces more than 100 × variance than exemplar reparameterization , despite with the same FLOPS.
[Table]
Ablation study on uncertainty threshold γ .
We perform an ablation study regarding γ on CIFAR10 to evaluate the hyper-parameter tolerance of the proposed method.
Table 5 presents the results.
The results reveal that values of γ ∈ [0 .
75 , 1 . 0] may be good choices for OOD detection, and also echo the observation that normal examples usually present < 0 .
75 uncertainty.
[Figure]
The impacts of ensemble number.
We draw the change of test accuracy w.r.t.
the number of MC samples S for estimating Eq.
(3) in Figure 4.
The model is trained by BayesAdapter on ImageNet.
The points on the red line represent the individual accuracies of the 100 parameter samples.
The yellow dashed line refers to the deterministic inference with only the Gaussian mean.
The green line displays the effects of Bayes ensemble – the predictive performance increases from < 74% to > 76% quickly before seeing 20 parameter samples, and gradually saturate after that.
That is why we usesamples for estimating uncertainty and crafting adversarial samples. [Figure] .2e [Figure]
0e
[Figure]
Uncertainty-based rejective decision.
In practice, we expect our models can reject to predict for data with relatively large uncertainty, and only care about the data that they are certain about.
In this spirit, we sort the validation data of ImageNet w.r.t.
the uncertainty provided by BayesAdapter , and split them into 10 buckets of equal size.
We depict the average accuracy of each bucket in Figure 5.
As expected, our BNN is more accurate for instances with smaller uncertainty.
Quantitatively, there are 95% instances with uncertainty less than 0.45, and their accuracy is 78.6%; there are 90% instances with uncertainty less than 0.37, and their accuracy is 80.7%; there are 80% instances with uncertainty less than 0.25, and their accuracy is 84.8%.
R ELATED W ORK
Fruitful works have emerged in the BNN community in the last decade (Graves, 2011;
Welling &
Teh, 2011; Blundell et al., 2015; Kingma & Welling, 2013; Balan et al., 2015; Liu & Wang, 2016;
Kendall & Gal, 2017).
However, most of the existing works cannot achieve the goal of practicability.
For example, Liu & Wang (2016); Louizos & Welling (2016; 2017); Shi et al.
(2018a); Sun et al.
(2019) trade learning efﬁciency for ﬂexible variational posteriors, leading to restrictive scalability.
Khan et al.
; Zhang et al.
; Osawa et al.
build Adam-like optimizers to do variational inference, but their parallel training throughput and compatibility with data augmentation are inferior to SGD.
Empirical Bayes methods, e.g., Monte Carlo (MC) dropout (Gal & Ghahramani, 2016), deep ensemble (Lakshminarayanan et al., 2017), and SWAG (Maddox et al., 2019), usually maintain impressive predictive performance, but suffer from degenerated uncertainty estimates (Fort et al., 2019) or expensive training/storage cost.
What’s worse, the existing works usually evaluate on impractical OOD data (Louizos & Welling, 2017; Pawlowski et al., 2017) to show the promise of Bayesian principle.
Instead, we offer a new evaluation standard in this work, which may beneﬁt the following works.
Laplacian approximation (Bleistein & Handelsman, 1986; Ritter et al., 2018) is a known approach to transform a DNN to a BNN, but it is inﬂexible due to its postprocessing nature and some strong assumptions made for practical concerns.
Alternatively, BayesAdapter works in the style of ﬁnetuning, which is more natural and economical for deep networks.
Bayesian modeling the last layer of a DNN is proposed recently (Kristiadi et al., 2020), and its combination with BayesAdapter deserves an investigation.
BayesAdapter connects to MOPED (Krishnan et al.)
in that their variational conﬁgurations are both based on MAP .
Yet, beyond MOPED, BayesAdapter is further designed to achieve good user-friendliness, improved learning stability, and trustable uncertainty estimation, by virtue of optimizers with built-in weight decay, exemplar reparameterization, and uncertainty regularization, respectively, which signiﬁcantly boost the practicability of BayesAdapter , especially in real-world and large-scale settings.
C ONCLUSION
In this work we propose a scalable BayesAdapter framework to learn practical BNNs.
Our core idea is to learn a BNN by ﬁrst pre-training its DNN counterpart and then performing Bayesian ﬁnetuning .
In BayesAdapter, we develop a plug-and-play instantiation of stochastic VI and propose exemplar reparameterization to reduce the gradient variance.
We also propose a generic uncertainty regularization to calibrate the uncertainty quantiﬁcation given low-cost supervisions.
We evaluate
BayesAdapter in diverse realistic scenarios and report promising results.
A T HE E XEMPLAR V ERSION OF P OPULAR O PERATORS
As introduced in Sec 2.2, the regular convolution can be elegantly converted into an exemplar version by resorting to group convolution.
The other popular operators are relatively easy to handle.
For example, we substitute the qualiﬁed batch matrix multiplication, which is highly optimized in the well-known autodiff libraries, for matrix multiplication.
For the afﬁne transformation in batch normalization (Ioffe & Szegedy, 2015), we can at ﬁrst sample dedicated afﬁne weight and bias for every exemplar in the batch, then perform transformation with these two batches of parameters by just not broadcasting on the batch dimension.
B M ORE E XPERIMENTAL D ETAILS
The only two important hyper-parameters are the weight decay coefﬁcient λ and the uncertainty threshold γ .
Other hyper-parameters for deﬁning PGD or specifying learning rates, etc., all follow standard practice in the DL community.
The number of fake data training (1000) and the number of
MC samples for evaluation ( S ) are ﬂexible and not tuned.
For λ , we keep it consistent between pre-training and ﬁne-tuning (stated in Algorithm 1), without elaborated tuning, for example, λ = 2 e − 4 for the wide-ResNet-28-10 architecture on CIFAR-10, λ = 1 e − 4 for ResNet-50 architecture on ImageNet, and λ = 5 e − 4 for MobileNet-V2 architecture on CASIA.
These values correspond to isotropic Gaussian priors with σ 20 as 0.1, 0.0078, and 0.
on CIFAR-10, ImageNet, and CASIA, respectively.
It is notable that for a “small” dataset like
CIFAR-10, a ﬂatter prior is preferred.
While on larger datasets with stronger data evidence, we need a sharper prior for regularization.
For γ , we use γ = 0 .
75 for training across all the scenarios.
But it is not used for OOD detection in the testing phase.
For estimating the results of OOD detection, we use the non-parametric metric average precision (see the metric part of Section 4), which is the Area Under the Precision-Recall
Curve and is more suitable than the ROC-AUC metric when there is class imbalance.
For the pre-training, we follow standard protocols available online.
On CIFAR-10, we perform
CutOut (DeVries & Taylor, 2017) transformation upon popular resize/crop/ﬂip transformation for data augmentation.
On ImageNet, we leverage the ResNet-50 checkpoint on PyTorch Hub as the converged deterministic model.
On face tasks, we train MobileNetV2 following popular hyperparameter settings, and the pre-training takes 90 epochs.
We use the same weight decay coefﬁcients in both the pre-training and the ﬁne-tuning.
For the ﬁne-tuning, we set lr ψ to decay at 1/4, 1/2, and 3/4 of the total ﬁne-tuning steps from 0.1, and set lr µ to be the ﬁnal value of lr ψ on the CIFAR-10, ImageNet, and face recognition benchmarks.
We add a coefﬁcient 3 before the L unc term in Line 8 of Algorithm 1 for Bayesian ﬁne-tuning on
ImageNet to achieve better uncertainty calibration.
For models on face recognition, we utilize the features before the last FC layer of the MobileNetV2 architecture to conduct feature distance-based face classiﬁcation in the validation phase, due to the open-set nature of the validation data.
The
Bayes ensemble is similarly achieved by assembling features from multiple runs as the ﬁnal feature for estimating predictive performance.
But we still adopt the output from the last FC layer for uncertainty estimation (i.e., calculating Eq.
The training perturbation budget is identical to the evaluation budget on CIFAR-10 and ImageNet.
But we set the budget of the uniform noise used for training in face tasks to be 1/4 of the evaluation budget to make the models more sensitive to the perturbed data.
We adopt PGD for generating adversarial samples in the validation phase.
Concretely, we attack the posterior predictive objective, i.e., Eq.
(3), with S = 20 MC samples.
On CIFAR-10, we set δ m = 0 .
031 and perform PGD forsteps with step size at 0.003.
On ImageNet and face recognition, we set δ m = 16 / 255 and perform
PGD for 20 steps with step size at 1 / 255 .
Regarding the fake data, we craft 1000 fake samples for training and 10000 ones for evaluation with SNGAN (Miyato et al., 2018) on CIFAR-10; we craft 1000 fake samples for training and 1000 ones for evaluation with BigGAN (Brock et al., 2018) on ImageNet; we randomly sample 1000 fake samples for training and 10000 ones for evaluation from DeepFakes (Deepfakes, 2018),
FaceSwap (Faceswap, 2018) and Face2Face (Thies et al., 2016) on face recognition.
[Table]
As for the MC dropout
[Table]
we add dropout-0.3 (0.3 denotes the dropout rate) before the second convolution in the residual blocks in wide-ResNet-28-10, dropout-0.2 after the second and the third convolutions in the bottleneck blocks in ResNet-50, and dropout-0.2 before the last fully connected (FC) layer in MobileNetV2.
For reproducing Deep Ensemble , we train 5 MAP s separately, and assemble them for prediction and uncertainty quantiﬁcation.
For reproducing SWAG , we take use of its ofﬁcial implementation, and leverage 20 MC samples for decision making.
C M ORE R ESULTS FOR U NCERTAINTY E STIMATION
We provide the comparison on the quality of uncertainty estimates on face recognition in Table 6.
It is an immediate observation that BayeAdapter outperforms the extensive baselines signiﬁcantly, and can detect almost all the OOD instances across the validation datasets.
By contrast, BayeAdapter- ,
MAP , and BNN are similarly unsatisfactory.
Surprisingly, MC dropout exhibits some capacity to detect adversarial instances and DeepFake ones in the face tasks.
Comparing these results with those of MC dropout on CIFAR-10 and ImageNet, we speculate that such results may stem from the location of deploying dropout in the architecture, which deserves a future investigation.
D V ISUALIZATION OF S OME OOD D ATA
We provide some random samples of the OOD data used for evaluation in Figure 6.
Obviously, these samples are pretty realistic and challenging.
E V ISUALIZATION OF THE L EARNED P OSTERIOR
We plot the parameter posterior of the ﬁrst convolutional kernel in ResNet-50 architecture learned by BayesAdapter on ImageNet.
The results are depicted in Figure 7.
The learned posterior variance seems to be disordered, unlike the mean.
We leave more explanations as future work.