On Uncertainty, Tempering, and Data Augmentation in Bayesian Classiﬁcation
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Aleatoric uncertainty captures the inherent randomness of the data, such as measurement noise.
In Bayesian regression, we often use a Gaussian observationmodel, where we control the level of aleatoric uncertainty with a noise variance parameter.
By contrast, for Bayesian classiﬁcation we use a categorical distributionwith no mechanism to represent our beliefs about aleatoric uncertainty.
Our workshows that explicitly accounting for aleatoric uncertainty signiﬁcantly improvesthe performance of Bayesian neural networks.
We note that many standard benchmarks, such as CIFAR- 10 , have essentially no aleatoric uncertainty.
Moreover,we show that data augmentation in approximate inference softens the likelihood,leading to underconﬁdence and misrepresenting our beliefs about aleatoric uncertainty.
Accordingly, we ﬁnd that a cold posterior, tempered by a power greaterthan one, often more honestly reﬂects our beliefs about aleatoric uncertainty thanno tempering — providing an explicit link between data augmentation and coldposteriors.
We show that we can match or exceed the performance of posteriortempering by using a Dirichlet observation model, where we explicitly control thelevel of aleatoric uncertainty, without any need for tempering.
Introduction
Uncertainty is often compartmentalized into epistemic and aleatoric uncertainty [26, 31, 39].
Epistemic uncertainty, sometimes called model uncertainty, is the reducible uncertainty about whichsolution is correct given limited information.
Bayesian methods naturally represent epistemic uncertainty through a distribution over model parameters, leading to a posterior distribution over functionsthat are consistent with data.
As we observe more data, this posterior concentrates around a singlesolution.
Aleatoric uncertainty is intrinsic irreducible uncertainty, often representing measurementnoise in regression or mislabeled training points in classiﬁcation [e.g. 3].
Although measurementnoise can be reduced, for example, with better instrumentation, it is often a ﬁxed property of thedata we are given.
Correctly expressing our assumptions about aleatoric uncertainty is crucial forachieving good predictive performance, both with Bayesian and non-Bayesian models [54].
In particular, our assumptions about aleatoric uncertainty profoundly affect predictive accuracy , notonly predictive uncertainty 1 .
In Figure 1(a,b) we show the predictive distributions of two Gaussianprocess (GP) regression models [52] trained on the same data and using the same RBF kernel function.
The only difference between these models is their assumptions about the aleatoric uncertainty.
The model in Figure 1(a) assumes a high observation noise (each point is corrupted by N (0 ,  2 ) ,with  2 = 1 ).
Consequently, this model explains many of the observations with noise — such thatthe predictive mean does not closely ﬁt the training datapoints. The model in Figure 1(b) assumes a
[Figure]
low observation noise (  2 = 10  2 ), and consequently the predictive mean runs through the trainingdata, leading to very different predictions compared to the model in Figure 1(a).
Our assumptions about the aleatoric uncertainty are just as important in classiﬁcation.
We illustratethis point in Figure 1(c), where we ﬁx a dataset and consider two possible scenarios.
Scenario A :suppose we know that the data contains label noise, with some incorrectly labeled training examples.
In this case, it is reasonable to believe that the data points shown in red in Figure 1(c) are highly likelyto be incorrectly labeled, and we should opt for a solution with a decision boundary shown with adotted red line.
Scenario B : now, suppose that we know that the data are correctly labeled.
In thiscase, we want our model to perfectly classify all the training data and would prefer a solution shownby the dashed blue line.
Both of these solutions are reasonable descriptions of the data, and we canonly make an informed choice between them by incorporating our assumptions about the aleatoricuncertainty.
While in regression problems we can easily express our assumptions about aleatoricuncertainty, such as through a noise variance parameter in a Gaussian likelihood, in classiﬁcationwe do not have a direct way to express these beliefs.
Indeed, in practice we might use the same
Bayesian neural network (BNN) with the same likelihood and prior in both scenarios A and Babove, necessarily leading to poor performance in at least one of these scenarios.
In this paper, we explicitly characterize aleatoric uncertainty in BNN classiﬁcation.
We show thatcold posteriors [59] more honestly reﬂect our beliefs about aleatoric uncertainty than standard
Bayesian posteriors.
As an alternative to cold posteriors, we describe a noisy Dirichlet model , whichcan be viewed as an input dependent prior, to provide direct control over aleatoric uncertainty for
BNN classiﬁers.
This approach achieves competitive performance on image classiﬁcation problemswith a valid likelihood and no need for cold posteriors.
Finally, we show theoretically how dataaugmentation counterintuitively softens the likelihood, leading to solutions that underﬁt the trainingdata.
These results precisely characterize the empirical link between data augmentation and the success of cold posteriors observed by Izmailov et al.
[30] and Fortuin et al.
[13], and show that coldposteriors in fact serve to correct for the effects of data augmentation on aleatoric uncertainty.
Related Work
Early work on Bayesian neural networks focused on hyperparameter learning and mitigating overﬁtting, using Laplace approximations, variational methods, and MCMC [e.g. 35, 24, 45].
Morerecently, Wilson and Izmailov [60] show how Bayesian model averaging is especially compellingfor modern deep networks.
A number of recent works have focused on making Bayesian deep learning practical
[e.g., 61, 36, 47, 63, 11, 10], with improved performance, and essentially no overhead.
Cold posterior effect.
Wenzel et al.
[59] demonstrated with several examples that raising theposterior to a power 1 /T , with T < 1 , in conjunction with SGLD inference, improves performanceover classical training, but T = 1 can provide worse results than classical training, in what has been termed the cold posterior effect .
However, in a study using HMC inference, Izmailov et al.
[30]showed that for all cases considered in Wenzel et al.
[59] there is no cold posterior effect when dataaugmentation is removed.
The sufﬁciency of data augmentation to observe the cold posterior effecthas been conﬁrmed by several works using SG-MCMC inference [13, 46, 43].
Several works havesuggested that misspeciﬁed priors explain cold posteriors, arguing that the current default choice ofisotropic Gaussian priors in BNNs are inadequate [59, 62, 13].
However, Fortuin et al.
[13] ﬁndthat the cold posterior effect cannot be alleviated for convolutional neural networks by using heavytailed or correlated priors under data augmentation.
With HMC inference and no data augmentation,
Izmailov et al.
[30] also show that standard Gaussian priors perform similarly to a range of other priors, such as heavy-tailed logistic or mixtures of Gaussian priors, and generally are high performing.
Noci et al.
[46] argue that many different factors — likelihoods, priors, data augmentation — cancause cold posteriors, and that there may be no one cause in general.
Aitchison [2] argue that BNNlikelihoods are misspeciﬁed since many benchmark datasets have been carefully curated by humanlabelers.
They also show that posterior tempering is less helpful in the presence of label noise.
Adlam et al.
[1] also consider model misspeciﬁcation, showing a cold posterior effect can arise evenwith exact inference in GP regression when the aleatoric uncertainty is mis-estimated.
Nabarro et al.[43] modify the likelihood to accommodate data augmentation, using as the observation model theaverage of the neural network outputs over augmentations, but still ﬁnd a cold posterior effect.
Modeling label noise.
Works also focus on modifying the loss function in the context of labelnoise for classiﬁcation.
For example, Collier et al.
[8, 9] propose input-dependent noise lossesfor label noise in classiﬁcation to improve accuracy, but do not focus on the uncertainty estimates.
Similarly, Zhang and Sabuncu [64] and Feng et al.
[12] propose modiﬁcations of the standard crossentropy loss to provide increased robustness to label noise, but do not focus on modeling the inherentuncertainty in label noise settings. Dirichlet-based uncertainty (DBU).
Recent literature proposes neural networks where the output is a Dirichlet distribution, capturing both the epistemic and aleatoric uncertainty [39, 55, 38, 40,6, 65, 44, 57, 56, 32].
These models are generally proposed as efﬁcient alternatives to ensemblesand Bayesian models, and targeted for improved uncertainty quantiﬁcation, anomaly detection andadversarial robustness.
In this paper, we study the Dirichlet distributions associated with the posterior over the predictive class probabilities, which arise naturally in standard Bayesian classiﬁcationmodels.
Contrary to the methods in the DBU family, we do not modify the observation model or thestructure of the network.
Our paper makes several distinctive contributions in the context of prior work: (1) we argue thatstandard likelihoods do not represent our beliefs about aleatoric uncertainty, and that standard benchmarks have essentially no aleatoric uncertainty; (2) we show how tempering and data augmentationspecify aleatoric uncertainty, beyond curation; (3) we show the precise way in which data augmentation with SGLD leads to underconﬁdence in the likelihood , a counterintuitive result which ﬁnallyresolves the empirical connection between data augmentation and cold posteriors; (4) we show that
a
T < 1 , particularly with data augmentation, is a more honest reﬂection of our beliefs about aleatoricuncertainty than T = 1 ; (5) we show how the noisy Dirichlet likelihood, originally used for tractable
GP classiﬁcation [41], can naturally reﬂect our beliefs about aleatoric uncertainty, and for the ﬁrsttime remove the cold posterior effect in the presence of data augmentation.
Background
Bayesian Model Averaging.
With Bayesian inference, we aim to infer the posterior distribution over parameters having observed a dataset D
= { ( x i , y i ) }
Ni =1 of input-output pairs, given 114 by p ( w | D ) / p
( D | w ) p ( w ) for a given observation likelihood under the i.i.d.
assumptionp ( D | w ) =
Q
Ni =1 p ( y i | x i , w ) , and prior over parameters p ( w ) .
For any novel input x ? , we estimate the posterior predictive distribution via Bayesian model averaging (BMA) given byp ( y ?
| x ? )
=
R p ( y ?
| x ? , w ) p ( w | D ) dw .
This integral cannot generally be expressed in closedform for a Bayesian neural network, and is typically approximated with variational inference (VI),the Laplace approximation, or Markov Chain Monte Carlo (MCMC)
[e.g., 60, 42].
Cold Posteriors.
Let U ( w ) =
log p ( D | w )  log p ( w ) denote the posterior energy function ofa BNN.
Following Wenzel et al.
[59], we then deﬁne a cold posterior for T < 1 as p cold ( w | D )
/exp ⇥
1 T U ( w ) ⇤ , which effectively raises both the likelihood and the prior to a power 1 /T .
T = 1 recovers the standard Bayes’ posterior.
A tempered likelihood posterior (e.g. Gr¨unwald [16]) onlyraises the likelihood to a power 1 /T as
[Equation]
Bayesian Classiﬁcation.
For a C -class classiﬁcation problem, a standard choice of observation likelihood is the categorical distribution p ( y | x, w ) =
Cat([ ⇡ 1 , ⇡ 2 , . . .
, ⇡ C ]) , where each classprobability is computed using a softmax transform of logits f ( x ; w ) 2 R C as ⇡ c / exp { [ f ( x ; w )] c } ,and hence called the softmax likelihood .
The prior over parameters p ( w ) is often chosen to be anisotropic Gaussian N  0 ,
I  with some ﬁxed variance  2 .
Approximate posterior samples areoften obtained with SGLD, and then used to approximate the BMA using simple Monte Carlo.
Aleatoric Uncertainty in Bayesian Classiﬁcation
We will now discuss how we represent aleatoric uncertainty in classiﬁcation (Section 4.1), howtempering reduces aleatoric uncertainty (Section 4.2), and how we can explicitly represent aleatoricuncertainty without tempering in a noisy Dirichlet observation model (Section 4.3).
4.1 How do we represent aleatoric uncertainty in classiﬁcation?
Let us consider the Bayesian neural network posterior over parameters w in a classiﬁcation problem:p ( w | D ) / p ( w )
Q x,y 2 D f y
( x, w ) , where we denote the output of the softmax layer of the modelcorresponding to class y on input x as f y ( x, w ) .
We can think of the class probability vectorsf ( x ) =
f ( x, w ) = ( f 1 ( x, w ) , . . .
, f C ( x, w )) as latent variables, where the prior distribution p ( w )over the parameters of the network implies a joint prior distribution over { f ( x ) } x 2 D .
We will 141 initially focus on the observation model.
First, we note that we can introduce a uniform prior overthe predicted class probabilities f ( x ) : [Equation]
x ) , 2where Dir. denotes the Dirichlet distribution and Dir.
, 1) is a uniform distribution over theclass probabilities f ( x ) (with support on the unit C -simplex), and the proportionality is with respectto the latent variables f ( x ) .
We can view the right hand side of this expression as the unnormalizedposterior in a multinomial model, where we use a uniform prior over the the parameters f ( x ) , andobserve a single count of class y .
Using the fact that the Dirichlet distribution is conjugate to themultinomial likelihood [e.g., 5, Ch. 2.2.1], we can rewrite the observation model as follows:
[Equation]
Eq.
(2) describes the distribution induced by the observation y on the predicted class probabilitiesfor the corresponding input x .
The posterior over the parameters w of the network can then bewritten as a product of Eq.
(2) and the prior p ( w ) :
[Equation]
Eq.
(3) provides intuition for how BNNs estimate aleatoric uncertainty in classiﬁcation.
If we ignorethe prior p ( w ) and assume that the implied prior over f ( x ) is uniform, then for an observation ( x, y )the posterior over f ( x ) is Dir.
. , 1) and the posterior mean for the probability of thecorrect class E f y ( x ) is 2 C +1 .
For example, for a dataset with 100 classes (e.g. CIFAR-100), themodel on average will only be 2 / 101 ⇡ 2% conﬁdent in the correct label on the training data !
In practice, the prior p ( w ) will imply a non-trivial joint distribution over { f ( x ) } x 2 D , so the actualposterior may be more (or less) conﬁdent than suggested by the analysis above.
Furthermore, wehave very limited understanding of the implied distribution [see 59, 60, for empirical analysis].
Inparticular, most practitioners use simple p ( w ) =
N (0 , ↵ 2 I ) priors, regardless of the amount of labelnoise in the data.
Explicitly constructing a prior in the parameter space that would lead to highlyconﬁdent f ( x ) is challenging, but we will show how modifying the likelihood can be viewed asproviding an input-dependent prior that more fully reﬂects our beliefs about aleatoric uncertainty.
The expression in Eq.
(3) suggests two natural ways of modifying the posterior to account for thealeatoric uncertainty:
increasing the Dirichlet concentration for the observed class (equivalent toposterior tempering) and decreasing the concentration for the unobserved classes (noisy Dirichletmodel).
Next, we describe both of these approaches in detail.
4.2 Likelihood tempering reduces aleatoric uncertainty
The tempered likelihood posterior (see Section 3) in BNN classiﬁcation can be written asp temp ( w
[Equation]
D
[Equation]
/ p
[Equation]
[Equation]
x,y 2 D f y
( x, w ) 1 /T , where T is the temperature.
Analogously to the derivation above, we can rewrite this posterior as
[Equation]
y [Equation] In other words, the tempered posterior corresponds to the same model as the regular posterior, butassumes that we observed 1 /T counts of class y for each input x in the training data.
Assuming theprior p ( w ) implies a uniform distribution over f ( x ) , the conﬁdence in the correct labels on the traindata under the tempered posterior E f y ( x ) = 1+1 /T
C +1 /T .
With 100 classes and at T = 10  2 we get anaverage conﬁdence of 50 .
5% , much higher than the 2% for the standard observation model.
Tempering the Likelihood vs Posterior.
Prior work has mostly considered tempering the full
Bayesian posterior as opposed to just the likelihood (see Section 3).
In Appendix F.2 we show thattempering the full Bayesian posterior is almost always equivalent to changing the prior distribution,and tempering the likelihood, and that likelihood tempering recovers the cold posterior effect.
How should we think about tempering?
Prior work has asserted that posterior tempering sharplydeviates from the Bayesian paradigm and the cold posterior effect is highly problematic [59, 46, 13].
We, on the other hand, argue that likelihood tempering is in fact a practical way to incorporate ourassumptions about aleatoric uncertainty.
Relatedly, Aitchison [2] argue that the cold posterior effectcan be caused by data curation.
For example, CIFAR- 10 and ImageNet datasets have relatively verylittle label noise and are carefully curated [e.g. 51].
Thus, we may expect that tempered likelihoodswith low temperatures will lead to optimal performance.
Is a Tempered Likelihood a Valid Likelihood?
In classical Bayesian inference, the observationmodel is a distribution p ( y | x ) over the labels conditioned on the input.
Wenzel et al.
[59] arguethat the tempered softmax likelihood is in general not a valid likelihood because it does not sum to1 over classes.
However, Wenzel et al.
[59] show that the tempered softmax likelihood with T < 1can be interpreted as a valid likelihood if we introduce a new class, which is not observed in thetraining data.
Moreover, from Eq.
(4), we can naturally interpret the tempered likelihood as usingthe multinomial observation model, assuming 1 /T counts of the label are observed for each of thetraining datapoints, which is uncontroversial and perfectly valid.
4.3 Noisy Dirichlet model: changing the prior over class probabilities
As we have seen in Eq.
(4), likelihood tempering increases the posterior conﬁdence by increasingthe Dirichlet concentration for the observed class y from 2 to 1 + 1 /T .
We can achieve a similareffect by decreasing the concentration of the unobserved classes 3 .
Indeed, consider the distribution
[Equation]
where ND in p ND stands for Noisy Dirichlet and ↵ ✏ is a tunable parameter.
The noisy Dirichletmodel was originally proposed by Milios et al.
[41] in the context of GP classiﬁcation, where theydesigned a tractable approximation to this model.
Using our running example, if p ( w ) induces auniform distribution over f ( x ) , for a problem with 100 classes and ↵ ✏ = 10  2 we have expectedconﬁdence E f y ( x ) = ↵ ✏ +1 C ↵ ✏ +1 = 50 . 5% , same as for the tempered likelihood with T = 10  2 .
Using the conjugacy of the Dirichlet and multinomial distributions, we can rewrite p ND as follows:
[Equation]
We can interpret q ND ( w ) = p ( w ) ·
Q x 2 D Dir.
( ↵ ✏ , . . .
, ↵ ✏ )( f ( x, w )) as a prior over the parameters wof the model.
Indeed, q ND ( w ) does not depend on the labels y , and simply forces the predicted classprobabilities to be conﬁdent in any one of the classes for the training data.
The Noisy Dirichlet prioris intuitively appealing: in many practical settings a priori we believe that the aleatoric uncertaintyon the training data is low, and the model should be conﬁdent in one of the classes.
At the sametime, we would not want to enforce low aleatoric uncertainty everywhere in the input space, as wedo not expect that all the possible inputs to the model should be classiﬁed with high conﬁdence.
In short, the Noisy Dirichlet model corresponds to using a valid likelihood — the softmax likelihood— with a prior that explicitly enforces the model predictions to be conﬁdent on the training data.
In
Section 6.2 we will see that the noisy Dirichlet model removes the cold posterior effect.
Gaussian approximation.
Milios et al.
[41] considered the distribution in Eq.
(6) in the context of
Gaussian process classiﬁcation, but with a different goal: they aimed to create a regression likelihoodwhich would approximate the softmax likelihood.
We describe their Gaussian approximation whichamounts to solving a regression problem in the space of logits z ( x ) with a Gaussian observationmodel in Appendix C. While theoretically we do not need to use the Gaussian approximation for
Bayesian neural networks and can directly use Eq.
(6), we found that in practice the approximationis much more stable numerically.
We use the Gaussian approximation in all experiments.
The Effect of Data Augmentation
Data augmentation is a key ingredient of modern deep learning.
Surprisingly, however, data augmentation has not been studied in the context of Bayesian methods until recently.
Several works haveshown that data augmentation often causes the cold posterior effect in BNNs [30, 62, 13, 43].
Weshow that data augmentation, as applied in practice 4 , is closely connected to likelihood temperingwith T > 1 .
In this case, the cold posteriors counterbalance the effect of data augmentation.
First, we consider the interaction between data augmentation and SGLD (or, equivalently, any other
SGMCMC method).
With SGLD, we aim to construct an unbiased estimator of the gradient ofthe posterior log-density (see Appendix B).
In prior work, e.g. in Wenzel et al.
[59], the stochasticgradient is estimated using a randomly augmented mini-batch of the data.
For a minibatch of the fulldataset D m = { x i , y i } mi =1 ⇢ D , and a ﬁnite set of augmentations T = { t 1 , . . .
, t K } , this stochasticgradient is given by r
[Equation]
p
[Equation]
x i )) + r w log p ( w ) , where thetransformations t j are sampled uniformly from T .
There are two sources of randomness in r e
U ( w )— the choice of the mini-batch D m and the choice of augmentation t j used for each x i .
The limitingdistribution of SGLD is determined by the expectation of the stochastic gradient above, given by
N
[Equation]
where t j ⇠ T are augmentations sampled uniformly from T .
Therefore, we conclude that thelimiting distribution of SGLD under data augmentation is given by
[Equation]
We can interpret the limiting distribution in Eq.
(8) as a tempered likelihood posterior for a newdataset D
= { ( t j ( x i ) , y i ) } i,j which contains all augmentations of every data point from the original 242 dataset D .
Furthermore, the likelihood is tempered with a temperature K > 1 , corresponding to a warm posterior .
In other words, the limiting distribution of SGLD with data augmentationcorresponds to a posterior over the augmented dataset with a softened likelihood, leading to lessconﬁdence about the labels.
By applying a cold temperature T = 1 /K to the posterior in Eq.
(8),we can recover the standard Bayesian posterior on the augmented dataset D 0 .
In Section 6.2, weexplore the effect of data augmentation on aleatoric uncertainty in practice and show that it doesindeed soften the likelihood and lead to increased aleatoric uncertainty.
What about Other Inference Procedures?
While the derivation in Eq.
(8) comes directly fromstudying the stochastic gradient evaluated in SGMCMC, we can equivalently derive the same posterior distribution in stochastic variational inference [25] as we show in Appendix H.
Should we always use temperature T = 1 /K ?
While using the temperature T = 1 /K recoversthe standard Bayesian posterior on the augmented dataset D 0 , it is not necessarily a correct approachto modeling data augmentation.
Indeed, the standard posterior p ( w | D 0 ) / p ( w )
Q
Ni =
Q
Kj =1 p ( y i |t
j ( x i )) assumes independence across both augmentations and data points.
In practice, however, weshare the same label
y i across all the augmented versions of the image; treating the observations asindependent then leads to underestimating the aleatoric uncertainty.
Indeed, consider the extremescenario, where all the augmentations t j ( x ) simply return x .
In this case, treating the observations( t j ( x ) , y ) as independent will simply raise the likelihood to a power of K , while in reality we havenot received any additional information from the augmentation policy.
Correcting the likelihood by afactor of 1 /K is the correct approach only when the predictions on t j ( x ) are completely independentfrom each other (see Section 6.1).
See also Nabarro et al.
[43] for a related discussion.
Experiments
In this section we provide empirical support for the observations presented in Sections 4 and 5.
In Appendix D we additionally illustrate the effects of tempering, noisy Dirichlet model and dataaugmentation using a Bayesian neural network on a synthetic 2-D classiﬁcation problem.
All resultsreport the mean and one standard deviation over 3 trials.
6.1 Visualizing the Effect of Data Augmentation
To illustrate our analysis in Section 5 we show the posterior in Eq.
(8) under data augmentation ina Gaussian process (GP) regression model.
We use a GP with a RBF kernel, noting that data pointswhere k ( x, x 0 ) ⇡ 0 are effectively independent from each other, which occurs when k x  x 0 k   .
Consequently, the implied correlations between inputs beyond the distance  are zero.
We useaugmented samples that are given by t j ( x ) = x + ⌧ j , for vectors { ⌧ j } Kj =1 such that k ⌧ j k   ,assuming
that k ( x, t j ( x 0 )) =
for all datapoints x, x 0 and all augmentations t j .
Thus, the outputsof the GP on the training data are independent from the outputs on the augmented datapoints.
We show the posterior near the original data in Eq.
(8) for K = 1 , 4 and 10 augmentations perdatapoint in Figure 2.
As the predictions on the train data are independent from the predictionson the augmented datapoints, the posterior in Eq.
(8) corresponds to tempering the posterior on theoriginal training data with a warm temperature equal to the number K of augmentations.
As a result,increasing K softens the likelihood, leading to a less conﬁdent ﬁt on the training data.
6.2 Image Classiﬁcation with Bayesian Neural Networks
In this section we experimentally verify the results of the analysis in Sections 4 and 5 for Bayesianneural networks in image classiﬁcation problems.
First, we show that both the noisy Dirichlet modeland tempered softmax likelihood can be successfully used to express our beliefs about the amountof label noise in the data. Then, we show that the noisy Dirichlet model does not require temperingto achieve optimal performance.
Finally, we show that data augmentation softens the likelihood for
BNNs, and that the optimal temperature depends on the complexity of the data augmentation policy.
For all experiments we use a ResNet-18 model [19] and the CIFAR-10 [33] and Tiny Imagenet [34]datasets.
We use the SGLD sampler with a cyclical learning rate schedule [58, 63] to sample fromthe posterior.
We provide details on the hyper-parameters in Appendix K. [Figure] Accounting for the label noise via the noisy Dirichlet model or the tempered softmax likelihood signiﬁcantly improves accuracy accross the board. Moreover, the best performance is achieved by different values of temperature T ⇤ in the tempered softmax likelihood or noise ↵ ⇤ ✏ in the noisy
Dirichlet model, i.e. no one model describes the data optimally across all levels of label noise.
Modeling label noise.
In classiﬁcation problems the aleatoric uncertainty corresponds to theamount of label noise in the data.
Throughout the paper we argued that aleatoric uncertainty ismisrepresented by standard Bayesian neural networks, and that likelihood tempering and the noisy
Dirichlet model are compelling approaches to incorporate information about the amount of labelnoise.
In Figure 3, we show the BMA test accuracy and negative log-likelihood for the standardsoftmax likelihood, tempered softmax likelihood and the noisy Dirichlet model on CIFAR- 10 and
Tiny Imagenet under varying amounts of label noise.
We plot the results for the best performingtemperature T 2 [10  5 , 10] or noise ↵ ✏ 2 [10  6 , 10  1 ] for the noisy Dirichlet model.
On bothdatasets and across label noises, we can signiﬁcantly improve performance over the softmax likelihood by explicitly modeling the aleatoric uncertainty with either tempering or the noisy Dirichletmodel.
Moreover, different amounts of label noise require different values of T and ↵ ✏ parameters.
with ↵ ✏ =10  6 and the softmax likelihood as a function of temperature on CIFAR10.
The noisy Dirichlet model shows no cold posterior effect.
No cold posterior effect in the noisy Dirichlet model.
We explore the effect of posterior tempering on the standard softmaxclassiﬁcation likelihood and the noisy Dirichlet model with noiseparameter ↵ ✏ =10  6 .
In Figure 4 we show the results on the CIFAR-10 dataset.
As reported by Wenzel et al.
[59], for the standard softmax likelihood, tempering is required to achieve optimalperformance, with T =10  3 providing the best results.
For thenoisy Dirichlet model on the other hand, tempering does not signiﬁcantly improve the results with roughly constant performanceacross the temperature values.
In particular, the noisy Dirichletmodel achieves near-optimal results at T =1 .
These results agreewith our analysis in Section 4: both tempering and the noisy Dirichlet model are alternative ways of expressing our beliefs about thealeatoric uncertainty in the data; with the noisy Dirichlet model, wecan achieve strong results without any need for tempering.
Data augmentation leads to underﬁtting on train data.
In
Figure 5a we show the BMA train negative log-likelihood for themodels trained with and without data augmentation.
Across temperatures for the tempered softmaxand ↵ ✏ parameters for the noisy Dirichlet model, adding data augmentations reduces the quality of
[Figure]
the ﬁt on the original training data.
This result is analogous to the GP results in Section 6.1 and ouranalysis in Section 5: data augmentation softens the likelihood, leading to a more diffuse ﬁt.
Complex augmentations require lower temperatures.
In Section 5 we showed that under dataaugmentation the likelihood is effectively tempered with a temperature K , assuming the predictions on the augmented datapoints are completely independent from the original datapoints.
Inpractice however, the augmentations can be close to the original images, and the corresponding predictions can be highly correlated.
In Figure 5b, we consider ﬁve separate types of augmentationsfor ResNet- 18 on CIFAR- 10 at various posterior temperatures: horizontal and vertical ﬂips, randomcrops, combinations of ﬂips and crops, and AugMix [22] — an augmentation policy employing avery diverse set of transformations.
We ﬁnd that, as predicted by the analysis in Section 5, theoptimal temperatures are different for different policies: in terms of test NLL, the simple policies(vertical and horizontal ﬂips) corresponding to K = 2 work best at warmer temperatures T = 1 ,intermediate policies (crops and crops + ﬂips) corresponding to K ⇡ 100 require lower temperatures
T 2 [10  2 , 10  1 ] , and the most complex AugMix policy requires the lowest temperature T  10  4 .
Robustness to corruptions. Finally, we consider robustness to dataset corruptions by consideringthe CIFAR-
10 C dataset [21], evaluating our models with both tempered softmax (TS) and noisy
Dirichlet models.
A full plot is shown in Appendix J; summarizing those results, we see that TS hasthe best performance at T = 10  3 with an accuracy of 93 .
2 (with T = 1 again under-performing),while again the noisy Dirichlet model performs well, with an accuracy of 92 . 6 (at ↵ ✏ = 10  4 ).
Discussion
A correct representation of aleatoric uncertainty is crucial to achieve strong performance with
Bayesian neural networks.
Standard Bayesian classiﬁers have no explicit mechanism to representour beliefs about aleatoric uncertainty, which can lead to inadequate ﬁts to the training data.
Thiseffect is exaggerated by data augmentation, which softens the likelihood, making the models underconﬁdent on the training data.
Posterior tempering is a simple and practical way to correct for thismisspeciﬁcation and express our beliefs about the aleatoric uncertainty: most benchmark datasetshave very low levels of label noise, and we can express this belief by tempering the softmax likelihood.
We also showed that we can achieve a similar effect without tempering, by using a prior thatforces the model to be conﬁdent on the training datapoints with the noisy Dirichlet model.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] Appendix A.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] Appendix
A.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...(
a ) Did you state the full set of assumptions of all theoretical results?
[Yes] See Eq.
Appendices F, H(
b) Did you include complete proofs of all theoretical results
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes] Unless otherwise stated, mean and standard deviation
over 3 trials.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] Appendix K.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] CIFAR- 10 [33], TinyImagenet [34].
(b) Did you mention the license of the assets?
[Yes] CIFAR- 10 , MIT License, TinyIma-
[Table]
genet, MIT License [34]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data
you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁ-
able information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]