U NCERTAINTY - BASED OUT - OF - DISTRIBUTION DETECTION REQUIRES SUITABLE FUNCTION SPACE PRIORS
Anonymous authors
Paper under double-blind review
A BSTRACT
The need to avoid conﬁdent predictions on unfamiliar data has sparked interest in out-of-distribution (OOD) detection.
It is widely assumed that Bayesian neural networks (BNNs) are well suited for this task, as the endowed epistemic uncertainty should lead to disagreement in predictions on outliers.
In this paper, we question this assumption and show that proper Bayesian inference with function space priors induced by neural networks does not necessarily lead to good OOD detection.
To circumvent the use of approximate inference, we start by studying the inﬁnitewidth case, where Bayesian inference can be exact due to the correspondence with
Gaussian processes.
Strikingly, the kernels induced under common architectural choices lead to distributions over functions which cause predictive uncertainties that do not reﬂect the underlying data generating process and are therefore unsuited for OOD detection.
Importantly, we ﬁnd this OOD behavior to be consistent with the corresponding ﬁnite-width networks.
To overcome this limitation, useful function space properties can also be encoded in the prior in weight space, however, this can currently only be applied to a speciﬁed subset of the domain and thus does not inherently extend to OOD data.
Finally, we argue that a trade-off between generalization and OOD capabilities might render the application of BNNs for
OOD detection undesirable in practice.
Overall, our study discloses fundamental problems when naively using BNNs for OOD detection and opens interesting avenues for future research.
I NTRODUCTION
One of the challenges that the modern machine learning community is striving to tackle is the detection of unseen inputs for which predictions should not be trusted.
This problem is also known as out-of-distribution (OOD) detection.
The challenging nature of this task is partly rooted in the fact that there is no universal mathematical deﬁnition that characterizes an unseen input x ∗ as OOD as discussed in Sec. 2.
Without such a deﬁnition, there is no foundation for deriving detection guarantees under veriﬁable assumptions.
In this work, we consider OOD points as points that are unlikely under the distribution of the data p ( x ) and points are outside the support of p ( x ) .
Importantly, we do not claim to provide a meaningful deﬁnition of OOD, but rather argue that the justiﬁcation of an OOD method can only be assessed theoretically once such a deﬁnition is provided.
While it appears natural to tackle the OOD detection problem from a generative perspective by explicitly modelling p ( x ) , this paper is solely concerned with the question of how justiﬁed it is to deploy the predictive uncertainty of a Bayesian neural network (BNN) for OOD detection.
As recent developments forecast an increasing integration of deep learning methods into industrial applications, it becomes essential to provide the theoretical groundings that justify the use of uncertainty for OOD detection, a task that is crucial for safety-critical applications of AI and reliable prediction-making.
When BNNs are used in supervised learning, the dataset is composed of inputs x ∈ X and targets y ∈ Y which are assumed to be generated according to some unknown process: D i.i.d.
∼ p ( x ) p ( y | x ) .The
goal of learning is to infer from D alone the distribution p ( y | x ) in order to make predictions on unseen inputs x ∗ .
In the case of deep learning, this problem is approached by choosing a neural network f ( · ; w ) parametrized by w , and predictions are made via the conditional p (cid:0) y | f
( x ; w ) (cid:1) (also called the likelihood of w for given x , y ).
Assuming that the induced class of hypotheses contains some ˆ w such that p (cid:0) y | f
( x ; ˆ w ) (cid:1) =
p ( y | x ) almost everywhere on the support of p ( x ) ,
Bayesian statistics can be used to infer plausible models p ( w | D ) under the observed data given some prior knowledge p ( w ) (see MacKay (2003)).
This parametric description together with the network implicitly induces a prior over functions p ( f ) (Williams, 1997; Fortuin, 2021).
Believing in the validity of a subjective choice of prior (O’Hagan, 2004), Bayesian inference comes with a multitude of beneﬁts as it is less susceptible to overﬁtting, allows to incorporate new evidence without requiring access to past data (Farquhar & Gal, 2018) and provides interpretable uncertainties.
prior variance over function values
The uncertainty captured by a BNN can be coarsely categorized into aleatoric and epistemic uncertainty.
Aleatoric uncertainty is irreducible and intrinsic to the data p ( y | x ) .
For instance, a blurry image might not contain enough information to identify unique object classes.
On the other hand, a BNN also models epistemic uncertainty by maintaining a distribution over parameters p ( w | D ) .
This distribution reﬂects the uncertainty about which hypothesis explains the data and can be reduced by observing more data.
Arguably, aleatoric uncertainty is of little interest for detecting OOD inputs. However, the recent advent of overparametrized models which further extend the hypothesis class, deceptively motivated the idea that the epistemic uncertainty under a Bayesian framework might be intrinsically suitable to detect unfamiliar inputs, and therewith implying, that BNNs can be used for OOD detection.
This conjecture is intuitively true if the following assumption holds: the hypotheses captured by p ( w | D ) must agree on their predictions for in-distribution samples but disagree for OOD samples (Fig. 1a).
Certain Bayesian methods satisfy this assumption, e.g. a Gaussian process regression with an RBF kernel (cf.
Sec. 4).
However, the uncertainty induced by Bayesian inference does not in general give rise to OOD capabilities.
This can easily be veriﬁed by considering the following thought experiment (Fig.
1b):
Assume a model class with only two hypotheses g ( x ) and h ( x ) such that g ( x ) (cid:54)
=
h ( x ) ∀ x and data being generated according to g ( x ) on a restricted domain.
Once data is observed, we can commit to the ground-truth hypothesis g ( x ) and thus lose epistemic uncertainty
in-
and out-of-distribution alike.
Finite-width neural networks, by contrast, form a powerful class of models, and are often put into context with universal function approximators (Hornik, 1991).
But is this fact in combination with Bayesian statistics enough to attribute them with good OOD capabilities?
The literature often seems to imply that a BNN is intrinsically good at OOD detection.
For instance, the use of OOD benchmarks when introducing new methods for approximate inference creates the false impression that the true posterior is a good OOD detector (Louizos & Welling, 2017; Pawlowski et al., 2017; Krueger et al., 2017; Henning et al., 2018;
Maddox et al., 2019; Ciosek et al., 2020; D’Angelo & Fortuin, 2021).
Our work is meant to start a discussion among researchers about the properties of OOD uncertainty of the exact Bayesian posterior of neural networks.
To initiate this, we contribute as follows:
• We emphasize the importance of the prior in function space for OOD detection, which is induced by the choice of architecture (Sec. 4) and weight space prior (Sec. 5).
• We empirically show that exact inference in inﬁnite-width networks under common architectural choices does not necessarily lead to desirable OOD behavior, and that these observations are consistent with results obtained via approximate inference in their ﬁnite-width counterparts.
• We furthermore study the OOD behavior in inﬁnite-width networks by analysing the properties of the induced kernels.
Moreover, we discuss desirable kernel features for OOD detection and show that also neural networks can approximately carry these features.
• We emphasize that the choice of weight-space prior has a strong effect on OOD performance, and that encoding desirable function space properties within unknown OOD domains into such prior is challenging.
• We argue that there is a trade-off between good generalization and having high uncertainty on OOD data. Indeed, improving generalization by incorporating prior knowledge (which is usually encoded in an input-domain agnostic manner) can negatively impact OOD uncertainties.
O N THE DIFFICULTY OF DEFINING OUT - OF - DISTRIBUTION INPUTS
While the intuitive notion of OOD sample points (or outliers) is commonly agreed upon, for instance as "an observation (or subset of observations) which appears to be inconsistent with the remainder of that set of data" (Barnett & Lewis, 1984), a mathematical formalization of the essence of an outlier is difﬁcult (cf.
SM A)
and requires subjective characterizations (Zimek & Filzmoser, 2018).
Often, methods for outlier detection are designed based on an intuitive notion (see Pimentel et al.
(2014) for a review), and therewith only implicitly deﬁne a method-speciﬁc deﬁnition of OOD points.
This also accounts for uncertainty-based OOD detection with neural networks, where a statistic of the predictive distribution that quantiﬁes uncertainty (e.g., the entropy) is used to decide whether an input is considered OOD (Hendrycks & Gimpel, 2017).
Commonly, the parameters w of a neural network are trained using an objective derived from E p ( x ) (cid:2) KL (cid:0) p ( y | x ) || p (cid:0) y | f
( x ; w ) (cid:1)(cid:1)(cid:3) (i.e., loss functions based on the negative log-likelihood such as the cross-entropy or mean-squared error loss).
Therefore, these networks only have calibrated uncertainties in-distribution with OOD uncertainties not being controlled for unless explicit training on OOD data is performed (eg., Hendrycks et al., 2019).
The question this study is concerned with is therefore: does the explicit treatment of parameter uncertainty via the posterior parameter distribution p ( w | D ) elicit provably high uncertainty OOD?
Note, that we do not consider the problem of outlier detection within the training set (Zimek &
Filzmoser, 2018), but rather ask whether a deployed model is able to know what it does not know .
B ACKGROUND
In this section, we brieﬂy introduce the concepts on which we base our argumentation in the coming sections.
We start by introducing BNNs, which rely on approximate inference.
We then discuss that in the non-parametric limit and under a certain choice of prior, a BNN converges to a Gaussian process, an alternative Bayesian inference framework where exact inference is possible.
The connection between BNNs and Gaussian processes will later allow us to make interesting conjectures about OOD behavior.
Finally, we introduce generalization bounds from the PAC-Bayes framework, which we will later use to argue that generalization and OOD detection can be conﬂicting objectives.
3. B AYESIAN NEURAL NETWORKS
In supervised deep learning, we typically construct a likelihood function from the conditional density p ( y | f ( x ; w )) , parameterized by a neural network f ( x ; w ) , and the training data D = { ( x i , y i ) } ni =1 .
In BNNs, this is used to form the posterior distribution of all likely network parameterizations: p ( w | D ) ∝ (cid:81) ni
=1 p ( y i | f ( x i ; w )) p ( w ) , where p ( w ) is the prior distribution over weights.
Crucially, when making a prediction with the Bayesian approach on a test point x ∗ , we do not only use a single parameter conﬁguration ˆ w but we marginalize over the whole posterior, thus taking all possible explanations of the data into account: p ( y ∗ | x ∗ , D )
= (cid:82) p ( y ∗ | f ( x ∗ ; w ))
p ( w | D ) d w .
3. G AUSSIAN PROCESS
Gaussian processes are established Bayesian machine learning models that, despite their strong scalability limitations, can offer a powerful inference framework.
Formally (Rasmussen, 2004):
Deﬁnition
A Gaussian process (GP) is a collection of random variables, any ﬁnite number of which have a joint Gaussian distribution.
Compared to parametric models, GPs have the advantage of performing inference directly in function space.
A GP is deﬁned by its mean m ( x ) =
E [ f ( x )] and covariance C
( f ( x ) , f ( x (cid:48) )) =
E [( f ( x ) − m ( x ))
( f ( x (cid:48) ) − m ( x (cid:48) ))] .
The latter can be speciﬁed using a kernel function k : R d × R d → R and implies a prior distribution over functions:
[Equation]
where K ( X, X ) ij = k ( x i , x j ) is the kernel Gram matrix on the training inputs X , and m ( x ) has been chosen to be 0 .
When observing the training data, the prior is reshaped to place more mass in the regions of functions that are more likely to have generated them, and this knowledge is then used to make predictions on unseen inputs X (cid:63) .
In probabilistic terms, this operation corresponds to conditioning the joint Gaussian prior: p ( f ∗ | X ∗ , X, f ) .
To model the data more realistically, we assume to not have direct access to the function values but to noisy observations:
y = f ( x )+ (cid:15) with (cid:15) ∼ N (0 , σ 2 I d ) .
This assumption is formally equivalent to a Gaussian likelihood p ( y | f )
=
N ( y | f , σ 2 ) .
The conditional distribution on the noisy observation can then be written as (Rasmussen, 2004):
[Equation]
where p ( f | X, y )
=
p ( y | f ) p ( f | X )) p ( D ) with p ( D ) being the marginal likelihood.
RBF kernel.
A commonly used kernel function for GPs is the squared exponential (RBF):
[Equation]
with the length scale l being a hyperparameter.
It is important to notice that the covariance between outputs is written exclusively as a function of the distance between inputs.
As a consequence, points that are close in the Euclidean space have unitary covariance that decreases exponentially with the distance.
As we will see in Sec. 4, this feature has important implications for OOD detection.
The relation of inﬁnite-width BNNs and GPs.
The connection between neural networks and GPs has recently gained signiﬁcant attention.
Neal (1996) showed that a 1-hidden layer BNN converges to a GP in the inﬁnite-width limit.
More recently, the work by Lee et al.
(2018) and de G. Matthews et al. (2018) extended this result to deeper networks, called neural network Gaussian processes (NNGP).
Crucially, the kernel function of the related GP strictly depends on the used activation function.
To better understand this connection we consider a fully connected network with L layers l = 0 , . . .
, L with width H l .
For each input we use x l ( x ) to represent the post-activation with x 0 = x and f l the pre-activation so that f li ( x ) =
b l − 1 i +
[Equation]
1 ij x l [Equation] 1 j with x lj = h ( f lj ) and h ( · ) a point-wise activation function.
Furthermore, the weights and biases are distributed according to b lj ∼ N (0 , σ 2 b ) and w lij ∼ N (0 , σ 2 w H l ) .
Given the independence of the weights and biases it follows that the post-activations are independent as well. Hence, the central limit theorem can be applied, and for H l − 1 → ∞ we obtain f l ( x ) ∼ GP (0 , C l ) .
The covariance C l and therefore the prior over functions is speciﬁed by the kernel induced by the network architecture (Lee et al., 2018):
[Equation]
with K l being the 2 × 2 covariance matrix formed by k l ( · , · ) using x and x (cid:48) .
For the input layer, where no activation function is applied, the kernel is simply given by k 0 ( x , x (cid:48) ) = [Equation]
+
[Equation] (cid:16) [Equation]
(cid:17) .
For the remaining layers, instead, the kernel expression is determined by the non-linearity.
Some activation functions like sigmoid or hyperbolic tangent do not admit a known analytical form and therefore require a Monte Carlo estimate of Eq.
4 (cf.
Fig.
S3).
For others, Eq. 4 can be computed in closed form (e.g., Williams, 1997; Cho & Saul, 2009;
Tsuchida et al., 2018; Pang et al., 2019; Pearce et al., 2020).
We list some kernel functions that are important for this study below, such as the one for ReLU networks (Cho & Saul, 2009):
[Equation]
Interestingly, for some non-linearities a kernel that carries similar properties as the RBF in Eq.can be induced by neural networks.
This is the case for the cosine activation function which has the
[Equation]
Similarly, also the exponential activation function as deployed in RBF networks (Broomhead & Lowe, 1988) leads to similar properties.
This particular class of neural networks deﬁnes computational units as linear combinations of radial basis functions f (
[Equation]
These networks, in the inﬁnite-width case and when assuming a Gaussian prior over the centers µ j ∼ N (0 , σ 2 µ I ) and w j ∼ N 0 , σ 2 w ) , also converges to a Gaussian process with an analytical kernel function:
[Equation]
where 1 /σ 2 e = 2 /σ 2 g + 1 /σ 2 µ , σ 2 s = 2 σ 2 g + σ 4 g /σ 2 µ and σ 2 m = 2 σ 2 µ + σ 2 g .
The equivalence with the
RBF kernel is explicit in the limit σ 2 µ → ∞ (Williams, 1997).
3. PAC-B AYES GENERALIZATION BOUND
Considering the supervised learning framework introduced in Sec. 1, the PAC theory (Valiant, 1984) establishes a probabilistic bound on the generalization error of a given predictor.
The theory is referred to as PAC-Bayes (McAllester, 1999; Catoni, 2007) when a prior distribution p is deﬁned to incorporate prior domain knowledge.
Considering a distribution 3 q on the hypothesis h ∈ H and the space B ( Y ) of all conditional distributions p ( y | h ( x )) , we can deﬁne a bounded loss function such that l :
B ( Y ) ×Y → [0 , 1] . This gives rise to the empirical and true risk as R D ( q ) :
= 1 N (cid:80) Ni =
1 E h ∼ q
[ l ( p ( y | h ( x i )) , y i )] and R ( q ) :
= E ( x (cid:63) , y (cid:63) ) ∼ p ( x , y )
E h ∼ q
[ l ( p ( y | h
( x (cid:63) )) , y (cid:63) )] , respectively, where p ( x , y )
= p ( x ) p ( y | x ) .
Note that when q is the Bayesian posterior as, for instance, in the GP case, the Bayes risk can be deﬁned as: R B ( q ) :
= E ( x (cid:63) , y (cid:63) ) ∼ p ( x , y )
[ l ( E h ∼ q [ p ( y | h ( x (cid:63) ))] , y (cid:63) )] and it can be shown that R B ( q ) ≤ 2 R ( q ) (Seeger, 2003).
Therefore, a bound over R also implies a bound over R B (Seeger, 2003).
The PAC bound gives a probabilistic upper bound on the true risk R ( q ) in terms of the empirical risk R D ( q ) for a training set D as formulated in the following theorem:
Theorem 1 (PAC-Bayes theorem (Catoni, 2007))
For any distribution p ( x , y ) over X × Y , for any distribution q and prior p on a hypothesis space H , for any δ ∈ (0 , 1] and β > 0 the following holds with probability at least 1 − δ over the training set D ∼ p ( x , y ) of cardinality N :
[Equation]
For a ﬁxed p , D , δ , minimizing Eq. 8 is equivalent to minimizing NR D ( q ) + KL ( q || p ) .
The requirement of a bounded loss function makes the use of the mean-squared error, and thus the negative log-likelihood, not applicable to this bound.
Nevertheless, as long as the loss function measures the quality of predictions, the bound in Eq.
8 can be used to assess the generalization capabilities of a model.
For this purpose, in our analysis we use as a surrogate loss the following Reeb et al.
(2018): l exp ( p ( y | h ) , y )
= 1 − exp (cid:2) − ( E y [ p ( y | h )] − y ) 2 σ(cid:3) .
Interestingly, for small deviations we can take the ﬁrst-order Taylor expansion and recover the MSE l exp ( E y [ p ( y | h )] , y , )
≈ ( E y [ p ( y | h )] − y ) 2 /σ 2 .
T HE ARCHITECTURE STRONGLY INFLUENCES OOD UNCERTAINTIES
In the previous section, we recalled the connection between BNNs and GPs, namely that Bayesian inference in an inﬁnite-width neural network can be studied in the GP framework (assuming a proper choice of weight-space prior).
This connection allows studying how the kernels induced by architectural choices shape the prior in function space, and how these choices ultimately determine
OOD behavior.
In this section, we analyze this OOD behavior for traditional as well as NNGPinduced kernels.
To minimize the impact of the approximations on our results, we exclusively focus on conjugate settings, i.e., regression (see SM C.1 for classiﬁcation results).
Furthermore, this choice [Figure] of Gaussian likelihood induces a direct correspondence between function values and outcomes up to noise corruptions.
Hence, prior knowledge about outcomes can be encoded in function space through the choice of a meaningful function space prior.
Uncertainty quantiﬁcation for OOD detection.
Uncertainty can be quantiﬁed in multiple ways, but is often measured as the entropy of the predictive posterior.
The predictive posterior, however, captures both aleatoric and epistemic uncertainty, which does not allow a distinction between OOD and ambiguous inputs (Mukhoti et al., 2021).
While our choice of likelihood does not permit the modelling of input-dependent uncertainty (Gaussian with ﬁxed variance), a softmax classiﬁer can capture heteroscedastic aleatoric uncertainty arbitrarily well by outputting an input-dependent categorical distribution.
For this reason, uncertainty should be quantiﬁed in a way that allows OOD detection to be based on epistemic uncertainty only.
The function space view of GPs naturally provides such measure of epistemic uncertainty by considering the standard deviation of the posterior over function values σ ( f ∗ ) ≡ (cid:112) diag ( C ( f ∗ ))
(illustrated in Fig. 1a).
Such measure can be naturally translated to BNNs by looking at the disagreement between network outputs when sampling from p ( w | D ) .
Note that in classiﬁcation tasks all models drawn from p ( w | D ) might lead to a highentropy softmax without disagreement, and therefore only an uncertainty measure based on model disagreement prevents one from misjudging ambiguous points as OOD.
As OOD detection is the focus of this paper, we always quantify uncertainty in terms of model disagreement. [Figure]
Bayesian statistics and OOD detection have no intrinsic connection.
As conceptually illustrated in Fig.
1b and demonstrated using exact Bayesian inference in Fig.
3b, predictive uncertainties are not necessarily reﬂective of p ( x ) (and thus not suited for OOD detection), irrespective of how well the underlying task is solved.
GP regression with an RBF kernel.
We next examine the predictive posterior of a GP with squared exponential (RBF) kernel (Eq. 3).
Fig.
3a shows epistemic uncertainty as the standard deviation of p ( f ∗ | X ∗ , X, y ) .
We can notice that σ ( f ∗ ) nicely captures the data manifold and is thus well suited for OOD detection.
This behavior can be understood by considering Eq. 2 while noting that k ( x , x (cid:48) )
= const if x = x (cid:48) , and that the variance of posterior function values can be written as σ 2 ( f ∗ )
=
k ( x ∗ , x ∗ ) − (cid:80) ni
=1 β i ( x ∗ ) k
( x ∗ , x i ) (cf.
SM B), where β i are dataset- and input-dependent.
The second term is reminiscent of using kernel density estimation (KDE) to approximate p ( x ) , applying a Gaussian kernel, while the ﬁrst term is the (constant) prior variance.
Hence, the link between Bayesian inference and OOD detection can be made explicit, as the posterior variance is inversely related to the input distribution.
Loosely speaking, in GP regression as in Eq. 2 and kernels where the KDE analogy holds, epistemic uncertainty can roughly be described as const − p ( x ) .
In this view, one starts with high (prior) uncertainty everywhere, which is only reduced where data is observed.
By contrast, learning a normalized generative model (e.g., using normalizing ﬂows (Papamakarios et al., 2019)) often requires to start from an arbitrary probability distribution.
The OOD behavior induced by NNGP kernels.
Fig.
2a illustrates σ ( f ∗ ) for an inﬁnite-width 2-layer ReLU network (see Fig.
S2 for other common architectural choices).
It is already visually apparent, that in this case the kernel is less suited for OOD detection compared to an RBF kernel.
Moreover, we cannot justify why OOD detection based on σ ( f ∗ ) would be principled for this kernel as the KDE analogy does not hold.
This is due to two reasons: (1) the prior uncertainty k
( x ∗ , x ∗ ) is not constant (Fig. S1), and (2) we empirically do not observe that k ( x , x (cid:48) ) can be related to a distance measure (Fig. S4).
We therefore argue that more theoretical work is necessary if one aims to justify uncertainty-based OOD detection with common architectural choices from the perspective of NNGP kernels.
On this note, maintaining parameter uncertainty and being able to detect OOD samples are often considered crucial requirements of systems deployed in safety-critical applications.
Given that Bayesian inference is not intrinsically linked to OOD detection, care should be taken to precisely communicate safety-relevant capabilities of BNNs to practitioners.
[Figure]
However, the NNGP perspective also allows us to choose an architecture such that the BNN’s uncertainty resembles the, for these problems, desirable OOD behavior of the GP with RBF kernel described above.
In particular, the cosine (Eq. 6) and RBF (Eq. 7) network induce kernels that are related to the RBF kernel.
The last term in Eq.quickly converges to zero for moderate norms of x or x (cid:48) (or high σ 2 b ), which explains why the uncertainty behavior of Fig.
2b is qualitatively identical to Fig. 3a.
In case of the RBF network, the RBF kernel is recovered if (cid:107) x (cid:107) and (cid:107) x (cid:48) (cid:107) are small compared to σ 2 m , explaining why the uncertainty faints towards the boundaries of Fig. 2c.
These examples show that in the non-parametric limit and for (low-dimensional) regression tasks,
BNNs can be constructed such that uncertainty-based OOD detection can be justiﬁed through mathematical argumentation.
We next study whether the observations made in the inﬁnite-width limit are relevant for studying ﬁnite-width neural networks.
Inﬁnite-width uncertainty is consistent with the ﬁnite-width uncertainty.
For ﬁnite-width BNNs exact Bayesian inference is intractable.
To mitigate the effects of approximate inference we resort to Hamiltonian Monte Carlo (HMC, Duane et al., 1987; Neal et al., 2011).
Fig.
2d to 2f show an estimate of σ ( f ∗ ) for ﬁnite-width networks corresponding to the non-parametric limits studied above (illustrations with another dataset can be found in Fig.
S5).
Already for moderate layer widths, the modelled uncertainty resembles the one of the corresponding NNGP.
Given this close correspondence, we conjecture that the tools available for the inﬁnite-width case are useful for designing architectural guidelines that enhance OOD detection.
We illustrated this on low-dimensional problems by studying desirable function space properties induced by the RBF kernel, which can be translated to BNN architectures.
T HE CHOICE OF WEIGHT SPACE PRIOR MATTERS FOR OOD DETECTION
For a neural network, the prior in function space is induced by the architecture and the prior in weight space (Wilson & Izmailov, 2020). [Figure]
In the previous section, we restricted ourselves to a particular class of weight space priors which allowed us to study the architectural choices in the inﬁnite-width limit.
While, in practice, a wide variety of weight space prior choices might lead to good generalization (with respect to test data from p ( x ) p ( y | x ) , e.g., cf.
Izmailov et al.
(2021b)) , the uncertainty behavior that is induced OOD might vary drastically.
This is illustrated in Fig.
4, where for the same network two different choices of p ( w ) lead to vastly different predictive uncertainties despite the fact that both choices ﬁt the data well.
In Sec. 4, we use the inﬁnite-width limit to obtain a function space view that allows to make interesting conjectures about predictive uncertainties on OOD data.
Recently, multiple studies suggested ways to either explicitly encode function space properties into the weight space prior p ( w ) or to use a function space prior when performing approximate inference in neural networks (Flam-Shepherd et al., 2017; Sun et al., 2019; Tran et al., 2020; Matsubara et al., 2021).
However, all these methods depend on the speciﬁcation of a set X R ⊆
X on which desired function space properties will be enforced (Fig. 5).
For instance, the Ridgelet prior proposed by Matsubara et al.
(2021) provides an asymptotically correct weight space prior construction that induces a given GP prior.
Thus, on X R this method allows to meaningfully encode prior knowledge to guide Bayesian inference.
But, an a priori speciﬁcation of a set X R which can be largely covered by samples x R ∈ X R to ensure the desired prior speciﬁcation on X R is practically challenging, and conceptually related to the idea of training on OOD data to calibrate respective uncertainties (cf.
Sec. 2).
Note, we do not aim to phrase this OOD effect as a drawback of these methods, as we do not consider Bayesian inference to be intrinsically related to OOD detectors.
It is, however, important to keep in mind that function space properties deemed beneﬁcial for OOD detection are not straightforward to induce when working in weight space.
A TRADE - OFF BETWEEN GENERALIZATION AND OOD DETECTION
and log marginal likelihood for the examples shown in Fig. 6. [Table] An important desideratum that modelers attempt to achieve when applying Bayesian statistics is good generalization through the incorporation of relevant prior knowledge.
Is this desideratum generally in conﬂict with having high uncertainty on OOD data?
In this section, we provide arguments indicating that the answer to this question can be yes .
Consider data with known periodic structure inside the support of p ( x ) (Fig. 6).
We can either choose to ignore our prior knowledge by selecting a GP prior with RBF kernel (Fig. 6a), or we explicitly incorporate domain knowledge using a function space prior that allocates its mass on periodic functions e.g. the exp-sine-squared (ESS) kernel.
In the former case, we know that OOD uncertainties will be useful for OOD detection (Sec. 4).
In the latter case, however, we see that uncertainties do not reﬂect p ( x ) .
By contrast, the roles are reversed when it comes to assessing generalization.
[Figure] Here, the choice of a periodic kernel has clear beneﬁts, as we can see visually and by comparing the generalization bound introduced in
Sec. 3.3 which can be tightened by minimizing NR D ( p ( f | D ))
+
KL ( p ( f | D ) || p ( f )) .
As reported in Table 1, the bound’s value for the ESS kernel is lower than the RBF.
Interestingly, the bound is minimized if both terms are minimized, and the second term explicitly asks the posterior to remain close to the prior.
Therefore, the second term beneﬁts from restricting the prior function space through the incorporation of prior knowledge, counteracting the need of a rich function space for inducing high OOD uncertainties (Fig. 1).
Consistent with this, also the log marginal likelihood log p ( D ) in Table 1 is higher for the ESS kernel.
An orthogonal problem often considered in the literature is generalization under dataset (or covariate) shift (Quiñonero-Candela et al., 2009; Snoek et al., 2019; Izmailov et al., 2021a). In this case, one deliberately seeks to provide meaningful predictions on test data p test ( x ) that might not overlap in support with our training input distribution p ( x ) . Therefore, prior knowledge needs to explicitly encode how to obtain "generalization on OOD data" as the data cannot speak for themselves.
Such priors are proposed in Izmailov et al.
(2021a), but also Fig. 6b can be viewed as an example of how prior encoding speciﬁes how to generalize to OOD data.  O N THE PRACTICAL VALIDATION OF OOD PROPERTIES
Relating the epistemic uncertainty induced by a BNN to p ( x ) opens up interesting new possibilities.
As we show in SM B, epistemic uncertainty of a GP with RBF kernel can be related to a KDE approximation of p ( x ) .
Therefore, one may consider the epistemic uncertainty as an energy function to create a generative model from which to sample from the input space (Fig. 7) [Figure]
.Moreover, this approach may be used to empirically validate the OOD capabilities of a BNN.
Indeed, OOD performance is commonly validated by selecting a speciﬁc set of known OOD datasets (Izmailov et al., 2021b).
However, as the OOD space comprises everything except the in-distribution data, it is infeasible to gain good coverage on high-dimensional data with such testing approach.
We therefore suggest a reverse approach that checks the consistency in regions of certainty instead of uncertain ones by sampling via the epistemic uncertainty.
Indeed, if these samples and thus the generative model based on uncertainty estimates are consistent with the indistribution data then a strong indication for trustworthiness is provided.
Additionally, a discrepancy measure (Liu et al., 2016; Gretton et al., 2012) can be used to compare the generated samples and training data points and quantitatively assess to which extent the epistemic uncertainty is related to p ( x ) .
Moreover, this approach can open up new research opportunities.
For instance, a continual learner may use its uncertainty to generate its own replay data to combat forgetting (cf. SM C.3).
C ONCLUSION
We challenge the common view that uncertainty-based OOD detection with BNNs is intrinsically justiﬁed.
Our arguments are all based on low-dimensional problems and cannot easily be generalized to real-world problems, but our observations are consistent with the fact that empirically BNNs are often only marginally superior in OOD detection compared to models that do not maintain epistemic uncertainty (Snoek et al., 2019;
Henning et al., 2021). Overall, this work highlights fundamental limitations of BNNs for OOD detection that are not solely explained by the use of approximate inference.
Ethics Statement.
When machine learning algorithms enter real-world applications, they are not anymore embedded in a controlled environment.
Instead, they have to know what they don’t know , which encloses the notion of OOD detection.
It is therefore of utmost importance to discuss and understand the justiﬁcations for employing uncertainty-based OOD detection.
The purpose of our paper is to disclose in a simple but clear manner that these justiﬁcations are not obvious, and thus calls for more research in this direction.
Our illustrative arguments should, however, not be misinterpreted or generalized, for instance, by assuming OOD properties we derive from an RBF kernel can be applied to high-dimensional image manifolds.
Reproducibility Statement.
The source code to reproduce all experiments is available at: anonymous link .
S
UPPLEMENTARY M ATERIAL : U NCERTAINTY - BASED OUT - OF - DISTRIBUTION DETECTION REQUIRES SUITABLE FUNCTION SPACE PRIORS
Anonymous authors
A
W HAT IS AN OUT - OF - DISTRIBUTION INPUT ?
Pimentel et al.
(2014) reviews methods for outlier detection, putting them coarsely into ﬁve categories: (1) probabilistic, (2) distance-based, (3) reconstruction-based, (4) domain-based, and (5) informationtheoretic.
Our focus lies on a probabilistic characterization of an OOD point, where a statistical criterion allows to decide whether a given input is signiﬁcantly different from the observed training population.
In this section, we discuss pitfalls regarding obvious choices for such a criterion in order to highlight the difﬁculties that arise when attempting to agree on a single (or application-dependent) mathematical deﬁnition of outliers.
There are many possible deﬁnitions that could be considered for a point to be OOD as we will outline below.
Any of these deﬁnitions may change the notion of OOD and will therefore affect how a
BNN should be designed such that predictive uncertainty adheres to the underlying OOD deﬁnition.
Considering a generative process p ( x ) , the ﬁrst question arising is regarding the regions outside the support of p ( x ) or even the space outside the manifold where samples x are deﬁned on.
For instance, assume the data to be images embedded on a lower-dimensional manifold.
Are points outside this manifold clearly OOD, given that minor noise corruptions are likely to leave the manifold?
Even disregarding these topological issues solely focusing on the density p ( x ) , makes the distinction between in- and out-of-distribution challenging.
For instance, a threshold-criterion on the density might cause samples in a zero probability region to be considered in-distribution (Nalisnick et al., 2019a).
To overcome such challenges, one may resort to concepts from information theory, such as the notion of a typical set (MacKay, 2003; Nalisnick et al., 2019b).
Unfortunately, an OOD criterion based on this notion would require looking at sets rather than individual points.
We hope that this short outline highlights the challenges regarding the deﬁnition of OOD, but also clariﬁes that a proper deﬁnition is relevant when assessing OOD capabilities on high-dimensional data, where a visual assessment as in Fig.
2 is not possible.
B O N THE RELATION BETWEEN GP REGRESSION AND KERNEL DENSITY ESTIMATION
GP regression with an RBF kernel allows a direct understanding of the OOD capabilities that
a
Bayesian posterior may possess, as the a priori uniform epistemic uncertainty is reduced in direct correspondence to density of p ( x ) .
Below, we rewrite the variance of an input x ∗ as deﬁned in Eq. 2:
[Equation]
with β i ( x ∗ ) =
(cid:80) nj =1 (cid:0) K
( X, X )
+ σ 2 I (cid:1) − 1 ij k ( x ∗ , x j ) .
Note, that k ( x ∗ , x ∗ ) is a positive constant for an RBF kernel, and that β i ( x ∗ ) ≥ 0 .
Furthermore, as the RBF kernel k ( x ∗ , x j ) behaves exponentially inverse to the distance between x ∗ and x j , β i ( x ∗ ) is approximately zero for all x ∗ that are far from all training points.
In addition, a training point x i can only decrease the prior variance if it is close to x ∗ .
This analogy closely resembles the philosophy of KDE, which becomes an exact generative model in the limit of inﬁnite data and bandwidth l → 0 .
C A DDITIONAL EXPERIMENTS AND RESULTS
In this section, we report additional experiments and results.
Fig.
S1 shows the prior’s standard deviation over function values (cid:112) k
( x ∗ , x ∗ ) for several NNGP kernels.
Note, a kernel that a priori treats locations x ∗ differently might not be desirable for OOD detection as the data’s inﬂuence on posterior uncertainties might be hard to interpret (cf.
Sec. 4).
Fig.
S2 shows GP regression results with the GMM dataset (Sec. D) for several NNGP kernels.
The plots show that the underlying task can be solved well with all considered architectures (as indicated by the predictive mean ¯ f ∗ that captures the ground-truth targets in-distribution), even though the uncertainty behavior OOD is vastly different and not reﬂective of p ( x ) .
[Figure] [Figure]
Fig.
S3 (in combination with Fig. S2) highlights that also approximated kernel values can be used to study architectures with no known closed-form solution for Eq. 4, as the posterior seems to be only marginally effected by the MC estimation.
Fig.
S4 visualizes that NNGP kernels for common architectural choices do not encode for Euclidean distances.
Kernels that monotonically decrease with increasing distance are, however, important to apply the KDE interpretation of SM B.
Finally, in Fig.
S5 we investigate a more challenging dataset with two concentric rings.
Note, that the center region as well as the region between the two rings can be considered OOD (not included in the support of p ( x ) ).
A good uncertainty-based OOD detector should therefore depict high uncertainties in those regions, which is not the case for ReLU networks.
[Figure]
whenever no analytic kernel expression is available (for instance, when using a hyberbolic tangent nonlinearity as in Fig. S2).
Here, we visualize the error caused by this approximation for ReLU and error function (erf) networks when computing kernel values k ( x ∗ , x ∗ ) .
Eq.
4 requires a recursive estimation of expected values, where we estimate each of them using N samples ( N ∈ [10 2 , 10 3 , 10 4 , 10 5 ] ).
Note, that even small errors can cause eigenvalues of the kernel matrix to become negative.
However, with our chosen likelihood variance of σ = 0 .
02 we experience no numerical instabilities during inference, and obtain consistent results using either analytic or estimated ( N = 10 5 ) kernel matrices.
[Figure]
kernel values k ( x q , x p ) plotted as a function of the Euclidean distance (cid:107) x q − x p (cid:107) 2 (using pairs of training points from the two Gaussian mixtures dataset).
As outlined in Sec. 4 and SM B, the interpretation of an RBF kernel as Gaussian kernel that can be used in a KDE of p ( x ) is important to justify implied OOD, at least for low-dimensional problems.
Unfortunately, the kernels induced by common architectures do not seem to be distance-aware and are thus not useful for KDE.
[Figure]
C. 2 D CLASSIFICATION
In this section, we consider the same dataset as in Fig 2 in a classiﬁcation setting instead of regression.
In this setting, exact inference is intractable even when using GPs and approximations are needed.
In particular, we study classiﬁcation with the logistic likelihood function p ( y | f
( x ; w ))
=
s ( − y f
( x ; w )) where s is the sigmoid function.
We use HMC to sample from the posterior distribution in the ﬁnite-width limit, using a standard normal prior where the variance is inversely scaled by the hidden-layer’s width.
The results are reported in Fig S6 for ReLU, cosine and
RBF networks.
As clearly depicted in the plots, our ﬁndings regarding the importance of the prior in function space can be extended also for the classiﬁcation case.
Indeed, the uncertainty captured by the ReLU architecture appears unsuitable for OOD detection given that the data distribution is not captured and the disagreement is low also in regions of the 2D plane that do not contain any training data points (also see Kristiadi et al., 2020; 2021).
This pathology appears in both cases: when the disagreement is considered over the functions σ ( f ∗ ) or over the sigmoid outputs σ ( s ( f ∗ )) .
By contrast, also for this new choice of likelihood, the cosine and RBF architecture exhibit uncertainty that conveys similar useful properties for OOD detection as seen in the regression example in Fig 2.
Despite this empirical correspondence, it is important to note that the direct correspondence with the KDE technique that justiﬁes the usefulness of OOD properties of the RBF kernel in the case of GP regression (see Sec.
B) does not directly apply for classiﬁcation.
Indeed in the latter, we do not have an analytical form of the posterior as in Eq.
2 due to the non-Gaussianity of the logistic likelihood.
Therefore our observations can not be readily generalized outside the settings of our experiments and further theoretical analysis are needed to asses an analogous justiﬁcation for non-Gaussian likelihoods.
C. S PLIT MNIST REGRESSION
In this section, we consider SplitMNIST tasks (Zenke et al., 2017), which are binary decision tasks where the original MNIST dataset is split into ﬁve tasks containing two digits each: 0/1, 2/3, 4/5, 6/7, 8/9.
To perform exact inference via Gaussian Processes, we consider each binary decision task as regression problem with labels -1/1.
For computational reasons, the training set of each task is reduced to 1000 samples, but test sets remain at their original size.
Results are reported in Table S1.
For the chosen length-scale parameters l , RBF kernels show best generalization.
Though, they don’t excel at OOD detection compared to other kernel choices.
In SM B, we discussed that epistemic uncertainties as induced by the RBF kernel can be viewed as an approximation to p ( x ) when the
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
dataset size goes to inﬁnity D → ∞ , and the length-scale goes to zero l → 0 .
As expected, such approximation to p ( x ) may deteriorate on high-dimensional image data, presumably as the Euclidean distance does not capture the geometry of the image manifold.
[Figure]
To gain a better intuition on why the RBF kernel does not excel in OOD detection for image data we visualize the uncertainty behavior in Fig.
S7.
The ﬁgure shows 2D linear subspaces of the 784D image space.
These subspaces are determined by three images (see caption for details).
As can be seen in Fig.
S7b and S7e, a too small length-scale might cause test points to be not included in the low-uncertainty regions.
On the other hand, if the length-scale is set too high, OOD points may fall inside low-uncertainty regions.
The situations depicted in the ﬁgure show that for the given training set there is no trade-off length-scale l that prevents such behavior (because some OOD points have less Euclidean distance to training points than some test points).
Using an RBF kernel with a metric that encodes similarities in image space rather than using Euclidean distance might overcome these problems.
C. C ONTINUAL LEARNING VIA UNCERTAINTY - BASED REPLAY
In Sec.
7 we mention that the desideratum of having low uncertainty only on in-distribution inputs opens the possibility of considering uncertainty as an energy function from which we can sample.
Thus, the uncertainty landscape can be used to construct a generative model.
In this section, we use continual learning (Parisi et al., 2019) to demonstrate this conceptual idea.
In continual learning, a sequence of tasks D (1) , . . .
, D ( T ) is learned sequentially such that each task has to be learned without access to data from past or future tasks.
A common approach to continual learning is via replay (Lin, 1992).
In this case, the current task t is learned with the available training data D ( t ) as well as datasets ˜ D (1) , . . . ,
˜ D ( t − 1) which are supposed to represent the data distributions of the previous tasks.
In the simplest case, ˜ D ( s ) is constructed by storing a subset of D ( s ) (Lin, 1992).
Other approaches learn a separate generative model that can be used replay (fake) data ˜ D ( s ) while a new task is learned (e.g., Shin et al., 2017; von Oswald et al., 2020)
We consider this (pseudo-)replay scenario without the need of maintaining a separate generative model.
In Fig.
S8, we consider a regression problem split into two tasks.
Note, that learning a series
[Figure]
of 1D regression problems
is challenging for most continual learning algorithms, especially those relying on the recursive Bayesian update: p ( w | D (1) , D (2) )
∝ p ( D (2) | w ) p ( w | D (1) )
(Henning et al., 2021).
By contrast, (pseudo-)replay methods can solve this task with ease as the structure of the data in such low-dimensional problem is easy to capture by a generative model.
In our experiment two tasks are created by splitting the dataset of a 1D function into two parts as illustrated in Fig.
S8a.
The left half is considered the ﬁrst task and the right half the second task, respectively.
The blue posteriors shown in all three subpanels are obtained with a GP using the NNGP kernel of a 1-layer Cosine network, which should exhibit OOD properties similar to an RBF kernel (cf.
Sec. 4).
In Fig.
S8a, the posterior is obtained using the combined datasets from both tasks (no continual learning).
In Fig.
S8b, the posterior is obtained using only the training data of the ﬁrst task.
After this (ﬁrst task’s) posterior is obtained, we can use it to perform pseudo-replay as outlined below to generate ˜ D (1) which can be mixed with the data of the second task D (2) to produce the posterior in
Fig.
S8c.
The posterior in Fig.
S8c looks qualitatively similar to the one in Fig.
S8a even though it has been obtained without direct access to the ﬁrst task’s training data D (1) .
To generate ˜ D (1) while learning the second task, we need access to the posterior of the ﬁrst task.
Note, the model from the previous task is often kept in memory by continual learning algorithms, for instance, to use it for regularization purposes (Kirkpatrick et al., 2017) or to replay data with previously trained generative models (Shin et al., 2017).
7 We generate in-distribution input locations (denoted by orange vertical bars in Fig.
S8b)
via rejection sampling as in Fig.
For that, we deﬁne epistemic uncertainty as an energy function E ( x ∗ )
≡ σ (cid:0) f ∗ ( x ∗ ) (cid:1) and construct a Boltzmann distribution ˜ p ( x ∗ )
{+∝ exp (cid:0) − E ( x ∗ ) /T (cid:1) , where we set the temperature T = 1 . 
+}
As uncertainty of the posterior of Fig.
S8b is only low for in-distribution inputs, we can assume obtained input locations are similar to those represented in D (1) .
To construct a dataset ˜ D (1) , we use the posterior of Fig.
S8b to sample predictions y ∗ (orange dots in Fig.
S8b and Fig.
S8c).
In summary, this section presents an example application that can arise if the desideratum of high uncertainty on all OOD inputs (i.e., robust OOD detection) is fulﬁlled by a Bayesian model.
Whether
BNNs can be constructed to fulﬁll this desideratum on real-world data is, however, unclear at this point.
D E XPERIMENTAL DETAILS
In this section, we report details on our implementation for the experiments conducted in this work.
In all two-dimensional experiments: the likelihood is ﬁxed to be Gaussian with variance 0 .
Unless noted otherwise, the prior in weight space is always a width-aware centered Gaussian with σ 2 w = 1 .and
σ 2 b = 1 . 0 except for the experiments involving RBF networks where we select σ 2 w = 200 . 0 .
The
RBF kernel bandwidth was ﬁxed to 1 .
in all corresponding experiments.
1D regression.
We construct the 1D regression example (e.g., Fig 1a) by deﬁning p ( x ) uniformly within the ranges [1 . 0 , 1 . 3] , [3 . 5 , 3 . 8] and [5 . 2 , 5 . 5] , and p ( y | x ) as f ( x ) + (cid:15) with f ( x ) = 2 sin( x ) + sin( √ 2 x ) + sin( √ 3 x ) and (cid:15) ∼ N (0 , 0 . 2 2 ) .
The training set has size 20.
Periodic 1D regression (only Fig. 6).
We construct this 1D regression example by deﬁning p ( x ) uniformly within the range [0 . 0 , 12 . 5] , and p ( y | x ) as f ( x )+ (cid:15) with f ( x ) = sin( x ) and (cid:15) ∼ N (0 , 3 2 ) . The training set has size 30.
Gaussian Mixture.
We created a two-dimensional mixture of two Gaussians with means µ 1 = ( − 2 , − 2) , µ 2 = (2 , 2) and covariance Σ = 0 .
5 · I and sampled 20 training data points.
Two rings.
We uniformly sampled 50 training data points from two rings centred in (0 , 0) with inner and outer radii R i, 1 = 3 , R o, 1 = 4 , R i, 2 = 8 , R o, 2 = 9 , respectively.
HMC.
We use 5 parallel chains for 5000 steps with each constituting 50 leapfrog steps with stepsize 0 .
001 for width 5 and 0 . 0001 for width 100.
We considered a burn-in phase of 1000 steps and collected a total of 1000 samples.