T OWARDS NEURAL NETWORKS THAT PROVABLY KNOW WHEN THEY DON ’ T KNOW
A BSTRACT
It has recently been shown that ReLU networks produce arbitrarily over-conﬁdent predictions far away from the training data.
Thus, ReLU networks do not know when they don’t know.
However, this is a highly important property in safety critical applications.
In the context of out-of-distribution detection (OOD) there have been a number of proposals to mitigate this problem but none of them are able to make any mathematical guarantees.
In this paper we propose a new approach to OOD which overcomes both problems.
Our approach can be used with ReLU networks and provides provably low conﬁdence predictions far away from the training data as well as the ﬁrst certiﬁcates for low conﬁdence predictions in a neighborhood of an out-distribution point.
In the experiments we show that stateof-the-art methods fail in this worst-case setting whereas our model can guarantee its performance while retaining state-of-the-art OOD performance
I NTRODUCTION
Deep Learning Models are being deployed in a growing number of applications.
As these include more and more systems where safety is a concern, it is important to guarantee that deep learning models work as one expects them to.
One topic that has received a lot of attention in this area is the problem of adversarial examples, in which a model’s prediction can be changed by introducing a small perturbation to an originally correctly classiﬁed sample.
Achieving robustness against this type of perturbation is an active ﬁeld of research.
Empirically, adversarial training (Madry et al., 2018) performs well and provably robust models have been developed (Hein & Andriushchenko, 2017;
Wong & Kolter, 2018; Raghunathan et al., 2018; Mirman et al., 2018;
Cohen et al., 2019).
On the other end of the spectrum it is also important to study how deep learning models behave far away from the training samples.
A simple property every classiﬁer should satisfy is that far away from the training data, it should yield close to uniform conﬁdence over the classes: it knows when it does not know.
However, several cases of high conﬁdence predictions far away from the training data have been reported for neural networks, e.g. fooling images (Nguyen et al., 2015), for out-of-distribution (OOD) images (Hendrycks & Gimpel, 2017a) or in medical diagnosis (Leibig et al., 2017).
Moreover, it has been observed that, even on the original task, neural networks often produce overconﬁdent predictions (Guo et al., 2017).
Very recently, it has been shown theoretically that the class of ReLU networks (all neural networks which use a piecewise afﬁne activation function), which encompasses almost all standard models, produces predictions with arbitrarily high conﬁdences far away from the training data (Hein et al., 2019).
Unfortunately, this statement holds for almost all such networks and thus without a change in the architecture one cannot avoid this phenomenon.
Traditionally, the calibration of the conﬁdence of predictions has been considered on the indistribution (Guo et al., 2017; Lakshminarayanan et al., 2017a).
However these techniques cannot be used for detection (Leibig et al., 2017).
Only recently the detection of OOD inputs (Hendrycks & Gimpel, 2017a) has been tackled.
The existing approaches are roughly of two types:
ﬁrst, postprocessing techniques that adjust the estimated conﬁdence (DeVries & Taylor, 2018; Liang et al., 2018) which includes the baseline ODIN.
Second, modiﬁcation of the classiﬁer training by integrating generative models like a VAE or GAN in order to discriminate out-distribution from in-distribution data (Lee et al., 2018a;
Wang et al., 2018; Lee et al., 2018b) or approaches which enforce low conﬁdence on OOD inputs during training (Hein et al., 2019; Hendrycks et al., 2019).
Worst-case aspects of OOD detection have previously been studied in Nguyen et al.
; Schott et al.
Hein et al.
(2019); Sehwag et al.
, but no robustness guarantees have yet been proposed for this setting.
A generalization guarantee for an out-of-distribution detection scheme is provided in Liu et al.
While this is the only guarantee we are aware of, it is quite different from the type of guarantees we present in this paper.
In particular, none of those approaches are able to guarantee that neural networks produce low conﬁdence predictions far away from the training data.
We prove that our classiﬁer satisﬁes this requirement even when we use ReLU networks as the classiﬁer modelwithout loosing performance on either the prediction task on the in-distribution nor the OOD detection performance, see Figure 1 for an illustration.
Moreover, our technique allows to give upper bounds on the conﬁdence over a whole neighborhood around a point (worst-case guarantees).
We show that most state-of-the-art OOD methods can be fooled by maximizing the conﬁdence in this ball even when starting from uniform noise images, which should be trivial to identify.
The central difference from existing OOD-methods is that we have a Bayesian framework for in-and out-distribution, where we model in-and out-distribution separately.
In this framework our algorithm for training neural networks follows directly as maximum likelihood estimator which is different from the more ad-hoc methods proposed in the literature.
The usage of Gaussian mixture models as the density estimator is then the essential key to get the desired provable guarantees.
A GENERIC MODEL FOR CLASSIFIERS WITH CERTIFIED LOW CONFIDENCE FAR AWAY FROM THE TRAINING DATA
The model which we propose in this paper assumes that samples from an out-distribution are given to us.
In image recognition we could either see the set of all images as a sample from the outdistribution (Hendrycks et al., 2019) or consider the agnostic case where we use use uniform noise on [0 , 1] d as a maximally uninformative out-distribution.
In both settings one tries to discriminate these out-distribution images from images coming from a particular image recognition task and the task is to get low conﬁdence predictions on the out-distribution images vs. higher conﬁdence on the images from the actual task.
From the general model we derive under minimal assumptions a maximum-likelihood approach where one trains both a classiﬁer for the actual task and density estimators for in- and out-distribution jointly.
As all of these quantities are coupled in our model for the conditional distribution p ( y | x ) we get guarantees by controlling the density estimates far away from the training data.
This is a crucial difference to the approaches of Lee et al.
(2018a)
; Wang et al.
; Hendrycks et al.
(2019) which empirically yield good OOD performance but are not able to certify the detection mechanism.
2. A PROBABILISTIC MODEL FOR IN - AND OUT - DISTRIBUTION DATA
We assume that there exists a joint probability distribution p ( y, x ) over the in- and out-distribution data, where y are the labels in { 1 , . . .
, M } , M is the number of classes, and x ∈ R d , where d is the input dimension.
In the following, we denote the underlying probabilities/densities with p ( y | x ) resp.
p ( x ) and the estimated quantities with ˆ p ( y | x ) and ˆ p ( x ) .
We are mainly interested in a discriminative framework, i.e. we want to estimate p ( y | x ) which one can represent via the conditional distribution of the in-distribution p ( y | x, i ) and out-distribution p ( y | x, o ) : p ( y [Equation] p
[Equation]
[Equation] p [Equation] Note that at ﬁrst it might seem strange to have a conditional distribution p ( y | x, o ) for out-distribution data, but until now we have made no assumptions about what in-and out-distribution are.
A realistic scenario would be that at test time we are presented with instances x from other classes (outdistribution) for which we expect a close to uniform p ( y | x, o ) .
Our model for ˆ p ( y | x ) has the same form as p ( y | x )
[Equation]
Typically, out-distribution data has no relation to the actual task and thus we would like to have uniform conﬁdence over the classes.
Therefore we set in our model
[Equation]
and
[Equation]
where f : R d → R M is the classiﬁer function (logits).
This framework is generic for classiﬁers trained with the cross-entropy (CE) loss (as the softmax function is the correct link function for the CE loss) and we focus in particular on neural networks.
For a ReLU network the classiﬁer function f is componentwise a continuous piecewise afﬁne function and has been shown to produce asymptotically arbitrarily highly conﬁdent predictions (Hein et al., 2019), i.e. the classiﬁer gets more conﬁdent in its predictions the further it moves away from its training data.
One of the main goals of our proposal is to ﬁx this behavior of neural networks in a provable way.
Note that with the choice of ˆ p ( y | x, o ) and non-zero priors for ˆ p ( i ) , ˆ p ( o ) , the full model ˆ p ( y | x ) can be seen as a calibrated version of ˆ p ( y | x, i ) , where ˆ p ( y | x ) ≈ ˆ p ( y | x, i ) for inputs with ˆ p ( x | i ) (cid:29) ˆ p ( x | o ) and ˆ p ( y | x ) ≈ 1 M if ˆ p ( x | i ) (cid:28) ˆ p ( x | o ) .
However, note that only the conﬁdence in the prediction ˆ p ( y | x ) is affected, the classiﬁer decision is still done according to ˆ p ( y | x, i ) as the calibration does not change the ranking.
Thus even if the OOD data came from the classiﬁcation task we would like to solve, the trained classiﬁer’s performance would be unaffected, only the conﬁdence in the prediction would be damped.
For the marginal out-distribution ˆ p ( x | o ) there are two possible scenarios.
In the ﬁrst case one could concentrate on the worst case where we assume that p ( x | o ) is maximally uniformative (maximal entropy).
This means that ˆ p ( x | o ) is uniform for bounded domains e.g. for images which are in [0 , 1] d , ˆ p ( x | o )
= 1 for all x ∈ [0 , 1] d , or ˆ p ( x | o ) is a Gaussian for the domain of R d (the Gaussian has maximum entropy among all distributions of ﬁxed variance).
However, in this work we follow the approach of Hendrycks et al.
(2019) where they used the 80 million tiny image dataset (Torralba et al., 2008) as a proxy of all possible images.
Thus we estimate the density of ˆ p ( x | o ) using this data.
In order to get guarantees, the employed generative models for ˆ p ( x | i ) and ˆ p ( x | o ) have to be chosen in a way that allows one to control predictions far away from the training data.
Variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014), normalizing ﬂows (Dinh et al., 2016;
Kingma & Dhariwal, 2018) and generative adversarial networks (GANs) (Goodfellow et al., 2014) are powerful generative models.
However, there is no direct way to control the likelihood far away from the training data.
Moreover, it has recently been discovered that VAEs, ﬂows and GANs also suffer from overconﬁdent likelihoods (Nalisnick et al., 2019; Hendrycks et al., 2019) far away from the data they are supposed to model as well as adversarial samples (Kos et al., 2017).
For ˆ p ( x | o ) and ˆ p ( x | i ) we use a Gaussian mixture model (GMM) which is less powerful than a VAE but has the advantage that the density estimates can be controlled far away from the training data:
[Equation]
where K i , K o ∈ N are the number of centroids and d : R d × R d → R is the metric
[Equation]
with C being a positive deﬁnite matrix and
[Equation]
We later ﬁx C as a slightly modiﬁed covariance matrix of the in-distribution data (see Section 4 for details).
Thus one just has to estimate the centroids µ k , ν l and the variances σ 2 k , θ 2 l .
The idea of this metric is to use distances adapted to the data-distribution.
Note that equation 4 is a properly normalized density in R d .
2. M AXIMUM LIKELIHOOD ESTIMATION
Given models for ˆ p ( y | x ) and ˆ p ( x ) we effectively have a full generative model and apply maximum likelihood estimation to get the underlying classiﬁer ˆ p ( y | x, i ) and the parameters of the Gaussian mixture models ˆ p ( x | i ) , ˆ p ( x | o ) .
The only free parameter left is the probability ˆ p ( i ) , ˆ p ( o ) which we write compactly as λ
=
ˆ p ( o ) ˆ p ( i ) .
In principle this parameter should be set considering the potential cost of over-conﬁdent predictions.
In our experiments we simply ﬁx it to λ = 1 .
[Equation]
In practice, we have to compute empirical expectations from ﬁnite training data from the indistribution ( x i , y i ) n i i =1 and out-distribution ( z j ) n o j =1 .
Labels for the out-distribution could be generated randomly via p ( y | x, o )
= 1 M , but we obtain an unbiased estimator with lower variance by averaging over all classes directly, as was done in Lee et al.
(2018a);
Hein et al.
; Hendrycks et al.
Now we can estimate the classiﬁer f and the mixture model parameters µ, ν, σ, θ via
[Equation]
with
[Equation]
and ˆ
[Equation]
Due to the bounds derived in Section 3, we denote our method by Certiﬁed Certain Uncertainty (CCU) .
Note that if one uses a standard neural network model with softmax, i.e.
ˆ p ( y | x ) =
ˆ p ( y | x, i ) =
[Equation]
1 e fm ( x ) , then the ﬁrst term in equation 6 would be the cross-entropy loss for the in-distribution data and the second term the cross entropy loss for the out-distribution data with a uniform distribution over the classes.
For this choice of ˆ p ( y | x ) and neglecting the terms for ˆ p ( x ) we recover the approach of Hein et al.
; Hendrycks et al.
for training a classiﬁer which outputs uniform conﬁdence predictions on out-distribution data where ˆ p ( i ) ˆ p ( o ) corresponds to that regularization parameter λ .
The key difference in our approach is that ˆ p ( y | x ) (cid:54) =
ˆ p ( y | x, i ) and the estimated densities for in- and out distribution ˆ p ( x | i ) and ˆ p ( x | o ) lead to a conﬁdence calibration of ˆ p ( y | x ) , and in turn the ﬁt of the classiﬁer inﬂuences the estimation of ˆ p ( x | i ) and ˆ p ( x | o ) .
The major advantage of our model is that we can give guarantees on the conﬁdence of the classiﬁer decision far away from the training data.
P ROVABLE GUARANTEES FOR CLOSE TO UNIFORM PREDICTIONS FAR AWAY FROM THE TRAINING DATA
In this section we provide two types of guarantees on the conﬁdence of a classiﬁer trained according to our model in equation 6.
The ﬁrst one says that the classiﬁer has provably low conﬁdence far away from the training data, where an explicit bound on the minimal distance is provided, and the second provides an upper bound on the conﬁdence in a ball around a given input point.
The latter bound resembles robustness guarantees for adversarial samples (Hein & Andriushchenko, 2017; Wong &
Kolter, 2018; Raghunathan et al., 2018; Mirman et al., 2018) and is quite different from the purely empirical evaluation done in OOD detection papers as we show in Section 4.
We provide our bounds for a more general mixture model which includes our GMM in equation 4 as a special case.
To our knowledge, these are the ﬁrst such bounds for neural networks and thus it is the ﬁrst modiﬁcation of a ReLU neural network so that it provably “knows when it does not know”
(Hein et al., 2019) in the sense that far away from the training data the predictions are close to uniform over the classes.
Theorem 3.1.
Let ( x ( i ) i , y ( i ) i ) ni
=1 be the training set of the in-distribution and let the model for the conditional probability be given as
[Equation]
where λ = ˆ p ( o )
ˆ p ( i ) > 0 and let the model for the marginal density of the in-distribution
ˆ p ( x | i ) and out-distribution p ( x | o ) be given by the generalized GMMs
[Equation]
with α k , β l > 0 and µ k , ν l ∈ R d ∀ k = 1 , . . .
K i , l = 1 , . . .
, K o and d : R d × R d →
R + a metric.
Let z ∈ R d and deﬁne k ∗ =
arg min k =1 ,...,K i d ( z,µ k ) σ k , i ∗ = arg min i =1 ,...,n d ( z, x i ) , l ∗ = arg max l =1 ,...,K o β l exp (cid:16) − d
( z,ν l ) [-
 [Equation]-]2 2 θ 2 l (cid:17) and ∆ =
θ 2 l ∗ σ 2 k ∗ − 1 .
For any (cid:15) > 0 , if min l θ l > max k σ k and min i =1 ,...,n d ( z, x
[Equation]
≥
[Equation]
[Equation]
then it holds for all m ∈ { 1 , . . .
, M } that
[Equation]
In particular, if min i d ( z, x i ) → ∞ , then ˆ p ( m | z ) → 1 M .
The proof is given in Appendix A. Theorem 3.1 holds for any multi-class classiﬁer which deﬁnes for each input a probability distribution over the labels.
Given the parameters of the GMM’s it quantiﬁes at which distance of an input z to the training set the classiﬁer achieves close to uniform conﬁdence.
The theorem holds even if we use ReLU classiﬁers which in their unmodiﬁed form have been shown to produce arbitrarily high conﬁdence far away from the training data Hein et al.
This is a ﬁrst step towards neural networks which provably know when they don’t know.
In the next corollary, we provide an upper bound on the conﬁdence over a ball around a given data point.
This allows to give “conﬁdence guarantees” for a whole volume and thus is much stronger than the usual pointwise evaluation of OOD methods.
Corollary 3.1.
Let x 0 ∈ R d and R > 0 , then with λ
=
ˆ p ( o ) ˆ p ( i ) it holds
[Equation]
where b =
[Equation]
The proof is in the Appendix B. We show in Section 4 that even though OOD methods achieve low conﬁdence on noise images, the maximization of the conﬁdence in a ball around a noise point (adversarial noise) yields high conﬁdence predictions for OOD methods, whereas our classiﬁer has provably low conﬁdence, as certiﬁed by Corollary 3.1.
The failure of OOD methods shows that the certiﬁcation of entire regions is an important contribution of CCU which goes beyond the purely sampling-based evaluation.
[Figure]
in boldface.
All other OOD methods fail on some dataset, see Table 1 for a quantitative version.
ODIN at high temperatures always returns low conﬁdence, so a value of 0.1 is not informative.
E XPERIMENTS
We evaluate the worst-case performance of various OOD detection methods within regions for which
CCU yields guarantees and by standard OOD on MNIST (LeCun et al., 1998), FashionMNIST (Xiao et al., 2017), SVHN (Netzer et al., 2011), CIFAR10 and CIFAR100 (Krizhevsky & Hinton, 2009).
We show that all other OOD methods yield undesired high conﬁdence predictions in the certiﬁed low conﬁdence regions of CCU and thus would not detect these inputs as out-distribution.
For calibrating hyperparameters resp.
training we use for all OOD methods the 80 Million Tiny Images (Torralba et al., 2008) as out-distribution Hendrycks et al.
(2019) which yields a fair and realistic comparison.
CCU: As the Euclidean metric is known to be a relatively bad distance between two images we instead use the distance d ( x, y ) =
(cid:13)(cid:13)(cid:13) C − 12 ( x − y )
(cid:13)(cid:13)(cid:13) , where C is generated as follows.
We calculate the covariance matrix C (cid:48) on augmented in-distribution samples (see C.1).
Let ( λ i , u i ) di
=1 be the eigenvalues/eigenvectors of C (cid:48) .
Then we set
[Equation]
(12) that is we ﬁx a lower bound on the smallest eigenvalue so that C has full rank.
In Hendrycks &
Gimpel (2017b) a similar metric has been used for detection of adversarial images.
We choose K i =
K o = 100 as the number of centroids for the GMMs.
We initialize the in-GMM on augmented in-data using the EM algorithm with spherical covariance matrices in the transformed space, as in equation 4.
For the out-distribution we use a subset of 20000 points for the initialization.
While, initially it holds that ∀ k, l : σ k < θ l , as required in Theorem 3.1, this is not guaranteed during the optimization of equation 6.
Thus, we enforce the constraint during training by setting: θ l (cid:55)→ max { θ l , 2 max k σ k } at every gradient step.
Since the “classiﬁer” and “density” terms in equation 6 have very different magnitudes we choose a small learning rate of 1 e − 5 for the parameters in the GMMs.
It is also crucial to not apply weight decay to these parameters.
The other hyperparameters are chosen as in the base model below.
Benchmarks: For all OOD methods we use LeNet on MNIST and a Resnet18 (for GAN and
MCD we use VGG) otherwise.
The hyperparameters used during training can be found in Appendix
[Table] C. The AUC (area under ROC) is computed by treating in-distribution versus out-distribution as a two problem using the conﬁdence/ of the as criterion.
Alternatively one could report the AUPR (area under precision-recall curve) which we do in Appendix G. MCD: MonteCarlo Dropout (Gal & Ghahramani, 2016) uses dropout at train and at test time.
Since it is not clear where to put the dropout layers in a ResNet, we use VGG instead.
We take the softmax from 7 forward passes (Shafaei et al., 2018) and use the mean of the output for prediction and the variance as score.
EDL: Evidential deep learning (Sensoy et al., 2018) replaces the softmax layer of a neural network and introduces a different loss function that encourages better uncertainty estimates.
DE:
Deep ensembles (Lakshminarayanan et al., 2017b) average the softmax outputs of ﬁve models that were adversarially trained via FGSM (Goodfellow et al., 2015) with step size (cid:15) = 0 .
GAN: The framework of conﬁdence-calibrated classiﬁers (Lee et al., 2017) relies on training a GAN alongside a classiﬁer such that the GAN’s generator is encouraged to generate points close to but not on the in-distribution.
On these points one then enforces uniform conﬁdence.
We used their provided code to train a VGG this way, as we were unable to adapt the method to a ResNet with an acceptable test error (e.g. TE < 30% on SVHN).
ODIN:
ODIN (Liang et al., 2017) consists of two parts: a temperature T by which one rescales the logits before the softmax layer [Table]
[Table] and a preprocessing step that applies a single FGSM-step (Goodfellow et al., 2015) of length (cid:15) before evaluating the input.
The two parameters are calibrated on the out-distribution.
Maha:
The approach in Lee et al. (2018c) is based on computing a class-conditional Mahalanobis distance in feature space and applying an ODIN-like preprocessing step for each layer.
Following Ren et al.
we use a single-layer version of Lee et al. (2018c) on our networks’ penultimate layers because the multi-layer version in the original code does not support gradient-based attacks.
OE: Outlier exposure (Hendrycks et al., 2019) enforces uniform conﬁdence on a large out-distribution.
We use their provided code to train a model with our chosen architecture.
ACET: Adversarial conﬁdence enhanced training (ACET) (Hein et al., 2019) enforces low conﬁdence on a ball around points from an out-distribution by running adversarial attacks during training.
In order to make the comparison with OE more meaningful we use 80M tiny images to draw the seeds rather than smoothed uniform noise as in Hein et al.
We refer to Appendix F for a discussion of the inﬂuence of this choice on the results.
Some of the above OOD papers optimize their hyperparameters on a validation set for each outdistribution they test on.
However, this leads to different classiﬁers for each out-distribution dataset [Table] which seems unrealistic as we want to have good generic OOD performance and not for a particular dataset.
Thus we keep the comparison realistic and fair by calibrating the hyperparameters of all methods on a subset of 80M tiny images and then evaluating on the other unseen distributions.
Certiﬁed robustness against adversarial noise: We sample uniform noise images as they are obviously out-distribution for all tasks and certify using Corollary 3.1 the largest ball around the uniform noise sample on which CCU attains at most 1 . 1 · uniform conﬁdence, that is 1 .
1% on
CIFAR100 and 11% on all other datasets.
We describe how to compute the radius of this ball in
Appendix D.
In principle it could be possible that the certiﬁed balls contain training or test images.
In Appendix E we show that this is not the case.
We construct adversarial noise samples for all OOD methods by maximizing the conﬁdence/minimizing the score via a PGD attack with 500 steps andrandom restarts on this ball.
Further details of the attack can be found in Appendix C.2.
In Tablewe show the results of running this attack on the different models.
We used 200 noise images and we report clean test error on the in-distribution, the success rate (SR) (fraction of adversarial noise points for which the conﬁdence resp.
score inside the ball is higher resp.
lower than the median of the in-distribution’s conﬁdence/score) and the AUC for the separation of the generated adversarial noise images and the in-distribution based on conﬁdence/score.
By construction, see Corollary 3.1, our method provably makes no overconﬁdent predictions but we nevertheless run the attack on CCU as well.
We note that only CCU performs perfectly on this task for all datasets - all other OOD methods fail at least on one dataset, most of them on all.
We also see that ACET achieves very robust performance which may be expected as it does some kind of adversarial training for OOD detection.
Nevertheless, even though they are very rare, high-conﬁdence adversarial noise images for ACET can be found on SVHN, CIFAR10 and CIFAR100 and ACET has no guarantees.
We illustrate the generated adversarial noise images for all methods in Figure 2.
OOD performance: For each dataset and method we report the AUC for the binary classiﬁcation problem of discriminating in- and out-distribution based on conﬁdence resp.
score.
The results are shown in Table 2.
The list of datasets we use for OOD detection can be seen in Table 2.
LSUN_CR refers to only the classroom class of LSUN and Imagenet- is a subset of 10000 resized Imagenet validation images, that have no overlap with CIFAR10/CIFAR100 classes.
The noise dataset was obtained as in Hein et al.
by ﬁrst shufﬂing the pixels of the test images in the in-distribution and then smoothing them by a Gaussian ﬁlter of uniformly random width, followed by a rescaling so that the images have full range.
GrCIFAR10 refers to the images in CIFAR10 being grayscaled and resized to 28x28 and Uniform describes images sampled uniformly at random from [0 , 1] d .
We see that OE and CCU have the best OOD performance.
MCD is worse than the base model which conﬁrms the results found in Leibig et al.
(2017) that MCD is not useful for OOD.
DE outperforms EDL but is not much better than the baseline for CIFAR10 and CIFAR100.
The performance of Maha is worse than what has been reported in Lee et al.
(2018c) which can have two reasons.
We just use their version where one uses the scores only from the last layer and we do not calibrate hyperparameters for each test set separately but just once on the Tiny Image dataset.
Especially on CIFAR10 we found that the results depend strongly on the step size.
The results of ACET, GAN and ODIN are mixed but clearly outperform the baseline.
Comparing Table 1 and Table 2 we see that most models perform well when evaluating on uniform noise but fail when ﬁnding the worst case in a small neighborhood around the noise point.
Thus we think that such worst-case analysis should become standard in OOD evaluation.
C ONCLUSION
In Hein et al.
(2019) it has recently been shown that ReLU networks produce arbitrarily highly conﬁdent predictions far away from the training data, which could only be resolved by a modiﬁcation of the network architecture.
With CCU we present such a modiﬁcation which explicitly integrates a generative model and provably show that the resulting neural network produces close to uniform predictions far away from the training data.
Moreover, CCU is the only OOD method which can guarantee low conﬁdence predictions over a whole volume rather than just pointwise
and we show that all other OOD methods fail in this worst-case setting.
CCU achieves this without loss in test accuracy or OOD performance.
In the future it would be interesting to use more powerful generative models for which one can also guarantee their behavior far away from the training data.
This is currently not the case for VAEs and GANs (Nalisnick et al., 2019; Hendrycks et al., 2019).
A CKNOWLEDGMENTS
The author acknowledge support from the BMBF through the Tübingen AI Center (FKZ: 01IS18039A) and by the DFG TRR 248, project number 389792660 and the DFG Excellence Cluster “Machine
Learning -New Perspectives for Science”, EXC 2064/1, project number 390727645.
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting
Alexander Meinke
A A PPENDIX - P ROOF OF T HEOREM 3.
Theorem 3.1.
Let ( x i , y i ) ni
=1 be the training set of the training distribution.
We deﬁne the model for the conditional probability over the classes y ∈ { 1 , . . .
, M } given x as
[Equation]
where λ = ˆ p ( o )
ˆ p ( i ) > 0 and M > 1 .
Further, let the model for the marginal density of the in-distribution ˆ p ( x | i ) and out-distribution p ( x | o ) be given by the generalized GMMs
[Equation]
with α k , β l > 0 and µ k , ν l ∈ R d ∀ k = 1 , . . .
K i , l = 1 , . . .
K o and d : R d × R d → R + a metric.
Let z ∈
R d and deﬁne k ∗ =
arg min k =1 ,...,K i d ( z,µ k ) σ k , i ∗
[Equation]
i
[Equation]
1 ,...,n d ( z, x i ) , l ∗ = arg min l =1 ,...,K o β l exp (cid:16) − d ( z,ν l ) 2 2 θ 2 l (cid:17) and ∆ =
θ 2 l ∗ σ 2 k ∗ − 1 .
For any (cid:15) > 0 , if min l θ l > max k σ k and min i =1 ,...,
[Equation]
then it holds for all m ∈ { 1 , . . .
, M } that
[Equation]
In particular [Equation] if min i d ( z, x i ) → ∞ , then ˆ p ( m | z ) → 1 M .
Proof.
The proof essentially hinges on upper bounding ˆ p ( z | i )
ˆ p ( z | o ) using the speciﬁc properties of the
Gaussian mixture model.
We note that
[Equation]
The last step holds because the function g ( ξ )
=
+ Mξ 1+ ξ is monotonically increasing
[Equation]
and
[Equation]
[Equation]
As the second deriviative is negative for ξ ≥ 0 , g is concave for ξ ≥ 0 and thus
[Equation]
[Equation]
In order to achieve the required result we need to show that M − 1 λ ˆ p ( x | i ) ˆ p ( x | o ) ≤ (cid:15) for x sufﬁciently far away from the training data.
We note that
[Equation]
where k ∗ = arg min k
[Equation]
and l
[Equation]
= arg[-min l β l exp( − d ( x,ν l ) 
-]
[Equation]
Using the triangle inequality, d ( x, ν l
[Equation]
) ≤ d ( x, µ k ∗ ) + d ( µ k ∗ , ν l ∗ ) , we get the desired condition as
[Equation]
Thus we get with a = (cid:0) 1 2 σ 2 k ∗ − 12 θ 2 l ∗ (cid:1) , b
= d ( µ k ∗ ,ν l ∗ ) θ 2 l ∗ and c = d ( µ k ∗ ,ν l ∗ ) 2 2 θ 2 l ∗ , d = log (cid:16) (cid:15)λM − 1 β l ∗ (cid:80)
k
α k (cid:17) , the quadratic inequality
[Equation]
where d < 0 for sufﬁciently small (cid:15)
We get the solution
[Equation]
It holds, using √ a + b ≤ √ a + √ b for a, b > 0 , b
[Equation]
a [Equation] (cid:115) max
[Equation]
d [Equation] (cid:27) ≤
[Equation]
a
[Equation]
(cid:114) c a + (cid:114)
[Equation]
One can simplify
[Equation]
Noting that d ( x, µ k ∗ ) ≥ | d ( x, x i ∗ ) − d ( x i ∗ , µ k ∗ ) | we get that
[Equation]
implies M − 1 λ ˆ p ( x | i ) ˆ p ( x | o ) ≤ (cid:15) .
The last statement follows directly by noting that by assumption a >(independently of the choice of l ∗ and k ∗ ) and b, c, d ( x i ∗ , µ k ∗ ) are bounded as K i , K o , n are ﬁnite.
With ∆ =
θ 2 l ∗ σ 2 k ∗ − 1 we can rewrite the required condition as
[Equation]
B A PPENDIX - P ROOF OF C OROLLARY 3.
Corollary 3.1.
Let x 0 ∈ R d and R > 0 , then with λ
=
ˆ p ( o ) ˆ p ( i ) it holds
[Equation]
where b =
[Equation]
Proof.
From the previous section we already know that ˆ p ( y | x ) ≤ 1 M 1+ M bλ 1+ bλ as long as p ( x | i ) p ( x | o ) ≤ b .
Now we can separately bound the numerator and denominator within a ball of radius R around x 0 .
For the numerator we have
[Equation]
where we have lower bounded min d
[Equation]
via the reverse triangle inequality
[Equation]
The denominator can similarly be bounded via
[Equation]
With both of these bounds in place the conclusion immediately follows.
C A PPENDIX - E XPERIMENTAL D ETAILS
Unless speciﬁed otherwise we use ADAM on MNIST with a learning rate of 1 e − 3 and SGD with learning rate 0 .
1 for the other datasets.
The learning rate for the GMM is always set to 1 e − 5 .
We decrease all learning rates by a factor of 10 after 50, 75 and 90 epochs.
Our batch size is 128, the total number of epochs 100 and weight decay is set to 5 e − 4 .
When training ACET, OE and CCU with 80 million tiny images we pick equal batches of in- and out-distribution data (corresponding to p ( i ) = p ( o ) ) and concatenate them into a batches of size 256.
Note that during the 100 epochs only a fraction of the 80 million tiny images are seen and so there is no risk of over-ﬁtting.
C. D ATA A UGMENTATION
Our data augmentation scheme uses random crops with a padding of 2 pixels on MNIST and FMNIST.
On SVHN, CIFAR10 and CIFAR100 the padding width is 4 pixels.
For SVHN we ﬁll the padding with the value at the boundary and for CIFAR we apply reﬂection at the boundary pixels.
On top of this we include random horizontal ﬂips on CIFAR.
For MNIST and FMNIST we generatesuch samples and for SVHN and CIFAR 50000 samples by drawing from the clean dataset without replacement.
This augmented data is used to calculate the covariance matrix from equation 12.
During the actual training we use the same data augmentation scheme in a standard fashion.
C. A TTACK DETAILS
We begin with a step size of 3 and for each of the 50 restarts we randomly initialize at some point in the ellipsoid.
Whenever a gradient step successfully decreases the losses we increase the step size by [Figure] a factor of 1.1.
Whenever the loss increases instead we use backtracking and decrease the step size by a factor of 2.
We apply normal PGD using the l 2 -norm in the transformed space to ensure that we stay on the ellipsoid and after each gradient step we rotate back into the original space to project onto the box [0 , 1] d .
The result is not guaranteed be on the ellipsoid so after the 500 steps we use the alternating projection algorithm (Bauschke & Borwein, 1996) for 10 steps which is guaranteed to converge to a point in the intersection of the ellipsoid and the box because both of these sets are convex.
D A PPENDIX - F INDING A THE CERTIFIABLE RADIUS
Since Corollary 3.1 does not explicitly give a radius, one has to numerically invert the bound.
The bound
[Equation]
(24) is monotonically increasing in R .
Thus, for a given sample x 0 one can ﬁx a desired bound max d ( x,x 0 ) ≤ R ˆ p ( x | i ) ≤ 1 M ν , where ν ∈ (1 , M ) and then ﬁnd the unique solution
[Equation]
(25) for R via bisection.
This radius ˆ R will then represent the maximal radius, that one can certify using
Corollary 3.1.
The presumption is, of course, that for R =
one has a sufﬁciently low bound in the ﬁrst place, i.e. that a solution exists.
In our experiments on uniform noise we did not encounter a single counterexample to this assumption.
We show the radii for the different datasets in Figure 3.
E A PPENDIX - A NALYSIS OF THE CERTIFIED BALLS AROUND UNIFORM NOISE IMAGES
As one can observe in Figure 2 the images which maximize the conﬁdence in the certiﬁed ball around the uniform noise image are sometimes quite far away from the original noise image.
As CCU certiﬁes low conﬁdence (the maximal conﬁdence is less than 1 . 1 × 1 M - so the predicted probability distribution over the classes is very close to the uniform distribution) over the whole ball, it is a natural question what these balls look like and what kind of images they contain.
In particular, it is in general not desired that the certiﬁed balls contain images from the training and test set.
For each dataset we certiﬁed balls around 200 uniform noise images and for each of the certiﬁed balls we check if it contains training or test images of the corresponding dataset.
We found that even though the certiﬁed balls are large, not a single training or test image was contained in any of them.
This justiﬁes the use of our proposed threat model.
A different problem could be that our threshold of 1 . 1 M for the certiﬁcation is too high and that many predictions on the test set have conﬁdence less than this threshold.
For this purpose we report in
Table 3 the smallest predicted conﬁdence of CCU on the test set T , that is
[Equation]
[Table]
for each dataset and the total number of test samples where the conﬁdence is below 1 . 1 M .
While for
MNIST and FMNIST, this never happens, and for SVHN this is negligible (less than 0 .
1% of the test set), for CIFAR10 and CIFAR100 this happens in 5 .
3% resp.
3% of the cases.
In theory, this could impair our AUC value for the detection of adversarial noise.
However, in practice our bound for the conﬁdence is quite conservative as the bound is only tight in very speciﬁc conﬁgurations of the centroids of the Gaussian mixture model which are unlikely to happen for any practical dataset, meaning that the actual maximal conﬁdence in the certiﬁed region is typically signiﬁcantly lower.
In fact the AUC values of CCU are always 100% which means that for allcertiﬁed balls the maximal value of the conﬁdence of CCU in any of these balls (found by our
PGD attack algorithm) is lower than the minimal conﬁdence of all predictions on the test set as reported in Table 3.
On the other hand assuming here also a worst case scenario in the sense we assume that the upper bound of the maximal conﬁdence is attained in all 200 certiﬁed balls, then the (certiﬁed) AUC value would be: 99.92% for SVHN, 94.71% for CIFAR10, and 98.70% for
CIFAR100.
Note that this theoretical lower bound on our performance is still better than all other models’ empirical performance on this task, as reported in Table 1 on both CIFAR10 and CIFAR100, and only marginally below the perfect AUC of GAN on SVHN.
F A PPENDIX - P ERFORMANCE OF ACET WHEN TRAINED ON ADVERSARIAL UNIFORM NOISE
Similar to our CCU, ACET Hein et al.
(2019) requires that one chooses a model for the out-distribution in order to generate their “adversarial noise” during training.
We trained the ACET model with the same out-distribution model as for all other models namely using the tiny image dataset as suggested in Hendrycks et al.
(2019) with a PGD attack that starts at the original point and takes 40 FGSM steps in order to maximize the maximal conﬁdence over the classes.
We use backtracking and halve the step size whenever the loss does not increase.
However, the authors of Hein et al.
(2019) used smoothed uniform noise and a l ∞ -threat model during training.
Since our worst case analysis for
OOD is based on attacking uniform noise images, this suggests that training ACET with uniform noise should improve the performance of ACET for the worst case analysis.
We report below the results of ACET2 (the original model in the paper using tiny images for training is called ACET) based on attacking using uniform noise images during training with a l ∞ -threat model with (cid:15)
=
.as suggested in Hein et al.
We report the normal OOD performance in Table 4 and the worst case analysis of adversarial noise in Table 5.
While it is not surprising that ACET outperforms
ACET2 on the standard OOD detection task in Table 4 as it has seen more realistic “noise” images during training, the worse performance of ACET2 for the worst case analysis in Table 5 is at ﬁrst sight counter-intuitive.
However, note that the threat model of the attacks in our worst case analysis is the Mahalanobis-type l 2 -type metric, see 12, while ACET2 uses an l ∞ -attack model with (cid:15) = 0 .during training.
As the size of the balls for the Mahalanobis-type l 2 -type metric is quite large, there is not much overlap between the two sets.
This explains why ACET2 fails here.
In summary, we have shown that by using tiny images as out-distribution during training, ACET improves in terms of OOD detection performance over ACET2, which is similar to the version suggested in Hein et al.
G A PPENDIX - P RECISION AND R ECALL
In addition to the AUC presented in Table 2 we follow Hendrycks & Gimpel (2017c) and report the area under the precision/recall curve (AUPR).
Precision at a speciﬁc threshold is deﬁned as the
[Table]
ACET ACET
MNIST
TE 0.
SR 0. AUC 100.
FMNIST
TE 4.
SR 0.
AUC 100.
SVHN
TE 3.
SR 3.
AUC 96.
[Table] [Table]
LSUN_CR 99.
Imagenet96.
Noise 89.
Uniform 98.
C
I F
A R
SVHN 92.
CIFAR 86.
LSUN_CR 99.
Imagenet84.
Noise 88.
Uniform 97.
C
I F
A R
SVHN 67.
[Table]
precision
[Equation]
Recall is deﬁned as
[Equation]
(27) where fn is the number of false negatives.
We report the AUPR for all models and all datasets in
Table 6.
Qualitatively we ﬁnd that the results do not differ from the ones reported in Table
[Table]