P ROVABLY R OBUST D ETECTION OF O UT - OFDISTRIBUTION D ATA ( ALMOST ) FOR FREE
Anonymous authors
Paper under double-blind review
A BSTRACT
The application of machine learning in safety-critical systems requires a reliable assessment of uncertainy.
However, deep neural networks are known to produce highly overconﬁdent predictions on out-of-distribution (OOD) data.
Even if trained to be non-conﬁdent on OOD data one can still adversarially manipulate OOD data so that the classiﬁer again assigns high conﬁdence to the manipulated samples.
In this paper we propose a novel method that combines a certiﬁable OOD detector with a standard classiﬁer from ﬁrst principles into an OOD aware classiﬁer.
This way we achieve the best of two worlds: certiﬁably adversarially robust OOD detection, even for OOD samples close to the in-distribution, without loss in either prediction accuracy or detection performance for non-manipulated OOD data.
Moreover, due to the particular construction our classiﬁer provably avoids the asymptotic overconﬁdence problem of standard neural networks.
I NTRODUCTION
Deep neural networks have achieved state-of-the-art performance in many application domains.
However, the widespread usage of deep neural networks in safety-critical applications, e.g. in healthcare, autonomous driving/aviation, manufacturing, raises concerns as deep neural networks have problematic deﬁciencies.
Among these deﬁciencies are overconﬁdent predictions on non-task related inputs (Nguyen et al., 2015; Hendrycks & Gimpel, 2017)which has recently attracted a lot of interest.
ReLU networks have even been shown to be provably overconﬁdent far away from the training data (Hein et al., 2019).
However, reliable conﬁdences of the classiﬁer on the classiﬁcation task (in-distribution) (Guo et al., 2017) as well as on the out-distribution (Hendrycks & Gimpel, 2017; Hein et al., 2019) are important to be able to detect when the deep neural network is working outside of its speciﬁcation, which can then be used to either involve a human operator or to fall back into a “safe state”.
Thus, solving this problem is of high importance for trustworthy ML systems.
Many approaches have been proposed for OOD detection, (Hendrycks & Gimpel, 2017; Liang et al., 2018; Lee et al., 2018a;b; Hendrycks et al., 2019;
Ren et al., 2019; Hein et al., 2019; Meinke &
Hein, 2020; Chen et al., 2020; Papadopoulos et al., 2021; Macêdo & Ludermir, 2021; Macêdo et al., 2021) and one of the currently best performing methods enforces low conﬁdence during training (“outlier exposure” (OE)) on a large and diverse set of out-distribution images (Hendrycks et al., 2019) which leads to strong separation of in- and out-distribution based on the conﬁdence of the classiﬁer.
Crucially, this also generalizes to novel test out-distributions.
However, current OOD detection methods are vulnerable to adversarial manipulations, i.e. small adversarial modiﬁcations of OOD inputs lead to large conﬁdence of the classiﬁer on the manipulated samples (Nguyen et al., 2015; Hein et al., 2019; Sehwag et al., 2019).
While different methods for adversarially robust OOD detection have been proposed (Hein et al., 2019; Sehwag et al., 2019;
Meinke & Hein, 2020; Chen et al., 2020; Bitterwolf et al., 2020)
there is little work on provably adversarially robust OOD detection (Meinke & Hein, 2020; Bitterwolf et al., 2020; Kopetzki et al., 2020; Berrada et al., 2021).
In CCU (Meinke & Hein, 2020) they append density estimators based on Gaussian mixture models for in- and out-distribution to the softmax layer.
By also enforcing low conﬁdence on a training out-distribution, they achieve similar OOD detection performance to (Hendrycks et al., 2019) but can guarantee that the classiﬁer shows decreasing conﬁdence as one moves away from the training data.
However, for close in-distribution inputs this approach yields no guarantee as the Gaussian mixture models are not powerful enough for complex image classiﬁcation
[Table]
gap even
Interval bound propagation ( et al., et al., 2018; Zhang et al et al
2021) has been shown to be one of the most effective techniques in certiﬁed adversarial robustness on the in distribution when applied during training.
In GOOD (Bitterwolf et al., 2020) they use IBP to compute upper bounds on the conﬁdence in an l ∞ -neighborhood of the input and minimize these upper bounds on a training out-distribution.
This yields classiﬁers with pointwise guarantees for adversarially robust OOD detection even for “close” out-distribution inputs which generalize to novel OOD test distributions.
However, the employed architectures of the neural network are restricted to rather shallow networks as otherwise the bounds of IBP are loose.
Thus, they obtain low classiﬁcation accuracy which is far from the state-of-the-art, e.g. 91% on CIFAR-10, and their approach does not scale to more complex tasks like ImageNet.
In particular, despite its low accuracy the employed network architecture is quite large and has higher memory consumption than a ResNet50.
Moreover, one does not get any guarantees on the asymptotic behavior of the classiﬁer as one moves away from the training data.
The authors of (Berrada et al., 2021) use SOTA veriﬁcation techniques (Dathathri et al., 2020) and get guarantees for OOD detection wrt.
l ∞ -perturbations for ACET models (Hein et al., 2019) that were not speciﬁcally trained to be veriﬁable but the guarantees obtained by training the models via IBP in (Bitterwolf et al., 2020) are signiﬁcantly better.
In this paper we propose ProoD which merges a certiﬁed binary discriminator for in-versus outdistribution with a classiﬁer for the in-distribution task in a principled fashion into a joint classiﬁer.
This combines the advantages of CCU (Meinke & Hein, 2020) and GOOD (Bitterwolf et al., 2020) without suffering from their downsides.
In particular, ProoD simultaneously achieves the following:
• Guaranteed adversarially robust OOD detection via conﬁdence upper bounds on l ∞ -balls around OOD samples.
• Additionally, it provably prevents the asymptotic overconﬁdence of deep neural networks.
• It can be used with arbitrary architectures and has no loss in prediction performance and standard OOD detection performance.
Thus, we get provable guarantees for adversarially robust OOD detection, ﬁx the asymptotic overconﬁdence (almost) for free as we have (almost) no loss in prediction and standard OOD detection performance.
We qualitatively compare the properties of our model to prior approaches in Table 1.
P ROVABLY R OBUST D ETECTION OF O UT - OF - DISTRIBUTION D ATA
In the following we consider feedforward networks for classiﬁcation, f : R d → R K , with K classes deﬁned with x (0) =
x as
[Equation]
[Figure]
where L ∈ N is the number of layers, W ( l ) and b ( l ) are weights and biases and σ ( l ) is either the
ReLU or leaky ReLU activation function of layer l .
We refer to the output of f as the logits and get a probability distribution over the classes via ˆ p ( y | x ) = [Equation]
[Figure] [Equation]
Kk [Equation] ( x ) for y = 1 , . . .
, K .
We deﬁne the conﬁdence as Conf( f ( x ))
=
max y
=1 ,...,K ˆ p ( y | x ) .
2. J OINT M ODEL FOR OOD D ETECTION AND C LASSIFICATION
In our joint model we assume that there exists an in- and out-distribution where the out-distribution samples are unrelated to the in-distribution task.
Thus, we can formally write the conditional distribution on the input as
[Equation]
where ˆ p ( i | x ) is the conditional distribution that sample x belongs to the in-distribution and ˆ p ( y | x, i ) is the conditional distribution for the in-distribution.
We assume that OOD samples are unrelated and thus maximally un-informative to the in-distribution task, i.e. we ﬁx ˆ p ( y | x, o )
= 1 K .
In (Meinke & Hein, 2020) they further decomposed ˆ p ( i | x ) =
ˆ p ( x | i )ˆ p ( i ) ˆ p ( x ) and used Gaussian mixture models to estimate ˆ p ( x | i ) with ﬁxed ˆ p ( i ) =
ˆ p ( o ) = 12 .
As the properties of the Gaussian mixture model can be directly controlled in comparison to a more powerful generative model such as VAEs, this allowed (Meinke & Hein, 2020) to prove that ˆ p ( y | x ) becomes uniform over the classes if x is far away from the training data.
However, the downside of the Gaussian mixture models is that this approach yields no guarantees for close out-distribution samples.
Instead in this paper we directly learn ˆ p ( i | x ) which results in a binary classiﬁcation problem and we train this binary classiﬁer in a certiﬁed robust fashion wrt.
an l ∞ -threat model so that even adversarially manipulated OOD samples are detected.
In order to avoid confusion with the multi-class classiﬁer, we will refer to ˆ p ( i | x ) as a binary discriminator.
In an l ∞ -ball of radius (cid:15) around x ∈ R d and for all y we have the upper bound (also see App. H)
[Equation]
so we can defer the certiﬁcation “work” to the binary discriminator.
Using a particular constraint on the weights of the binary discriminator, we get similar asymptotic properties as in (Meinke &
Hein, 2020) but additionally get certiﬁed adversarial robustness for close out-distribution samples as in (Bitterwolf et al., 2020).
In contrast to (Bitterwolf et al., 2020) this comes without loss in test accuracy or non-adversarial OOD detection performance as in our model the neural network used for the in-distribution classiﬁcation task ˆ p ( y | x, i ) is independent of the binary discriminator.
Thus, we have the advantage that the classiﬁer can use arbitrary deep neural networks and is not constrained to certiﬁable networks.
We call our approach Pr ovable o ut-
o f- D istribution detector (ProoD) and visualize its components in Figure 1.
Certiﬁably Robust Binary Discrimination of In- versus Out-Distribution The ﬁrst goal is to get a certiﬁably adversarially robust OOD detector ˆ p ( i | x ) .
We train this binary discriminator independent of the overall classiﬁer as the training schedules for certiﬁed robustness are incompatible with the standard training schedules of normal classiﬁers.
For this binary classiﬁcation problem we use a logistic model ˆ p ( i | x ) = 1 1+ e − g ( x ) , where g : R d → R are logits of a neural network (we denote the weights and biases of g by W g and b g in order to discriminate it from the classiﬁer f introduced in the next paragraph).
Let ( x r , y r )
Nr =1 be our in-distribution training data (we use the class encoding +1 for the in-distribution and − 1 for out-distribution) and ( z s )
Ms =1 be our training out-distribution data.
Then the optimization problem associated to the binary classiﬁcation problem becomes:
[Equation]
where we minimize over the parameters of the neural network g under the constraint that the weights of the output layer W ( L g ) g are componentwise negative and ¯ g ( z ) ≥ max u ∈ B p ( z,(cid:15) )
g ( u ) is an upper bound on the output of g around OOD samples for a given l p -threat model B p ( z, (cid:15) )
= { u ∈ [0 , 1] d | (cid:107) u − z (cid:107) p ≤ (cid:15) } .
In this paper we always use an l ∞ -threat model.
This upper bound could, in principle, be computed using any certiﬁcation technique but we will use interval bound propagation (IBP) since it is simple, fast and has been shown to produce SOTA results (Gowal et al., 2018).
Note that this is not standard adversarial training for a binary classiﬁcation problem as here we have an asymmetric situation: we want to be (certiﬁably) robust to adversarial manipulation on the out-distribution data but not on the in-distribution and thus the upper bound is only used for out-distribution samples.
The negativity of the output layer’s weights W ( L g ) g is enforced by using the parameterization ( W ( L g ) g )
[Equation] j componentwise and optimizing over h j .
In Section 3 we show how the negativity of W ( L g ) g allows us to control the asymptotic behavior of the joint classiﬁer.
For the reader’s convenience we quickly present the upper x ( l ) and lower x ( l ) bounds on the output of layer l in a feedforward neural network produced by IBP:
[Equation]
where W + = max(0 , W ) and W − = min(0 , W ) (min/max used componentwise).
For an l ∞ -threat model one starts with the upper and lower bounds for the input layer x
=
x + (cid:15) and x (0) =
x − (cid:15) and then iteratively computes the layerwise upper and lower bounds x ( l ) , x ( l ) which fulﬁll
[Equation]
While in (Bitterwolf et al., 2020) they also used IBP to upper bound the conﬁdence of the classiﬁer this resulted in a bound that took into account
all O
( K 2 ) logit differences between all classes.
In contrast, our loss in Eq.
(4) is signiﬁcantly simpler as we just have a binary classiﬁcation problem and therefore only need a single bound.
Thus, our approach easily scales to tasks with a large number of classes and training the binary discriminator with IBP turns out to be signiﬁcantly more stable than the approach in (Bitterwolf et al., 2020) and does not require many additional tricks.
(Semi)-Joint Training of the ﬁnal Classiﬁer Given the certiﬁably robust model ˆ p ( i | x ) for the binary classiﬁcation task between in- and out-distribution, we need to determine the ﬁnal predictive distribution ˆ p ( y | x ) in Eq.
On top of the provable OOD performance that we get from Eq.
(3), we also want to achieve SOTA performance on unperturbed OOD data.
In principle we could independently train a model for the predictive in-distribution task ˆ p ( y | x, i ) , e.g. using outlier exposure (OE) (Hendrycks et al., 2019) or any other state-of-the-art OOD detection method and simply combine it with our ˆ p ( i | x ) .
While this does lead to models with high OOD performance that also have guarantees, it completely ignores the interaction between ˆ p ( i | x ) and ˆ p ( y | x, i ) during training.
Instead we propose to train ˆ p ( y | x, i ) by optimizing our ﬁnal predictive distribution ˆ p ( y | x ) .
Note that in order to retain the guarantees of ˆ p ( i | x ) we only train the parameters of the neural network f : R d → R K and need to keep ˆ p ( i | x ) resp.
g ﬁxed.
Because g stays ﬁxed we call this semi-joint training.
We use OE (Hendrycks et al., 2019) for training ˆ p ( y | x ) with the cross-entropy loss and use the softmax-function in order to obtain the predictive distribution ˆ p [Equation]
( y | x
[Equation]
=
[Equation]
f :
[Equation]
where the ﬁrst term is the standard cross-entropy loss on the in-distribution but now for our joint model for ˆ p ( y | x ) and the second term enforces uniform conﬁdence on out-distribution samples.
In App.
B we show that semi-joint training leads to stronger guarantees than separate training.
The loss in Eq.
(4) implicitly weighs the in-distribution and worst-case out-distribution equally, which amounts to the assumption p ( i ) = 12 = p ( o ) .
This highly conservative choice simpliﬁes training the binary discriminator but may not reﬂect the expected frequency of OOD samples at test time and in effect means that ˆ p ( i | x ) tends to be quite low.
This typically yields good guaranteed AUCs but can have a negative impact on the standard out-distribution performance.
In order to better explore the trade-off of guaranteed and standard OOD detection, we repeat the above semi-joint training with different shifts of the offset parameter in the output layer
[Equation]
where ∆ ≥ 0 leads to increasing ˆ p ( i | x ) .
This shift has a direct interpretation in terms of the probabilities p ( i ) and p ( o ) .
Under the assumption that our binary discriminator g is perfect, that is p ( i | x
[Equation]
then it holds that e g ( x ) =
p ( x | i ) p
( i ) p ( x | o )
p ( o ) .
A change of the prior probabilities ˜ p
( i ) and ˜ p ( o ) without changing p ( x | i ) and p ( x | o ) then corresponds to a novel classiﬁer e ˜
[Equation]
with ∆ = log
[Equation]
Note that ˜ p ( i ) > p ( i ) corresponds to positive shifts.
In a practical setting, this parameter can be chosen based on the priors for the particular application.
Since no such priors are available in our case we determine a suitable shift by evaluating on the training out-distribution, see Section 4.2 for details.
Please note that we explicitly do not train the shift parameter since this way the guarantees would get lost as the classiﬁer implicitly learns a large ∆ in order to maximize the conﬁdence on the in-distribution, thus converging to a normal outlier exposure-type classiﬁer without any guarantees.
G UARANTEES ON A SYMPTOTIC C ONFIDENCE
In this section we show that our speciﬁc construction provably avoids the issue of asymptotic overconﬁdence that was pointed out in (Hein et al., 2019).
Note that the resulting guarantee (as stated in Theorem 1) is different from and in addition to the robustness guarantees discussed in the previous section (see Eq.
The previous section dealt with providing conﬁdence upper bounds on neighborhoods around OOD samples whereas this section deals with ensuring that a classiﬁer’s conﬁdence decreases asymptotically as one moves away from all training data.
We note that a ReLU neural network f : R d → R K as deﬁned in Eq.
(1) using ReLU or leaky ReLU as activation functions, potential max-or average pooling and skip connection yields a piece-wise afﬁne function (Arora et al., 2018;
Hein et al., 2019), i.e. there exists a ﬁnite set of polytopes Q r ⊂ R d with r = 1 , . . .
, R such that ∪ Rr =1 Q r = R d and f restricted to each of the polytopes is an afﬁne function.
Since there are only ﬁnitely many polytopes some of them have to extend to inﬁnity and on these ones the neural network is essentially an afﬁne classiﬁer.
This fact has been used in (Hein et al., 2019) to show that ReLU networks are almost always asymptotically overconﬁdent in the sense that if one moves to inﬁnity the conﬁdence of the classiﬁer approaches 1 (instead of converging to 1 /K as in these regions the classiﬁer has never seen any data).
The following result taken from (Hein et al., 2019) basically says that as one moves to inﬁnity by upscaling a vector one eventually ends up in a polytope which extends to inﬁnity.
Lemma 1 ((Hein et al., 2019)) .
Let { Q r } Rr =1 be the set of convex polytopes on which a ReLUnetwork f : R d → R K is an afﬁne function, that is for every k ∈ { 1 , . . .
, R } and x ∈ Q k there exists
V k ∈ R K × d and c k ∈ R K such that f ( x ) =
V k x + c k .
For any x ∈ R d with x (cid:54) =
there exists α ∈ R and t ∈ { 1 , . . .
, R } such that βx ∈ Q t for all β ≥ α .
The following theorem now shows that, opposite to standard ReLU networks (Hein et al., 2019), our proposed joint classiﬁer gets provably less conﬁdent in its decisions as one moves away from the training data which is a desired property of any reasonable classiﬁer.
Theorem 1.
Let x ∈ R d with x (cid:54) = 0 and let g : R d
→ R be the ReLU-network of the binary discriminator and denote by { Q r } Rr =1 the ﬁnite set of polytopes such that g is afﬁne on these polytopes which exists by Lemma 1.
Denote by Q t the polytope such that βx ∈ Q t for all β ≥ α and let x ( L − 1)
( z ) =
Uz + d with U ∈ R n L − 1 × d and d ∈ R n L − 1 be the output of the pre-logit layer of g for z ∈ Q t .
If Ux (cid:54) = 0 , then
[Equation]
The proof is in App. C.
In App.
A we see that the condition Ux (cid:54) = 0 is not restrictive, as this property holds in all cases where we checked it for our joint classiﬁer.
The negativity condition on the weights W ( L g ) g of the output layer of the in-vs. out-distribution discriminator g is crucial for the proof.
This condition may seem restrictive, but we did not encounter any negative inﬂuence of this constraint on test accuracy, guaranteed or standard OOD detection performance.
Thus the asymptotic guarantees come essentially for free.
In (Meinke & Hein, 2020) they derived non-asymptotic guarantees and it would be relatively easy to also achieve this for the joint classiﬁer via a decay factor for ˆ p ( i | x ) that depends on the distance to the training data but we prefer not to enforce this explicitly.
E XPERIMENTS 4. T RAINING OF P ROO D
We provide experiments on CIFAR10, CIFAR100 (Krizhevsky & Hinton, 2009) and Restricted Imagenet (R.ImgNet) (Tsipras et al., 2018).
The latter consists of images from the ILSVRC subset of ImageNet (Deng et al., 2009; Russakovsky et al., 2015) belonging to 9 types of animals.
Training the Binary Discriminator We train the binary discriminator between in-and outdistribution using the loss in Eq.
(4) with the bounds over an l ∞ -ball of radius (cid:15) = 0 .
01 for the out-distribution following (Bitterwolf et al., 2020).
We use relatively shallow CNNs with only 5 layers plus pooling layers, see App.
D. For the training out-distribution, we could follow previous work and use 80M Tiny Images (Torralba et al., 2008) for CIFAR10 and CIFAR100.
However, there have been concerns over the use of this dataset (Birhane & Prabhu, 2021) because of offensive class labels.
Although we do not use any of the class labels, we choose to use OpenImages (Kuznetsova et al., 2020) as training OOD instead.
In order to ensure a fair comparison with prior work we also present results that were obtained using 80M Tiny Images in App.
E.
For R.ImgNet we use the ILSVRC train images that do not belong to R.ImgNet as training out-distribution (NotR.ImgNet).
Semi-Joint Training For the classiﬁer we use a ResNet18 architecture on CIFAR and a ResNet on R.ImgNet.
Note that the architecture of our binary discriminator is over an order of magnitude smaller than the one in (Bitterwolf et al., 2020) (11MB instead of 135MB) and thus the memory overhead for the binary discriminator is less than a third of that of the classiﬁer.
All schedules, hardware and hyperparameters are described in App.
D.
As discussed in Section 2.1 when training the binary discriminator one implicitly assumes that in- and (worst-case) out-distribution samples are equally likely.
It seems very unlikely that one would be presented with such a large number of OOD samples in practice but as discussed in Section 2.1, we can adjust the weight of the losses after training [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure] [Figure] the discriminator (but before training the classiﬁer) by shifting the bias b ( L g ) g in the output layer of the binary discriminator. We train several ProoD models for binary shifts in { 0 , 1 , 2 , 3 , 4 , 5 , 6 } and then evaluate the AUC and guaranteed AUC (see 4.2) on a subset of the training out-distribution
OpenImages (resp. NotR.ImgNet).
For all bias shifts we use the same ﬁxed provably trained binary discriminator and only train the classiﬁer part.
As our goal is to have provable guarantees with minimal or no loss on the standard OOD detection task, among all solutions which have better AUC than outlier exposure (OE) (Hendrycks et al., 2019) we choose the one with the highest guaranteed
AUC on OpenImages (on CIFAR10/CIFAR100) resp.
NotR.ImgNet (on R.ImgNet).
If none of the solutions has better AUC than OE on the training out-distribution we take the one with the highest AUC.
We show the trade-off curves for the example of CIFAR10 in Figure 2.
The corresponding ﬁgures for CIFAR100 and R.ImgNet can be found in App.
D.
4. E VALUATION
Setup For OOD evaluation for CIFAR10/100 we use the test sets from CIFAR100/10, SVHN (Netzer et al., 2011), the classroom category of downscaled LSUN (Yu et al., 2015) (LSUN_CR) as well as smooth noise as suggested in (Hein et al., 2019) and described in App.
D. For R.ImgNet we use
Flowers (Nilsback & Zisserman, 2008), FGVC Aircraft (Maji et al., 2013), Stanford Cars (Krause et al., 2013) and smooth noise as test out-distributions.
Since the computation of adversarial AUCs (next paragraph) requires computationally expensive adversarial attacks, we restrict the evaluation on the out-distribution to a ﬁxed subset of 1000 images (300 in the case of LSUN_CR) for the CIFAR experiments and 400 for the R.ImgNet models.
We still use the entire test set for the in-distribution.
Guaranteed and Adversarial AUC We use the conﬁdence of the classiﬁer as the feature to discriminate between in- and out-distribution samples.
While in standard OOD detection one uses the area under the receiver-operator characteristic (AUC) to measure discrimination of in- from out-distribution, Bitterwolf et al. (2020) introduced the worst-case AUC (WCAUC) which is deﬁned as the minimal AUC one can achieve if all out-distribution samples are allowed to be perturbed to reach maximal conﬁdence within a certain threat model, which in our case is an l ∞ -ball of radius (cid:15) .
The AUC and WAUC of a feature h : R d → R is deﬁned as:
[Equation]
where p 1 , p 2 are in-resp.
out-distribution and with slight abuse of notation the indicator functionreturns 1 if the expression in its argument is true and 0 otherwise. For all but one of our baselines, the
OOD detecting feature h is the conﬁdence of the classiﬁer.
Since the exact evaluation of the WCAUC is computationally infeasible, we compute an upper bound and lower bound on the WCAUC by ﬁnding h ( z ) ≤ max [Equation] h [Equation]
z [Equation] ) ≤ ¯ h ( z ) . We ﬁnd the upper bound on the WCAUC, the adversarial
AUC (AAUC), by maximizing the conﬁdence using an adversarial attack inside the l ∞ -ball (i.e. ﬁnding an h ) and we compute a lower bound on the WCAUC, the guaranteed AUC (GAUC), by using upper on the conﬁdence inside the l ∞ -ball via IBP (i.e. ¯ h ).
For non-provable methods, no non-trivial upper bound ¯ h < ∞ is available so their GAUCs are always 0 .
Note that our threat model is different from adversarial robustness on the in-distribution which neither our method nor the baselines pursue.
Gradient obfuscation (Papernot et al., 2017; Athalye et al., 2018) poses a signiﬁcant challenge for the evaluation of AAUCs (Bitterwolf et al., 2020)
so we employ an ensemble of different versions of projected gradient descent (PGD) (Madry et al., 2018) as well as SquareAttack (Andriushchenko et al., 2020) with 5000 queries.
We use APGD (Croce & Hein, 2020) (except on RImgNet, due to a memory leak) with 500 iterations and 5 random restarts.
We also use a 200-step PGD attack with momentum of 0.9 and backtracking that starts with a step size of 0.1 which is halved every time a gradient step does not increase the conﬁdence and gets multiplied by 1.1 otherwise.
This
PGD is applied to different starting points: i) a decontrasted version of the image, i.e. the point that minimizes the l ∞ -distance to the grey image 0 .
5 · (cid:126) 1 within the threat model, ii) 3 uniformly drawn samples from the threat model and iii) 3 versions of the original image perturbed by Gaussian noise with σ = 10 − 4 and then clipped to the threat model.
We always clip to the box [0 , 1] d at each step of the attack.
For all attacks and all models we directly optimize the ﬁnal score that is used for OOD detection.
Using different types of starting points is crucial for strong attacks on these OOD points, as some models have precisely zero gradient on many OOD samples.
Baselines We compare to a normally trained baseline (Plain) and outlier exposure (OE), both trained using the same architecture and hyperparameters as the classiﬁer in ProoD. For both ATOM and ACET we found the models’ OOD detection to be much less adversarially robust than claimed in (Chen et al., 2020)
(see App.
E)
so we retrained their models using the same architecture, threat model and training out-distribution with their original code (for CIFAR10/100).
Running these adversarial training procedures on ImageNet resolution is infeasibly expensive.
For GOOD we also retrain using OpenImages as training OOD with the code from (Bitterwolf et al., 2020) (comparisons with their pre-trained models can be found in App. E).
Since they are only available on CIFAR10, we attempted to train models on CIFAR100 using their code and the same hyperparameters and schedules as they used for CIFAR10.
This only lead to models with accuracy below 25% , so we do not include these models in our evaluation.
Since CCU was already shown to not provide beneﬁts over OE on OOD data that is not very far from the in-distribution (e.g. uniform noise) (Meinke & Hein, 2020;
Bitterwolf et al., 2020)
we do not include it as a baseline.
We also evaluate the OOD-performance of the provable binary discriminator (ProoD-Disc) that we trained for ProoD. Note that this is not a classiﬁer and so it is included simply for reference.
All results are shown in Table 2.
Results ProoD achieves non-trivial GAUCs on all datasets.
As was also observed in (Bitterwolf et al., 2020), this shows that the IBP guarantees not only generalize to unseen samples but even to unseen distributions.
In general the gap between our GAUCs and AAUCs is extremely small.
This shows that the seemingly simple IBP bounds can be remarkably tight, as has been observed in other works (Gowal et al., 2018; Jovanovi ´ c et al., 2021).
It also shows that there would be very little beneﬁt in applying stronger veriﬁcation techniques like (Cheng et al., 2017;
Katz et al., 2017; Dathathri et al., 2020) in ProoD.
The bounds are also much tighter than for GOOD, which is likely due to the fact that for GOOD the conﬁdence is much harder to optimize during an attack because it involves maximizing the conﬁdence in an essentially random class.
For CIFAR10, on 3 out of 4 out-distributions ProoD’s GAUCs are higher than ATOM’s and ACET’s
AUUCs, i.e. our model’s provable adversarial robustness exceeds the SOTA methods’ empirical adversarial robustness in these cases.
Note that this is not due to our retraining, because the authors’ pre-trained models perform even more poorly (as shown in App. E).
On CIFAR100 ProoD’s guarantees are weaker and ATOM produces strong AAUCs.
However, we observe that training both ACET and ATOM can produce inconsistent results, i.e. sometimes almost no robustness is achieved.
For the successfully trained robust ATOM model on CIFAR100 we observe drastically reduced accuracy.
Due to the difﬁculty in attacking these models, it is not unlikely that a more sophisticated attack could produce even lower AAUCs.
Combined with the fact that both ACET and ATOM rely on expensive adversarial training procedures we argue that using ProoD is preferable in practice.
On CIFAR10 we see that ProoD’s GAUCs are comparable to, if slightly worse than the ones of both
GOOD 80 and GOOD 100 .
Note that although the presented GOOD models are retrained, the same observations hold true when comparing to the pre-trained models (see App. E).
However, we want to point out that ProoD achieves this while retaining both high accuracy and OOD performance, both of which are lacking for GOOD.
It is also noteworthy that the GOOD models’ memory footprints
[Table]
are over twice as large as ProoD’s.
Generally, for ProoD the accuracy is comparable to OE and the
OOD performance is similar or marginally worse.
Thus ProoD shows that it is possible to achieve certiﬁable adversarial robustness on the out-distribution while keeping very good prediction and
OOD detection performance.
Note that all methods struggle on separating CIFAR10 and CIFAR when using OpenImages as training OOD (as compared to 80M Tiny Images in App. E).
To the best of our knowledge with R.ImgNet we provide the ﬁrst worst case OOD guarantees on high-resolution images.
The GAUCs are higher than on CIFAR which indicates that meaningful certiﬁcation on higher resolution is more achievable on this task than one might expect. FGVC and
Cars may seem simple to separate from the animals in R.ImgNet but the same cannot be said for Flowers which are difﬁcult to provably distinguish from images of insects with ﬂowers in them.
C ONCLUSION
We have demonstrated how to combine a provably robust binary discriminator between in- and out-distribution with a standard classiﬁer in order to simultaneously achieve high accuracy, high OOD detection performance as well as worst-case OOD guarantees that are comparable to previous works. Thus, we have combined the best properties of previous work with only a small increase in total model size and only a single hyperparameter.
This suggests that certiﬁable adversarial robustness on the out-distribution (as opposed to the in-distribution) is indeed possible without losing accuracy.
We further showed how in our model simply enforcing negativity in the ﬁnal weights of the discriminator ﬁxes the problem of asymptotic overconﬁdence in ReLU classiﬁers.
Training ProoD models is simple and stable and thus ProoD provides OOD guarantees that come (almost) for free.
R EPRODUCIBILITY
All details necessary for reproducing our experiments are given in Section 4 and App.
D, including hyperparameters and hyperparameter selection.
In order to further aid reproducibility we include the source code in the supplemental material (github link in ﬁnal version), together with instructions for how to use it.
We will also provide pre-trained models for public download in the ﬁnal version.
E THICS
The only potential ethical concern that we see in this work is our use of the retracted dataset 80M Tiny
Images (Torralba et al., 2008).
However, we clearly explain that the only reason we use the dataset is providing a fair comparison to prior work.
We ﬁrmly believe that our work helps the community move away from the use of this dataset because we provide extensive experiments on a different training out-distribution, going as far as retraining several previous baselines on it
A A DVERSARIAL A SYMPTOTIC O VERCONFIDENCE
According to the authors of (Hein et al., 2019), under mild conditions, we should expect to ﬁnd asymptotic overconﬁdence in all ReLU networks and almost all directions.
In order to empirically evaluate this, we take different models that were trained on CIFAR10 and evaluate their conﬁdence on different CIFAR100 samples.
For each sample x we track the conﬁdence, max k
ˆ p ( k | x ) , along a trajectory in a uniform noise direction x + αn , where n ∈ [ − 0 .
d and α ≥ 0 .
The mean conﬁdence across 100 such trajectories is shown on the left side of Figure 3.
Even the models that produce low conﬁdences on the original OOD sample asympotically converge to maximal conﬁdence far away.
The only exceptions here are GOOD and ProoD and only ProoD can guarantee that the conﬁdence cannot converge to 1 .
However, even though the architecture provably prevents arbitrarily overconﬁdent predictions and
Theorem 1 ensures that most directions will indeed converge to uniform, it is, in principle, possible to ﬁnd directions where the conﬁdence ˆ p ( i | x ) remains constant
if the condition Ux (cid:54) = 0 in Theoremis not satisﬁed.
We attempted to ﬁnd such directions by running the following type of attack.
We start from a random point x ∈ [ − 0 .
5] d that we project onto a sphere of radius 100 .
We now run gradient descent (for 20000 steps), maximizing g ( x ) while projecting onto the sphere at each step (unnormalized gradients with step size 0 .
1 for the ﬁrst 10000 steps and 0 .
01 for the last 10000 steps).
We then increase the radius to 1000 and run an additional 20000 steps with step size 0 .
We rescale the resulting direction vector down to an l ∞ -ball of norm 1 and compute the conﬁdence ˆ p ( i | x ) as a function of the scaling in the adversarial directions.
We show the resulting scale-wise maximum over 100 adversarial directions in Figure 3.
Note that even the worst-case over 100 adversarially found directions decays to 0 asymptotically, thus empirically conﬁrming the practical utility of Theorem 1.
Note that the value of ˆ p ( i | x ) converging to 0 implies that the conﬁdence of the ProoD model ˆ p ( y | x ) converges to 10% .
In Figure 3 GOOD also stands out as having low conﬁdence in all directions that we studied.
This is because in all the asymptotic regions that we looked at, the pre-activations of the penultimate layer are all negative.
If one moves outward and these pre-activations only get more negative in all directions far away from the data, the conﬁdence does, in fact, remain low.
Unfortunately, it also leads to gradients that are precisely zero, which is why the same attack can not be applied here.
However, there is no guarantee that GOOD does not also get in some direction asymptotically overconﬁdent.
[Table]
B S EPARATE T RAINING FOR P ROO D
In Section 2.1 we describe semi-joint training of ˆ p ( y | x ) .
However, as pointed out in that section, it is possible to separately train a certiﬁable binary discriminator ˆ p ( i | x ) and an OOD aware classiﬁer ˆ p ( y | x, i ) and to then simply combine them via Eq.
We call this method separate training ProoD-S and evaluate it by using an OE trained model for ˆ p ( y | x, i ) .
We show the results in Table 4, where we repeat the results for OE and ProoD for the reader’s convenience.
Note that OE and ProoD-S must always have the same accuracy on the in-distribution since they use the same model for classiﬁcation (note that (2) preserves the ranking of ˆ p ( y | x, i ) ).
We see that the AUCs of ProoD-S are almost identical to those of OE.
Without almost any loss in performance ProoD-S manages to provide non-trivial GAUCs.
However, as one would expect, the semi-jointly trained ProoD provides stronger guarantees at similar clean performance.
Nonetheless, this post-hoc method of adding some amount of certiﬁability to an existing system may be interesting in applications where retraining a deployed model from scratch is infeasible.
to Table 2 showing the AUCs, GAUCs and AAUCs of
ProoD-S on all datasets.
The accuracy must always be identical to that of OE and the clean AUCs are also very similar to those of OE.
The guarantees are strictly weaker than those provided by the semi-jointly trained ProoD.
In: CIFAR
CIFAR
SVHN
LSUN_CR
Smooth
Acc
[Table]
AAUC
OE 94.91 91.1 0.
ProoD-S ∆=3 94.91 89.3 44.
ProoD ∆= 94.99 89.8 46.
In: CIFAR
CIFAR
[Table]
Smooth
[Table]
OE 77.25 77.4 0.
ProoD-S ∆=5 77.25 77.4 17.
ProoD ∆= 77.16 76.6 17.
In: R.ImgNet
Flowers
[Table]
Cars
Smooth
[Table]
OE 97.10 96.9 0.
ProoD-S ∆=4 97.10 96.9 50.
ProoD ∆= 97.25 96.9 57.
C P ROOF OF T HEOREM
Theorem 1.
Let x ∈ R d with x (cid:54) = 0 and let g : R d
→ R be the ReLU-network of the binary discriminator and denote by { Q r } R r =1 the ﬁnite set of polytopes such that g is afﬁne on these polytopes which exists by Lemma 1.
Denote by Q t the polytope such that βx ∈ Q t for all β ≥ α and let x ( L − 1)
( z ) =
Uz + d with U ∈ R n L − 1 × d and d ∈ R n L − 1 be the output of the pre-logit layer of g for z ∈ Q t .
If Ux (cid:54) = 0 , then
[Equation]
Proof.
We note that with a similar argument as in the derivation of (3) it holds
[Equation]
For all β ≥ α it holds that βx ∈ Q t so that
[Equation]
As x ( L − 1) i ( x ) ≥ 0 for all x ∈ R
d it has to hold ( βUx + d ) i ≥ 0 for all β ≥
α and i = 1 , . .
. , n L − 1 .
This implies that ( Ux ) i ≥ 0 for all i = 1 , . . .
, n L − 1 and since Ux (cid:54) =
there has to exist at least one component i ∗ such that ( Ux ) i ∗ > 0 .
Moreover, W ( L g ) g has strictly negative components and thus for all β ≥ α it holds
[Equation]
As (cid:68) W ( L g ) g , Ux (cid:69) <
we get lim β →∞ g ( x ) =
−∞ and thus lim β →∞ ˆ p ( i | βx ) = 0 .
Plugging this into (12) yields the result.
D E XPERIMENTAL D ETAILS
Datasets We use CIFAR10 and CIFAR100 (Krizhevsky & Hinton, 2009) (MIT license),
SVHN (Netzer et al., 2011) (free for non-commercial use), LSUN (Yu et al., 2015) (no license), the ILSVRC2012 split of ImageNet (Deng et al., 2009; Russakovsky et al., 2015) (free for noncommercial use), FGVC-Aircraft (Maji et al., 2013) (free for non-commercial use), Stanford
Cars (Krause et al., 2013) (free for non-commercial use), OpenImages v4 (Kuznetsova et al., 2020) (images have a CC BY 2.0 license), Oxford 102 Flower (Nilsback & Zisserman, 2008) (no license) as well as 80 million tiny images (Torralba et al., 2008)
(no license given, see also App. E).
For the train/test splits we use the standard splits, except on 80M Tiny Images where we treat a random but ﬁxed subset of 1000 images in the ﬁrst 1,000,000 as our test set.
For all datasets that get used as a test out-distribution we use a random but ﬁxed subset of 1000 images.
Following (Bitterwolf et al., 2020), the smooth noise that is used is generated as follows.
Uniform noise is generated and then smoothed using a Gaussian ﬁlter with a width that is drawn uniformly at random in [1 , 2 . 5] .
Each datapoint is then shifted and scaled linearly to ensure full range in [0 , 1] , i.e. x (cid:48)
= x − min( x ) max( x ) − min( x ) .
Binary Training The architecture that we use for the binary discriminator is relatively shallow (5 linear layers).
The architecture is shown in Table 3.
Our results are fairly robust to the exact choice of architecture and signiﬁcantly larger models do not necessarily lead to better results as we show in App.
I. Similarly to (Zhang et al., 2020; Bitterwolf et al., 2020), we use long training schedules, running Adam for 1000 epochs, with an initial learning rate of 1 e − 4 that we decrease by a factor of 5 on epochs 500 , 750 and 850 and with a batch size of 128 from the in-distribution and 128 from the out-distribution (for R.ImgNet: 50 epochs with drops at 25, 35, 45, batch sizes 32 ).
In order to avoid large losses we also use a simple ramp up schedule for the (cid:15) used in IBP and we downweight the out-distribution loss during the initial phase of training by a scalar κ .
Both (cid:15) and κ are increased linearly from 0 to their ﬁnal values ( 0 . 01 and 1 , respectively) over the ﬁrstepochs (for R.ImgNet over the ﬁrst 25 epochs).
Compared to the training of (Bitterwolf et al., 2020) [Figure] we plot the test accuracy, AUC and GAUC function the bias shift ∆ which sometimes fails, we found that training of the binary discriminator is very stable and evenepochs on CIFAR would be sufﬁcient
[Figure]
but we found that longer training lead to slightly better results.
Weight decay is set to 5 · 10 − 4 , but is disabled for the weights in the ﬁnal layer.
As data augmentation we use AutoAugment (Cubuk et al., 2019) for CIFAR and simple 4 pixel crops and reﬂections on R.ImgNet.
The strict negativity of the weights leads to a negative bias of g which can cause problems at an early stage if the b ( L g ) g is initialized at 0 and thus we choose 3 as initialization.
All binary discriminators were trained on single 2080Ti GPUs, managed on a SLURM cluster.
Overall, the training of a provable discriminator takes around 16h on CIFAR and 44h on R.ImgNet (wall clock time including evaluations and logging on each epoch).
Semi-Joint Training On CIFAR we train for 100 epochs using SGD with momentum of 0 .
9 and a learning rate of 0 .
1 that drops by a factor of 10 on epochs 50 , 75 and 90 (on R.ImgNet 75 epochs with drops at 30 and 60).
For all datasets we train using a batch size of 128 (plus 128 out-distribution samples in the case of OE).
The CIFAR experiments were run on single 2080Ti GPUs.
This takes about 4h20min in wall clock time.
In order to ﬁt batches of 128 in-distribution samples andout-distribution samples on R.ImgNet we had to train using 4 V100 GPUs in parallel.
Because of batch normalization in multi-GPU training it is important to not simply stack the batches but to interlace in- and out-distribution samples.
The wall clock time was around 15h for the semi-joint training on R.ImgNet.
For selecting the bias we use the procedure described in Section 4.
The trade-off curves for the AUC and GAUC on CIFAR100 and R.ImgNet are given in Figure 4.
E 80M T INY I MAGES AS T RAINING O UT -D ISTRIBUTION
The 80M Tiny Images dataset has been retracted by the authors due to concerns over offensive class labels (Birhane & Prabhu, 2021).
We support the decision of the community to move away from the use of 80M Tiny Images, so we choose to train our CIFAR models using a downscaled version of OpenImages v4 (Kuznetsova et al., 2020) as a training out-distribution.
However, since all prior work used this dataset, we present results on 80M Tiny Images here in order to compare ProoD’s performance to prior baselines.
We encourage the community to use the results in Table 2 for future comparisons.
Our results are shown in Table 5.
Apart from the models shown in Table 2 we add here the pre-trained energy-based OOD detector EB from (Liu et al., 2020) as an additional baseline for clean OOD detection.
As EB is not trained robustly, as expected EB has low AAUCs.
In terms of clean OOD detection it performs similar to OE but with worse results on the more difﬁcult OOD detection task
CIFAR10 vs CIFAR100 and vice versa.
For GOOD we use the pre-trained models from (Bitterwolf et al., 2020).
For ATOM and ACET we use the pre-trained models from (Chen et al., 2020).
Note that these use the densenet architecture and were actually trained to withstand attacks in the much stronger threat model of (cid:15) = 8255 .
In the original paper, the authors claim near perfect AAUCs on this task but we can show that this is a case of gross overestimation of robustness.
Even on the easier threat model (cid:15) = 0 .
01 that we test in Table 5, their robustness is almost non-existent for both ATOM models and the CIFAR100 ACET model.
The fact that their adversarial attacks were unable to ﬁnd these samples clearly demonstrates that evaluating models adversarially is very difﬁcult and potentially unreliable.
Because of this we believe that our guarantees are a valuable contribution to the community.
[Table]
F F ALSE P OSITIVE R ATES
Since in a practical setting a threshold for OOD detection ultimately has to be chosen [Table] it can be interesting to study the false positive rate at a ﬁxed threshold.
It is relatively standard to pick the false positive rate at 95% true positive rate (called FPR in Table 6), where low values are desirable.
We show the results for all methods and datasets in Table 6.
Although ProoD has similarly good performance as OE on this task, it fails to give non-trivial guarantees.
Therefore achieving stronger bounds on the worst-case FPR is an interesting task for future work. [Table]
G A DDITIONAL D ATASETS
In addition to the results shown in Table 2, it is interesting to study how ProoD performs on additional datasets as well as the test set of the out-distribution it was trained on.
For the CIFAR datasets, we report LSUN crops, LSUN_resize, Places365 (Zhou et al., 2017), iSUN (Xu et al., 2015), Textures (Cimpoi et al., 2014), 80M Tiny Images and uniform noise in Table 7. On RImgNet we report uniform noise and the training out-distribution in Table 8.
As in Table 2 the clean performance of ProoD is comparable to that of OE, but it achieves non-trivial
GAUC.
On CIFAR10, GOOD 100 achieves almost perfect GAUC against uniform noise, which comes at the price of signiﬁcantly worse clean AUCs on all other out-distributions, see Table 2.
Almost all methods achieve very high AAUCs on uniform noise, but it is not clear if a sufﬁciently powerful attack could lower those scores signiﬁcantly.
The unusually large gap between the GAUC and AAUC of
ProoD would seem to indicate that that might be the case.
Surprisingly, the robustness characteristics of both ATOM and ACET vary wildly between the different datasets, sometimes appearing to be perfectly robust and on other datasets displaying no robustness at all.
Especially surprising are the relatively low AAUCs on the test set of the training out-distribution.
[Table]
OpenImages
[Table]
[Table]
H D ERIVATION OF E Q . (3)
We restate the derivation in Eq.
(3) in a slightly more verbose form.
For all y we have
[Equation]
The inequality is obtained by using the trivial bound ˆ p ( y | x, i ) ≤ 1 .
This allows us to compute guarantees on the conﬁdence of the entire model while treating the classiﬁer itself entirely as a black-box.
Because of this, there are no restrictions whatsoever on the architecture or training procedure that are used for ﬁtting ˆ p ( y | x, i ) .
I S IZE A BLATION FOR B INARY D ISCRIMINATOR
Since larger models should typically lead to better performance, we investigated the impact that model size has on the performance of our binary discriminator.
We retrained ProoD-Disc models with CIFAR10 as in-distribution and 80M Tiny images as the out-distribution (since our dataloader is faster than for OpenImages which speeds up training).
Since longer schedules only slightly improve results we also used shorter schedules with only 300 epochs where (cid:15) and κ linearly increase fromto 0 .
01 and 1 . 0 respectively within the ﬁrst 100 epochs and the learning rate drops occur at 150,and 250 epochs.
As architectures we use 10 different CNNs with different widths and depths ranging from 5 to 8 layers (for the precise architectures please refer to sizes { S, XL_b, XS, SR, SR2, C1, C3s,
C3, C2, C4 } in the ﬁle provable_classifiers.py of the accompanying code).
We present scatter plots of the models’ their performance on CIFAR100 and 80M Tiny Images in Figure 5 against their size.
Clearly, there is no correlation between model size and performance and most differences are rather small, justifying our choice of a fairly small architecture in the main paper.
[Figure]
J G ENERALIZATION TO LARGER THREAT MODEL
Since (cid:15) = 0 .
01 is a relatively weak threat model we evaluate if ProoD’s guarantees actually generalize to the much stronger (cid:15)
{+= 
+}
≈ 0 .
031 that is standard in much of the literature on adversarial robustness.
We use the exact same CIFAR models from Table 2 and show the results of our evaluation at (cid:15) = 8 255 in Table 9.
[Table]
[Table]
Perhaps surprisingly, ProoD’s guarantees generalize remarkably well to the much larger radius on
CIFAR10.
The same holds for GOOD, as was already observed in (Bitterwolf et al., 2020).
On the other hand ATOM and ACET do not show any robustness at this radius, despite having lower accuracy, lower clean OOD performance and being more expensive to train.
On the other hand
ProoD’s guarantees on CIFAR100 are quite weak at this radius, but given ATOM’s and ACET’s low
AAUCs here and the fact that GOOD cannot be trained at all on CIFAR100, the results are not worse than for the competitors.
K E RROR B ARS
In order to be mindful of our resource consumption we restrict the computation of error bars to our experiments on CIFAR10.
Additionally, because the dataloader was much faster we ran these experiments using 80M Tiny Images as an out-distribution as opposed to OpenImages.
We reran our experiments using the same hyperparameters 5 times.
We computed the mean and the standard deviations for our models for all metrics shown in Table 5.
The results are shown in Table 10.
We see that the ﬂuctuations across different runs are indeed rather small.
Furthermore, the clean performance of OE and ProoD show no signiﬁcant discrepancies.
[Table]