Your Out-of-Distribution Detection Method is Not Robust!
Abstract
Out-of-distribution (OOD) detection has recently gained substantial attention dueto the importance of identifying out-of-domain samples in reliability and safety.
Although OOD detection methods have advanced by a great deal, they are stillsusceptible to adversarial examples, which is a violation of their purpose.
Tomitigate this issue, several defenses have recently been proposed.
Nevertheless,these efforts remained ineffective, as their evaluations are based on either smallperturbation sizes, or weak attacks.
In this work, we re-examine these defensesagainst an end-to-end PGD attack on in/out data with larger perturbation sizes, e.g.up to commonly used ϵ = 8 / 255 for the CIFAR-10 dataset.
Surprisingly, almostall of these defenses perform worse than a random detection under the adversarialsetting.
Next, we aim to provide a robust OOD detection method.
In an idealdefense, the training should expose the model to almost all possible adversarialperturbations, which can be achieved through adversarial training.
That is, suchtraining perturbations should based on both in- and out-of-distribution samples.
Therefore, unlike OOD detection in the standard setting, access to OOD, as well asin-distribution, samples sounds necessary in the adversarial training setup.
Thesetips lead us to adopt generative OOD detection methods, such as OpenGAN, as abaseline.
We subsequently propose the Adversarially Trained Discriminator (ATD),which utilizes a pre-trained robust model to extract robust features, and a generatormodel to create OOD samples.
We noted that, for the sake of training stability, inthe adversarial training of the discriminator, one should attack real in-distributionas well as real outliers, but not generated outliers.
Using ATD with CIFAR-10and CIFAR-100 as the in-distribution data, we could significantly outperform allprevious methods in the robust AUROC while maintaining high standard AUROCand
Introduction
Advances in the deep neural networks have led to their widespread use in real-world applications, suchas object detection and image classification [1, 2].
These models have a high degree of generalizabilityto the extent that they assign an arbitrarily high probability even to the samples not belonging tothe training set [3].
This phenomenon causes problems in safety-critical applications like medicaldiagnosis or autonomous driving that should treat anomalous data differently.
The problem of identifying out-of-distribution data has been widely explored in different categoriessuch as Novelty Detection, Open-Set Recognition and Out-of-Distribution (OOD) detection [4, 5, 6].
These categories are almost similar since they all consider two disjoint sets called the closed (ornormal) and open (or anomalous) sets, and the model should discriminate them [7].
So far, outstandingmodels have been proposed in this research field [8, 9, 10].
Still, these models need to be examinedfrom other aspects, specifically robustness against adversarial perturbations added to the input data.
Deep networks are vulnerable against adversarial examples.
This phenomenon was first noticed in the image classification, but it also extends to other domains [11, 12, 13].
Adversarial examplesare the inputs that are slightly perturbed with a bounded perturbation (e.g. through bounding theℓ p -norm) to cause a high prediction error in the model.
Several defenses have been proposed tomake the image classification robust, none of which is as effective as adversarial training and itsvariants [14, 15, 16].
Similar to other learning domains, OOD detection methods also suffer from theexistence of adversarial examples.
A small perturbation can cause a sample from the closed set to beclassified as anomaly and vise-versa [17, 18].
Robust OOD detection is the connection between these two safety-critical issues.
Since a sampledoes not change semantically under appropriately bounded perturbation in the adversarial attacks, the
OOD detector model is expected to be able to label them as open/closed correctly.
As a first thought,one might envision solving the issue by using the image classification defenses.
Despite advances inthese defenses, they have no intuition of the OOD data and not seen them in the training, makingthem less efficient against end-to-end attacks on the OOD detection method.
Two main approaches are introduced in previous works for the robust OOD detection.
The first oneuses some OOD data, uniformly sampled from the open set classes and disjoint from the closed set,and employs adversarial training to encourage the model to assign a uniform label, as opposed toone-hot labels, to anomalous samples even
under adversarial perturbations [19]. The second approachconducts adversarial training only on the closed set, but encourages the model to learn more semanticfeatures using representation learning techniques such as auto-encoders, self-supervised learning, anddenoising methods [20, 21].
Despite the benefits of these methods, they still have fundamental issuessuch as:
• They did not evaluate their method with an end-to-end attack on the detection method, e.g. theyattack the classification model , and not the detector, to evaluate the detection method.
We showthat this is not an effective and strong attack on the detector.• They consider small perturbation sizes (e.g. ϵ = 1255 in the CIFAR-10 dataset) to protect thestandard classification accuracy, while we show that images do not change perceptually with largerperturbation sizes (e.g. ϵ = 8255 in the CIFAR-10 dataset).
Thus, these methods should be trainedand evaluated against larger perturbations.•
The second approach is still unaware of OOD samples during training.
Furthermore, the firstapproach also cannot cover various aspects of OOD data since they can only use a small numberof OOD training samples.
In this work, we first evaluate previous defenses as well as standard OOD detection methods againstan end-to-end PGD attack [14].
As shown in Fig.
1, almost all of the earlier methods performworse than random in the binary OOD detection with ϵ = 8255 in CIFAR-10.
As a result, the robust
OOD detection method is still far from being solved at present.
In spite of this, it is possible todesign a robust model by addressing the drawbacks of previous methods.
To this end, we propose
Adversarialy Trained Discriminator (ATD) that uses a GAN-based method to generate OOD samplesand discriminate them from the closed set.
An ideal generator naturally deceives the discriminator, and therefore produces adversarial outputs from the open set [22, 23, 24].
This leads to the implicitrobustness of the discriminator against adversarial attacks on the open set.
In addition, we alsoconduct adversarial training on the closed set and the tiny real open set (known as outlier exposure)to achieve robustness on them as well.
Our results (e.g., Fig. 1) show that ATD outperforms previousmethods by a large margin.
Therefore, this article has taken a significant step in this area by revealingthe vulnerability of all previous methods and providing a solution.
Background
2.1 Out-of-Distribution Detection
There are various methods for open/closed set binary classification, which can be categorized intothree main groups:
Likelihoods: A simple but effective method for anomaly detection is the Maximum Softmax Probability (MSP) [25].
This method is applied on a K -class classifier and returns max c ∈{ 1 , 2 ,...,k } f c ( x )as the closed-set membership score of the sample x .
Currently, it is shown that using the MSPwith Vision Transformers (ViT) [26] leads to SOTA results in cross-dataset open set recognition[10].
Other variants of MSP such as calibrated MSP [27] and entropy [28] could replace MSP.
Also, OpenMax [29] replaces the softmax layer with a layer that calibrates the logits by fitting a class-wiseprobability model such as the Weibull distribution [30].
Distances:
The anomaly sample can be detected by its distance to the class conditional distributions.
Mahalanobis distance (MD) [31] and Relative Mahalanobis distance (RMD) [32] are the main methods in this regard.
For an in-distribution with K classes, both of these methods fit a class conditional
Gaussian distribution N ( µ k , Σ) to the pre-logit features z .
The mean vector and covariance matrixare calculated as:
[Equation]
In addition, to use RMD, one has to fit a N ( µ 0 , Σ 0 ) to the whole in-distribution.
Next, the distancesand anomaly score for the input x ′ with pre-logits z ′ are computed as:
[Equation]
[Equation]
Discriminators: Previous categories define the score function based on a K -way classifier, butone can directly train a binary discriminator for this purpose.
Outlier Exposure (OE) [33] exploitssome outlier data to learn a binary discriminator for the open set discrimination.
GOpenMax [34]and OSRCI [35] augment the closed set with fake images through a GAN-based generator.
Also, OpenGAN [24] selects the best discriminator model with an open validation set during training dueto unstable training of GANs.
2.2 Adversarial Attacks
For an input x with the ground-truth label y , an adversarial example x ∗ is crafted by adding smallnoise to x such that the predictor model loss J ( x ∗ , y ) is maximized.
The ℓ p norm of the adversarialnoise should be less than a specified value ϵ , i.e. ∥ x − x ∗ ∥ p ≤ ϵ , to ensure that the image does notchange semantically.
Fast Gradient Sign Method (FGSM) [12] maximizes the loss function with asingle step toward the sign of the gradient of J ( x, y ) with respect to the x :
[Equation]
where the noise meets the ℓ ∞ norm bound ϵ .
Moreover, this method can be conducted iteratively [36]with a smaller step size α :
[Equation]
where the noise should be projected to the ℓ ∞ -ball with ϵ radius at each step, which is called Projected
Gradient Descent (PGD) attack [14].
There are a number of other attacks for evaluating robustness ofthe models [37, 38, 39], but PGD is regarded as a standard and powerful attack.
2.3 Adversarial Defenses
The most effective defenses for image classification and OOD detection are variants of adversarialtraining, which are described in the following.
Adversarial Training (AT) [14]: Through the use of adversarial examples during training, the modelcan learn robust features to withstand adversarial perturbations.
This method is called AT and can beused to optimize model f θ as:
[Equation]
where the inner maximization can be approximated with PGD and the outer minimization with SGD.
AT naturally reduces the standard accuracy, i.e. the accuracy on unperturbed samples [40], whichcan harm the anomaly detection score [10].
Therefore, we also propose Helper-Based Adversarial
Training (HAT) [41] as a baseline that achieves a better trade-off between accuracy and robustness.
Adversarial Learning with inliner and Outlier Exposure (ALOE) [19]: AT lacks informationabout the outlier data.
Therefore, it is only robust on the closed set and not on outliers.
To mitigatethis issue, ALOE includes some outliers in the AT similar to OE [33].
Outliers are used with uniformlabel U K and attacked during training to obtain a model that is robust on both in- and out-distribution.
The objective function of ALOE is summarized as:
[Equation]
Alternatively, one could consider using clean outliers samples in the training.
In addition to ALOE,we evaluate this alternative method as a baseline, and refer to it as Adversarial OE (AOE).
Open-Set Adversarial Defense (OSAD) [20, 21]: Instead of including outlier data in the AT, OSADtries to learn more semantic features in the training.
To this end, OSAD adds dual-attentive denoisinglayers to the model architecture, and combines the AT loss with an auto-encoder loss, whose aimis to reconstruct the clean image from its adversarial version.
A self-supervision loss is also addedby applying transformations on the input image.
They claim that this combination improves OSRrobustness, but they did not evaluate the model against an end-to-end attack.
Method
An OOD detection method should be robust against adversarial perturbations that are added to theclosed or open datasets.
Despite previous efforts to provide robust OOD detection methods, the evaluations have yielded a false sense of robustness due to the weak evaluation attacks.
In Fig.
2, wehave evaluated ALOE and OSAD methods using an end-to-end PGD attack with ϵ = 8255 .
Accordingto the results, both the in and out detection score distributions remarkably change with this attack.
So,these methods are not sufficiently robust.
In the following, we aim to provide a more robust solutionby addressing the drawbacks of previous works.
We then use a toy example to provide more insightsinto the problem and our solution.
3.1 ATD: Adversarially Trained Discriminator
A robust OOD detection method must consider several factors to achieve robustness on both openand closed sets.
There has been an extensive study of closed set robustness, which shows that variantsof AT are the most effective defenses.
Still, pure AT cannot provide robustness in OOD detectionfor two primary reasons.
First, an AT model achieves a lower accuracy due to the trade-off betweenaccuracy and robustness [40, 15].
Note that accuracy plays an important role in the OOD detection[10].
Second, the AT does not consider samples from the open set during training.
ALOE tried tomitigate this issue using open datasets in training.
However, this strategy works only when the opentraining data is a close proxy for the OOD data.
Unfortunately, this is not true in most cases.
To resolve these issues, we consider OpenGAN as our baseline.
Here, the discriminator performsthe in/out binary classification.
In this setting, a generator crafts OOD images to deceive thediscriminator by images similar to the closed set.
An ideal generator would cover a broader rangeof OOD distributions, which is a perfect setting for our problem.
Furthermore, as the generator istrained to deceive the discriminator, the generated samples are naturally adversarial examples to thediscriminator, which meets the needs in the AT.
Hence no adversarial attack is need on such samples.
Thus, we only need to perturb the closed set to adversarially train the discriminator.
We call thismethod Adverarially Trained Discriminator (ATD), which can be summarized as:
[Equation]
where the D and G are the discriminator and generator, respectively.
Moreover, the inner minimizationis approximated with the PGD attack and the outer minimization and maximization with SGD.
Usingadversarial perturbations to train GANs makes them more robust to adversarial examples, just likeother architectures [42, 43].
On the other hand, adding perturbations to the closed set makes itdifficult to optimize the discriminator.
As a result, the generator would not be trained to cover a broadrange since it could easily fool the discriminator in the min-max game for training GANs.
To get closer to the ideal generator that can cover a broad range of OOD distributions, unlikepreviously proposed robust GANs [42], we make the generator craft features instead of images.
It isevident that generating and discriminating the low-dimensional features is an easier problem than the high-dimensional images, which brings the generator closer to the ideal case.
As a result, wealso need a model that can extract robust features from the closed set.
For this purpose, we usea pre-trained model with the HAT [41] method that achieves a satisfactory accuracy along withrobustness on the closed set, considering the trade-off between accuracy and robustness.
Finally, similar to the OpenGAN and ALOE, we utilize an open dataset in the training to stabilizethe training.
This data is also attacked during training since it is neither originally adversarial, nor iscrafted by the generator.
With these in mind, the objective function of the ATD can be reformulatedas follows:
[Equation]
(9) where f θ is the pre-trained robust feature extractor and α controls the weight of real open dataset inthe optimization.
Fig.
3 shows a schematic representation of ATD.
3.2 Toy Example
Here, we provide an insightful visualization of the ATD using a 2D toy example.
To this end, werepresent open and closed sets with disjoint distributions in a 2D space.
Also, some samples areselected randomly in this space as the generated data.
Fig. 4(a) shows the distributions used for thispurpose, which the blue, orange, and green samples are closed, open, and generated data, respectively.
Next, we train a multi-layer feed-forward neural network as the discriminator to perform the binaryclassification of in/out on these data in different cases.
It should be noted that the open and closeddata are fixed during the training, but the generated data is resampled in every epoch to simulate ATDtraining.
Finally, the discriminator output is displayed all over the space with a blue (in) and orange(out) background in each case.
In our analysis, different conditions are taken into account.
First, standard training is conducted onthe data without attacking any samples.
The results in Fig. 4(b) show that the model has learned theclassification decision boundaries very well.
In each of the next three cases, one of the closed, openand generated sets is attacked during training with ϵ = 1 .
Fig.
4(c) shows that attacking the closed set leads to a larger margin in the decision boundaries around them.
Having this margin ensuresrobustness against attacks that seek to misclassify the in-distribution samples.
Additionally, note thatthis margin is smaller in the right side that an open set is present, which shows the impact of usingopen sets during training.
In Fig. 4(d), only the open set is attacked during the training.
This hascaused a tighter decision boundaries around the closed set where the open set is present.
Therefore,using attacked open set in the training can slightly improve the robustness on OOD data, but it cansimultaneously reduce the robustness on the closed set.
As a result, α in Eq.
9 that controls theoptimization weight of open set should be selected carefully.
In contrast with the ATD method, wealso attack the generated data to see how it affects the result.
According to Fig 4(e), this does notcause robustness in the OOD data in contrast to Fig 4(d), which is another reason that ATD do notattack the generated data during training.
It is noteworthy that the main cause for this observationmay be the instability of the generated samples during the training, which is inevitable and makes theadversarial optimization goal infeasible.
Eventually, we attack both the open and closed sets during training as in ATD.
The results in Fig.
4(f)show that the margin around the closed set increases in the directions that an open set is not presentto provide robustness on both in and out samples, which can be considered as a mixture of results in
Fig. 4(c) and 4(d).
Also, this plot demonstrates that the robustness of the open and closed sets maybe at odds with each other.
Experiments
In this section, we perform extensive experiments to evaluate existing OOD detection methods,including the standard and adversarially trained ones, and our ATD method against an end-to-end
PGD attack.
To this end, we first give details about the setting of the experiments.
Next, we compareall the methods, which shows that ATD significantly outperforms the other methods.
Toward the end,we conduct some additional experiments to investigate some aspects of our solution.
[Table]
Detection Methods: Apart from the methods that use discriminators for the detection, other modelsare evaluated using MSP, OpenMax, MD and RMD as likelihood and distance based methods.
In-distribution Datasets: CIFAR-10 and CIFAR-100 [44] are used as the in-distribution datasets.
The images pixel values are normalized to be in the range of 0 to 1.
Out-of-distribution Datasets: Following the setting in earlier works [19, 33], we use eight differentdatasets that are disjoint from the in-distribution sets, including MNIST [45], TinyImageNet [46],
Places365 [47], LSUN [48], iSUN [49], Birds [50], Flowers [51], and COIL-100 [52].
The resultsare averaged over these datasets to peform a comprehensive evaluation on different OOD datasets.
The SVHN [53] dataset is also used as the open training set and a disjoint OOD validation set toselect the best discriminator during the training.
Baselines: Various methods have been considered in our comparisons.
The ViT architecture and
OpenGAN are used as the SOTA methods in the standard OOD detection.
AT and HAT are consideredas the effective defenses in the image classification, and AOE, ALOE, and OSAD as the effectivedefenses in the OOD detection.
Note that we are the first work that uses HAT for the OOD detection.
All the defenses are trained with ϵ = 8255 to have the best results against attack with this perturbationbudget.
Moreover, these methods are trained based on the guidelines in their original work.
ATD Hyperparameters: A simple DCGAN [54] is used for the generator and discriminator architecture in the ATD.
Furthermore, ATD is trained for 30 epochs using Adam [55] optimizer with learningrate of 1 e − 4 .
Also, α is selected from the set { 0 . 0 , 0 . 25 , 0 . 5 } through validation.
Details of the
ATD method is available in section 3.1.
Evaluation Attack: All the models are evaluated against an end-to-end PGD attack with ϵ = 8 255 .
For the baseline methods, we only use a 10-step attack, but ATD is evaluated with 100 steps to ensureits robustness.
Also, we show that this perturbation budget does not change the images semanticallyand 100 steps is more than enough to ensure the robustness of ATD.
Evaluation Metric: We use AUROC as a well-known classification criterion.
The AUROC value isin the range [0 , 1] , and the closer it is to 1 , the better the classifier performance.
[Table]
(OpenMax) 0.
HAT (MSP) 0.
HAT (MD) 0.
HAT (RMD) 0.
HAT (OpenMax) 0.
OSAD (MSP) 0.
OSAD (MD) 0.
OSAD (RMD) 0.
OSAD (OpenMax) 0.
AOE (MSP) 0.
AOE (MD) 0.
AOE (RMD) 0.
[Table]
4.2 Results
OOD detection under adversarial attack: To perform a comprehensive study, AUROC is computedin four different settings for each method.
First, the standard OOD detection without any attack isconducted (Clean).
Next, either in- or out-datasets are attacked (In/Out).
Finally, both the in- andout-sets are attacked (In and Out).
Note that resistance against the attack to both in- and out-sets ismuch harder than other cases since the perturbation budget has effectively been doubled.
Based onthe results in Table 1, ViT+MD and OpenGAN-fea have the best performance in the standard OODdetection, but they completely fail under the adversarial setting.
Early defenses improved adversarialperformance at the cost of a decrease in the clean detection rate.
Among them, HAT+OpenMax and
HAT+MD are the most effective defenses in CIFAR-10 and CIFAR-100, respectively.
Still, they arenot as effective as our method ATD, which significantly outperforms them in both datasets whilepreserving the clean detection performance.
Black-box evaluation: To evaluate ATD under the black-box setting, we generate adversarialperturbations by attacking the introduced baselines as the source models and transferring them to the
ATD as the target model.
This test can also be considered as a sanity check for detecting gradientobfuscation in the model [56].
Since the single-step attack enjoys better transferability than the [Table] [Table] multi-step ones [36], we use FGSM for attacking the source models.
Based on the results in Table 2,
ATD is sufficiently robust against all the transferred attacks
[Table]
Ablation study: ATD uses HAT as the feature extractor and performs adversarial training on openand closed sets to robustify the discriminator.
As an ablation study, we replace HAT with a standardtrained model to check its effectiveness.
Also, in another experiment, we replace the discriminatoradversarial training with the standard training.
The results in Table 3 demonstrate that these settingsare not as effective as ATD in achieving a robust detection model.
As another ablation, we considerremoving the feature extractor and generating images instead of features.
This method completelyfails since it reaches a low AUROC in the standard OOD detection.
Moreover, we check the effect ofattacking the generated images, which leads to a lower AUROC score.
This confirms that attackinggenerated data is not helpful in the training.
Classification attack instead of end-to-end attack: OSAD method is evaluated against an attackon the classification, and not the OOD detection, in their original work.
Here, we show that thisis not a strong attack on the detection method.
This is done by checking whether an attack onclassification or detection method is effective on the other.
Results in Table 4 for OSAD and ATDmethods demonstrate that attacking classification and detection is significantly more effective ontheir self than the other.
Thus, an end-to-end attack is a better basis for the evaluation of the defenses.
[Table]
Conclusion
The existing OOD detection methods are far from being robust against strong attacks, contrary towhat has been claimed previously.
To mitigate this issue, we proposed ATD that uses an adversariallytrained discriminator to classify in and out samples.
Moreover, ATD utilizes HAT to extract robustfeatures from the real samples, and a generator to craft a broad range of OOD data features.
Thismethod could significantly outperform earlier methods against an end-to-end PGD attack.
Also, it issufficiently robust against black-box attacks.
The primary advantage of ATD is that it preserves thestandard OOD detection AUROC along with the robustness, which is needed in real-world situations.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] . You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description. For example: • Did you include the license to the code and datasets?
[Yes] See Section x.•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[No] We do notforesee any potential negative impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplemen-
tal material.(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No] Error bars were too small to have any visual impact.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] We have used a single RTX
Super GPU.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No] Data are publicly available, which
have been cited.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]