SSD: A U NIFIED F RAMEWORK FOR S ELFS UPERVISED O UTLIER D ETECTION
A BSTRACT
We ask the following question: what training information is required to design an effective outlier/out-of-distribution (OOD) detector, i.e., detecting samples that lie far away from the training distribution?
Since unlabeled data is easily accessible for many applications, the most compelling approach is to develop detectors based on only unlabeled in-distribution data.
However, we observe that most existing detectors based on unlabeled data perform poorly, often equivalent to a random prediction.
In contrast, existing state-of-the-art OOD detectors achieve impressive performance but require access to ﬁne-grained data labels for supervised training.
We propose SSD , an outlier detector based on only unlabeled in-distribution data.
We use self-supervised representation learning followed by a Mahalanobis distance based detection in the feature space.
We demonstrate that SSD outperforms most existing detectors based on unlabeled data by a large margin.
Additionally, SSD even achieves performance on par, and sometimes even better, with supervised training based detectors.
Finally, we expand our detection framework with two key extensions.
First, we formulate few-shot OOD detection , in which the detector has access to only one to ﬁve samples from each class of the targeted OOD dataset.
Second, we extend our framework to incorporate training data labels, if available.
We ﬁnd that our novel detection framework based on SSD displays enhanced performance with these extensions, and achieves state-of-the-art performance 1 .
I NTRODUCTION
Deep neural networks are at the cornerstone of multiple safety-critical applications, ranging from autonomous driving (Ramanagopal et al., 2018) to biometric authentication (Masi et al., 2018;
G ¨ unther et al., 2017).
When trained on a particular data distribution, referred to as in-distribution data, deep neural networks are known to fail against test inputs that lie far away from the training distribution, commonly referred to as outliers or out-of-distribution (OOD) samples (Grubbs, 1969;
Hendrycks & Gimpel, 2017).
This vulnerability motivates the use of an outlier detector before feeding the input samples to the downstream neural network modules.
However, a key question is to understand what training information is crucial for effective outlier detection?
Will the detector require ﬁne-grained annotation of training data labels or even access to a set of outliers in the training process?
Since neither data labels nor outliers are ubiquitous, the most compelling option is to design outlier detectors based on only unlabeled in-distribution data.
However, we observe that most of the existing outlier detectors based on unlabeled data fail to scale up to complex data modalities, such as images.
For example, autoencoder (AE)
(Hawkins et al., 2002) based outlier detectors have achieved success in applications such as intrusion detection (Mirsky et al., 2018), and fraud detection (Schreyer et al., 2017).
However, this approach achieves close to chance performance on image datasets.
Similarly, density modeling based methods, such as PixelCNN++ (Salimans et al., 2017) and Glow (Kingma & Dhariwal, 2018) are known to assign even a higher likelihood to outliers in comparison to indistribution data (Nalisnick et al., 2019).
In contrast, existing state-of-the-art OOD detectors achieve high success on image datasets but assume the availability of ﬁne-grained labels for in-distribution samples (Hendrycks & Gimpel, 2017;
Bendale & Boult, 2016; Liang et al., 2018; Dhamija et al., 2018; Winkens et al., 2020).
This is a strong assumption since labels, in-particular ﬁne-grained labels, can be very costly to collect in some applications (Google AI Pricing, 2020), which further motivates the use of unlabeled data.
The inability of supervised detectors to use unlabeled data and poor performance of existing unsupervised approaches naturally give rise to the following question
Can we design an effective out-of-distribution (OOD) data detector with access to only unlabeled data from training distribution?
A framework for outlier detection with unlabeled data 2 involves two key steps: 1) Learning a good feature representation with unsupervised training methods 2)
Modeling features of in-distribution data without requiring class labels.
For example, autoencoders attempt to learn the representation with a bottleneck layer, under the expectation that successful reconstruction requires learning a good set of representations.
Though useful for tasks such as dimensionality reduction, we ﬁnd that these representations are not good enough to sufﬁciently distinguish in-distribution data and outliers.
We argue that if unsupervised training can develop a rich understanding of key semantics in in-distribution data then absence of such semantics in outliers can cause them to lie far away in the feature space, thus making it easy to detect them.
Recently, self-supervised representation learning methods have made large progress, commonly measured by accuracy achieved on a downstream classiﬁcation task (Chen et al., 2020;
He et al., 2020;
Oord et al., 2018; Misra & Maaten, 2020; Tian et al., 2020).
We leverage these representations in our proposed cluster-conditioned framework based on the Mahalanobis distance (Mahalanobis, 1936).
Our key result is that self-supervised representations are highly effective for the task of outlier detection in our self-supervised outlier detection ( SSD ) framework where they not only perform far better than most of the previous unsupervised representation learning methods but also perform on par, and sometimes even better, than supervised representations.
What if access to a fraction of OOD data or training data labels is available?
How do we move past a detector based on unlabeled data and design a framework which can take advantage of such information?
Though access to outliers during training is a strong assumption, it may be feasible to obtain a few prior instances of such outliers (G ¨ ornitz et al., 2013).
We characterize this setting as few-shot OOD detection , where we assume access to very few, often one to ﬁve, samples from the targeted set of outliers.
While earlier approaches (Liang et al., 2018; Lee et al., 2018b) mostly use such data to calibrate the detector, we ﬁnd that access to just a few outliers can bring an additional boost in the performance of our detector.
Crucial to this success is the reliable estimation of ﬁrst and second order statistics of OOD data in the high dimensional feature space with just a few samples.
Finally, if class labels are available in the training phase, how can we incorporate them in the SSD framework for outlier detection?
Recent works have proposed the addition of the supervised crossentropy and self-supervised learning loss with a tunable parameter, which may require tuning for optimal parameter setting for each dataset (Hendrycks et al., 2019b; Winkens et al., 2020).
We demonstrate that incorporating labels directly in the contrastive loss achieves 1) a tuning parameterfree detector, and 2) state-of-the-art performance.
1. K EY C ONTRIBUTIONS
SSD for unlabeled data.
We propose SSD , an unsupervised framework for outlier detection based on unlabeled in-distribution data.
We demonstrate that SSD outperforms most existing unsupervised outlier detectors by a large margin while also performing on par, and sometimes even better than supervised training based detection methods.
We validate our observation across four different datasets: CIFAR-10, CIFAR-100, STL-10, and ImageNet.
Extensions of SSD .
We provide two extensions of SSD to further improve its performance.
First, we formulate few-shot OOD detection and propose detection methods which can achieve a signiﬁcantly large gain in performance with access to only a few targeted OOD samples.
Next, we extend
SSD , without using any tuning parameter, to also incorporate in-distribution data labels and achieve state-of-the-art performance.
R ELATED WORK
OOD detection with unsupervised detectors.
Interest in unsupervised outlier detection goes back to Grubbs (1969).
We categorize these approaches in three groups 1) Reconstruction-error based detection using Auto-encoders (Hawkins et al., 2002; Mirsky et al., 2018; Schreyer et al., 2017) or
Variational auto-encoders (Abati et al., 2019; An & Cho, 2015) 2) Classiﬁcation based, such as DeepSVDD (Ruff et al., 2018; El-Yaniv & Wiener, 2010; Geifman & El-Yaniv, 2017) and 3) Probabilistic detectors, such as density models like Glow and PixelCNN++ (Ren et al., 2019; Nalisnick et al., 2019;
Salimans et al., 2017; Kingma & Dhariwal, 2018).
We compare with detectors from each category and ﬁnd that SSD outperforms them by a wide margin.
OOD detection with supervised learning.
Supervised detectors have been most successful with complex input modalities, such as images and language (Chalapathy et al., 2018a; DeVries & Taylor, 2018; Dhamija et al., 2018; Jiang et al., 2018; Yoshihashi et al., 2018; Lee et al., 2018a).
Most of these approaches model features of in-distribution data at output (Liang et al., 2018; Hendrycks &
Gimpel, 2017; Dhamija et al., 2018) or in the feature space (Lee et al., 2018b; Winkens et al., 2020) for detection.
We show that SSD can achieve performance on par with these supervised detectors, without using data labels.
A subset of these detectors also leverages generic OOD data to boost performance (Hendrycks et al., 2019a; Mohseni et al., 2020).
Access to OOD data at training time.
Some recent detectors also require OOD samples for hyperparameter tuning (Liang et al., 2018;
Lee et al., 2018b; Zisselman & Tamar, 2020).
We extend
SSD to this setting but assume access to only a few OOD samples, referred to as few-shot OOD detection , which our framework can efﬁciently utilize to bring further gains in performance.
In conjunction with supervised training.
Vyas et al.
(2018) uses ensemble of leave-one-out classiﬁer, Winkens et al.
(2020) uses contrastive self-supervised training, and Hendrycks et al.
(2019b) uses rotation based self-supervised loss, in conjunction with supervised cross-entropy loss to achieve state-of-the-art performance in OOD detection.
Here we extend SSD , to incorporate data labels, when available, and achieve better performance than existing state-of-the-art.
Anomaly detection .
In parallel to OOD detection, this research direction focuses on the detection of semantically related anomalies in applications such as intrusion detection, spam detection, disease detection, image classiﬁcation, and video surveillance.
We refer the interested reader to Pang et al.
(2020) for a detailed review.
While a large number of works focuses on developing methods particularly for single-class modeling in anomaly detection (Perera et al., 2019;
Schlegl et al., 2017;
Ruff et al., 2018;
Chalapathy et al., 2018b; Golan & El-Yaniv, 2018; Wang et al., 2019), some recent work achieve success in both OOD detection and anomaly detection Tack et al.
; Bergman & Hoshen (2020).
We provide a detailed comparison of our approach with previous work in both categories.
SSD: S ELF - SUPERVISED OUTLIER / OUT - OF - DISTRIBUTION DETECTION
In this section, we ﬁrst provide the necessary background on outlier/out-of-distribution (OOD) detection and then present the underlying formulation of our self-supervised detector ( SSD ) that relies on only unlabeled in-distribution data.
Finally, we describe two extensions of SSD to (optionally) incorporate targeted OOD samples and in-distribution data labels (if available).
Notation.
We represent the input space by X and corresponding label space as Y .
We assume in-distribution data is sampled from P in X×Y .
In the absence of data labels, it is sampled from marginal distribution P in X .
We sample out-of-distribution data from P ood X .
We denote the feature extractor by f : X → Z , where Z ⊂ R d , a function which maps a sample from the input space to the d -dimensional feature space ( Z ).
The feature extractor is often parameterized by a deep neural network.
In supervised learning, we obtain classiﬁcation conﬁdence for each class by g ◦ f : X → R c .
In most cases, g is parameterized by a shallow neural network, generally a linear classiﬁer.
Problem Formulation: Outlier/Out-of-distribution (OOD) detection.
Given a collection of samples from P in X × P ood X , the objective is to correctly identify the source distribution, i.e., P in X or P ood X , for each sample.
We use the term supervised OOD detectors for detectors which use in-distribution data labels, i.e., train the neural network ( g ◦ f ) on P in X×Y using supervised training techniques.
Unsupervised OOD detectors aim to solve the aforementioned OOD detection tasks, with access to only P in X .
In this work, we focus on developing effective unsupervised OOD detectors.
Background: Contrastive self-supervised representation learning.
Given unlabeled training data, it aims to train a feature extractor, by discriminating between individual instances from data, to learn a good set of representations.
Using image transformations, it ﬁrst creates two views of each image, commonly referred to as positives.
Next, it optimizes to pull each instance close to its positive instances while pushing away from other images, commonly referred to as negatives.
Assuming that ( x i , x j ) are positive pairs for the i th image from a batch of N images and h ( . ) is a projection header, τ is the temperature, contrastive training minimizes the following loss, referred to as Normalized temperature-scaled cross-entropy ( NT-Xent ), over each batch.
[Equation]
3. U NSUPERVISED OUTLIER DETECTION WITH SSD
Leveraging contrastive self-supervised training.
In the absence of data labels, SSD consists of two steps: 1) Training a feature extractor using unsupervised representation learning, 2) Developing an effective OOD detector based on hidden features which isn’t conditioned on data labels.
[Figure]
Higher eigenvalues dominates euclidean distance, but are least helpful for outlier detection.
Mahalnobis distance avoid this bias with appropriate scaling and performs much better.
We leverage contrastive self-supervised training for representation learning in our outlier detection framework, particularly due to its state-of-the-art performance (Chen et al., 2020; Tian et al., 2020).
We will discuss the effect of different representation learning methods later in
Section 4.2.
Cluster-conditioned detection.
In absence of data labels, we develop a cluster-conditioned detection method in the feature space.
We ﬁrst partition the features for in-distribution training data in m clusters.
We represent features for each cluster as Z m .
We use k-means clustering method, due to its effectiveness and low computation cost.
Next, we model features in each cluster independently, and calculate the following outlier score ( s x ) =
min m D ( x, Z m ) for each test input x , where D ( ., . ) is a distance metric in the feature space.
We discuss the choice of the number of clusters in Section 4.2.
Choice of distance metric: Mahalanobis distance.
We use Mahalanobis distance to calculate the outlier score as follows:
[Equation]
where µ m and Σ m are the sample mean and sample covariance of features ( Z ) of the in-distribution training data.
We justify this choice with quantitative results in Figure 1.
With eigendecomposition of sample covariance ( Σ m =
Q m Λ m Q − 1 m ), s x = min m (cid:0) Q Tm
( z x − µ m ) (cid:1) T Λ − 1 m (cid:0) Q Tm ( z x − µ m ) (cid:1) which is equivalent to euclidean distance scaled with eigenvalues in the eigenspace.
We discriminate between in-distribution (CIFAR-10) and OOD (CIFAR-100) data along each principal eigenvector (using AUROC, higher the better).
With euclidean distance, i.e., in absence of scaling, components with higher eigenvalues have more weight but provide least discrimination.
Scaling with eigenvalues removes the bias, making Mahalnobis distance effective for outlier detection in the feature space.
3. F EW -S HOT OOD DETECTION ( SSD k )
In this extension of the SSD framework, we consider the scenario where a few samples from the
OOD dataset used at inference time are also available at the time of training.
We focus on one-shot and ﬁve-shot detection, which refers to access to only one and ﬁves samples, from each class of the targeted OOD dataset, respectively.
Our hypothesis is that in-distribution samples and OOD samples will be closer to other inputs from their respective distribution in the feature space , while lying further away from each other.
We incorporate this hypothesis by using following formulation of outlier score.
[Equation]
where µ in , Σ in and µ ood , Σ ood are the sample mean and sample covariance in the feature space for in-distribution and OOD data, respectively.
Challenge.
The key challenge is to reliably estimate the statistics for OOD data, with access to only a few samples.
Sample covariance is not an accurate estimator of covariance when the number of samples is less than the dimension of feature space (Stein, 1975), which is often in the order of thousands for deep neural networks.
Shrunk covariance estimators and data augmentation.
We overcome this challenge by using following two techniques: 1) we use shrunk covariance estimators (Ledoit & Wolf, 2004), and 2) we amplify number of OOD samples using data augmentation.
We use shrunk covariance estimators due to their ability to estimate covariance better than sample covariance, especially when the number of samples is even less than the feature dimension.
To further improve the estimation we amplify the number of samples using data augmentation at the input stage.
We use common image transformations, such as geometric and photometric changes to create multiple different images from a single source image from the OOD dataset.
Thus given a set of k
OOD samples { u 1 , u 2 , . . .
, u k } , we ﬁrst create a set of k × n samples using data augmentation, U = { u 11 , . . .
, u n 1 , . . .
u 1 k , . . .
, u nk } .
Using this set, we calculate the outlier score for a test sample in the following manner.
[Equation]
where µ U and S U are the sample mean and estimated covariance using shrunk covariance estimators for the set U , respectively.
3. H OW TO BEST USE DATA LABELS ( SSD + )
If ﬁne-grained labels for in-distribution data are available, an immediate question is how to incorporate them in training to improve the success in detecting OOD samples.
Conventional approach: Additive self-supervised and supervised training loss.
A common theme in recent works (Hendrycks et al.
2019b; Winkens et al.
2020) is to add self-supervised ( L ssl ) and supervised ( L sup ) training loss functions, i.e., L training =
L sup + αL ssl , where the hyperparameter α is chosen for best performance on OOD detection.
A common loss function for supervised training is cross-entropy.
Our approach: Incorporating labels in contrastive self-supervised training.
As we show in
Section 4.2, even without labels, contrasting between instances using self-supervised learning is highly successful for outlier detection.
We argue for a similar instance-based contrastive training, where labels can also be incorporated to further improve the learned representations.
To this end, we use the recently proposed supervised contrastive training loss function (Khosla et al., 2020), which uses labels for a more effective selection of positive and negative instances for each image.
In particular, we minimize the following loss function.
[Equation]
5) where N y i refers to number of images with label y i in the batch.
In comparison to contrastive
NT-Xent loss (Equation 1), now we use images with identical labels in each batch as positives.
We will show the superior performance of this approach compared to earlier approaches, and note that it is also a parameter-free approach which doesn’t require additional OOD data to tune parameters.
We further use the proposed cluster-conditioned framework with Mahalnobis distance, as we ﬁnd it results in better performance than using data labels.
We further summarize our framework in
Algorithm 1.
Function getFeatures( X ): return
∀ x i ∈ X} ;
Function SSDScore( Z µ
Σ ): return {
− µ
Σ −
− µ ∀ ∈
Function SSDkScore( Z , µ in , Σ in , µ ood , Σ ood ): return µ T µ end
Parition X in in training set ( X train ) and calibration set ( X cal ); if Y in is not available then
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
else
[Equation]
; end
Train
Z train =
( X train
Z
=
( X cal )
Z test = test if ood is available : Z ood =
getFeatures ( X ood ); if X ood is not available then s cal =
SSDScore ( Z cal , µ train , Σ train ) ; # Sample mean and convariance of Z train s test =
SSDScore ( Z test , µ train , Σ train )
# outlier score; else # Using convariance estimation techniques from Section 3.2 for Z ood s cal =
SSDkScore ( Z cal , µ train , Σ train , µ ood , Σ ood ) ; s test =
SSDkScore ( Z test , µ train , Σ train , µ ood , Σ ood ) ; end x i
∈
X test is an outlier if s itest > ( s cal threshold at TPR = T );
E XPERIMENTAL RESULTS 4. C OMMON SETUP ACROSS ALL EXPERIMENTS
We use recently proposed NT-Xent loss function from SimCLR (Chen et al., 2020) method for self-supervised training.
We use the ResNet-50 network in all key experiments but also provide ablation with ResNet-18, ResNet-34, and ResNet-101 network architecture.
We train each network, for both supervised and self-supervised training, with stochastic gradient descent for 500 epochs, 0.5 starting learning rate with cosine decay, and weight decay and batch size set to 1e-4 and 512, respectively.
We set the temperature parameter to 0.5 in the NT-Xent loss.
We evaluate our detector with three performance metrics, namely FPR (at TPR=95%), AUROC, and AUPR.
For the supervised training baseline, we use identical training budget as SSD while also using the Mahalanobis distance based detection in the feature space.
Due to space constraints, we present results with AUROC, which refers to area under the receiver operating characteristic curve, in the main paper and provide detailed results with other performance metrics in Appendix B.5.
Our setup incorporates six image datasets along with additional synthetic datasets based on random noise.
We report the average results over three independent runs in most experiments.
Number of clusters.
We ﬁnd the choice of the number of clusters dependent on which layer we extract the features from in the Residual neural networks.
While for the ﬁrst three blocks, we ﬁnd an increase in AUROC with the number of clusters, the trend is reversed for the last block (Appendix B.2).
Since the last block features achieve the highest detection performance, we model the in-distribution features as a single cluster in subsequent experiments.
4. P ERFORMANCE OF SSD
Comparison with unsupervised learning based detectors.
We present this comparison in Table 1.
We ﬁnd that SSD improves average AUROC by up to 55, compared to standard outlier detectors
[Table]
based on Density modeling (PixelCNN++ (Salimans et al., 2017)), input reconstruction (Autoencoder (Hawkins et al., 2002), Variational Auto-encoder (Kingma & Welling, 2014)), and One-class classiﬁcation (Deep-SVDD Ruff et al.
A common limitation of each of these three detectors is to ﬁnd images from the SVHN dataset as more in-distribution when trained on CIFAR-10 or
CIFAR-100 dataset.
In contrast, SSD is able to successfully detect a large fraction of outliers from
SVHN dataset.
We also experiment with Rotation-loss Gidaris et al.
(2018), a non-contrastive self-supervised training objective.
We ﬁnd that SSD with contrastive NT-Xent loss achieves 9.6% higher average AUROC compared to using Rotation-loss.
Ablation studies.
We ablate along individual parameters in self-supervised training with CIFARas in-distribution data (Figure 2).
While architecture doesn’t have a very large effect on AUROC for most OOD dataset, we ﬁnd that the number of training epochs and batch size plays a key role in detecting outliers from the CIFAR-100 dataset, which is hardest to detect among the four OOD datasets.
We also ﬁnd an increase in the size of training dataset helpful in the detection of all four OOD datasets.
[Figure]
Comparison with supervised representations.
We earlier asked the question whether data labels are even necessary to learn representations crucial for OOD detection?
To answer it, we compare
SSD with a supervised network, trained with an identical budget as SSD while also using Mahalanobis distance in the feature space, across sixteen different pairs of in-distribution and out-of-distribution datasets (Table 2).
We observe that self-supervised representations even achieve better performance than supervised representations for 56% of the tasks in Table 2.
Success in anomaly detection .
We now measure the performance of SSD in anomaly detection where we consider one of the CIFAR-10 classes as in-distribution and the rest of the classes as a source of anomalies.
Similar to the earlier setup, we train a ResNet-50 network using self-supervised training with NT-Xent loss function.
While the contrastive loss attempt to separate individual instances in the feature space, we ﬁnd that adding an (cid:96) 2 regularization in the feature space helps in improving performance.
In particular, we add this regularization (with a scaling coefﬁcient of 0.01) to bring individual instance features close to the mean of all feature vectors in the batch.
Additionally, we reduce the temperature from 0.5 to 0.1 to reduce the separability of individual instances due to the contrastive loss.
Overall, we ﬁnd that our approach outperforms all previous works and achieves competitive performance with the concurrent work of Tack et al.
(Tabel 3).
4. F EW - SHOT OOD DETECTION ( SSD k )
Setup.
We focus on one-shot and ﬁve-shot OOD detection, i.e., set k to 1 or 5 in Equation 4 and use
Ledoit-Wolf (Ledoit & Wolf, 2004) estimator for covariance estimation.
To avoid a bias on selected samples, we report average results over 25 random trials.
[Table] [Table]
Results.
Compared to the baseline SSD detector, one-shot and ﬁve-shot settings improve the average
AUROC, across all OOD datasets, by 1.6 and 2.1, respectively (Table 1, 2).
In particular, we observe large gains with CIFAR-100 as in-distribution and CIFAR-10 as OOD where ﬁve-shot detection improves the AUROC from 69.6 to 78.3.
We ﬁnd the use of shrunk covariance estimator most critical in the success of our approach.
Use of shrunk covariance estimation itself improves the AUROC from 69.6 to 77.1.
Then data augmentation further improves it 78.3 for the ﬁve-shot detection.
With an increasing number of transformed copies of each sample, we also observe improvement in AUROC, though it later plateaus close to ten copies (Appendix B.3).
What if additional OOD images are available Note that some earlier works, such as Liang et al. (2018), assume that 1000 OOD inputs are available for tuning the detector.
We ﬁnd that with access to this large amount of OOD samples, SSD k can improve the state-of-the-art by an even larger margin.
For example, with CIFAR-100 as in-distribution and CIFAR-10 as out-of-distribution, it achieves 89.4 AUROC, which is 14.2% higher than the current state-of-the-art (Winkens et al., 2020).
4. S UCCESS WHEN USING DATA LABELS ( SSD + )
Now we integrate labels of training data in our framework and compare it with the existing state-ofthe-art detectors.
We report our results in Table 4.
Our approach improves the average AUROC by 0.
over the previous state-of-the-art detector.
Our approach also achieves equal or better performance than previous state-of-the-art across individual pairs of in and out-distribution dataset.
For example, using labels in our framework improves the AUROC of Mahalanobis detector from 55.5 to 72.
for CIFAR-100 as in-distribution and CIFAR-10 as the OOD dataset.
Using the simple softmax probabilities, training a two-layer MLP on learned representations further improves the AUROC to 78.3.
Combining SSD + with a ﬁve-shot OOD detection method further brings a gain of 1.4 in the average AUROC.
D ISCUSSION AND C ONCLUSION
On tuning hyperparameters in SSD .
In our framework, we either explicitly avoid the use of additional tuning-parameters (such as when combining self-supervised and supervised loss functions
[Table]
in SSD + ) or refrain from tuning the existing set of parameters for each OOD dataset.
For example, we use a standard set of parameters for self-supervised training and model the learned features with a single-cluster.
[Table]
Why contrastive self-supervised learning is effective in the SSD framework?
We focus on the NT-Xent loss function, which is parameterized by a temperature variable ( τ ).
Its objective is to pull positive instances, i.e., different transformations of an image, together while pushing away from other instances.
Earlier works have shown that such contrastive training forces the network to learn a good set of feature representations.
However, a smaller value of temperature quickly saturates the loss, discouraging it to further improve the feature representations.
We ﬁnd that the performance of SSD also degrades with lower temperature, suggesting the necessity of learning a good set of feature representation for effective outlier detection (Table 5).
How discriminative ability of feature representations evolves over the course of training.
We analyze this effect in Figure 3 where we compare both SSD and supervised training based detector over the course of training.
While discriminative ability of self-supervised training in SSD is lower at the start, it quickly catches up with supervised representations after half of the training epochs.
[Figure]
Performance of SSD improves with the amount of available unlabeled data.
A compelling advantage of unsupervised learning is to learn from unlabeled data, which can be easily collected.
As presented in Figure 2, we ﬁnd that performance of SSD increases with the size of training dataset.
We conduct another experiment with the STLdataset, where in addition to the 5k training images, we also use additional 10k images from the unlabeled set.
This improves the AUROC from 94.7 to 99.4 for CIFAR100 as the OOD dataset, further demonstrating the success of SSD in leveraging unlabeled data (Appendix B.4).
In conclusion, our framework provides an effective & ﬂexible approach for outlier detection using unlabeled data.
Acknowledgments We would like to thank Chong Xiang,
Liwei Song, and Arjun Nitin
Bhagoji for their helpful feedback on the paper.
This work was supported in part by the National Science Foundation under grants CNS-1553437 and CNS-1704105, by a
Qualcomm Innovation Fellowship, by the Army Research Ofﬁce Young Investigator Prize, by Army
Research Laboratory (ARL) Army Artiﬁcial Intelligence Institute (A2I2), by Ofﬁce of Naval Research (ONR) Young Investigator Award, by Facebook Systems for ML award, by Schmidt DataX Fund, and by Princeton E-fﬁliates Partnership.
A A DDITIONAL DETAILS ON EXPERIMENTAL SETUP A. T RAINING AND EVALUATION SETUP FOR DEEP NEURAL NETWORKS .
We use ResNet-50 architecture for all our major experiments and ResNet-18 for ablation studies.
We also provide results with ResNet-34 and ResNet-101 architecture.
We use a two-layer fully connected network as the projection header ( h ( . ) ).
To contrast with a large number of negatives, NT-Xent loss requires a much larger batch size compared to the supervised cross-entropy loss function.
We train it using a batch size of 512.
When evaluating self-supervised models, even when we incorporate labels in SSD we achieve the best performance when modeling in-distribution features with only a single cluster.
However, for supervised training, which refers to the supervised baseline in the paper, we ﬁnd that increasing the number of clusters helps.
For it we report the best of the results obtained from cluster indexes or using true labels of the data.
For each dataset we use the test set partition, if it exists, as the OOD dataset.
For consistent comparison we re-implement Softmax-probabilities (Hendrycks & Gimpel, 2017), ODIN (Liang et al., 2018) and Mahalanobis detector (Lee et al., 2018b) and evaluate their performance on the identical network, trained with supervised training for 500 epochs.
We set the perturbation budget to 0.0014 and temperature to 1000 for ODIN, since these are the most successful set of parameters reported in the original paper (Liang et al., 2018).
We primarily focus on one-shot and ﬁve-shot OOD detection, i.e., set k to one or ﬁve.
It implies access to one and ﬁve images, respectively, from each class of the targeted OOD dataset.
We create ten randomly transformed samples from each available OOD image in the SSD k detector.
With very small k , such as one, we ﬁnd that increasing number of transformations may degrade performance in some cases.
In this case we simply resort to using only one transformation per sample.
To avoid any hyperparameter selection, we set the image augmentation pipeline to be the same as the one used in training.
Finally, we use the Ledoit-Wolf method (Ledoit & Wolf, 2004) to estimate the covariance of the OOD samples.
A. P ERFORMANCE METRICS FOR OUTLIER DETECTORS
We use the following three performance metrics to evaluate the performance of outlier detectors.
• FPR at TPR=95%.
It refers to the false positive rate (= FP / (FP+TN)), when true positive rate (= TP / (TP + FN)) is equal to 95%.
Effectively, its goal is to measure what fraction of outliers go undetected when it is desirable to have a true positive rate of 95%.
• AUROC.
It refers to the area under the receiver operating characteristic curve.
We measure it by calculating the area under the curve when we plot TPR against FPR.
• AUPR.
It refers to the area under the precision-recall curve, where precision = TP / (TP+FP) and recall =
TP / (TP+FN).
Similar to AUROC, AUPR is also a threshold independent metric.
A. D ATASETS USED IN THIS WORK
We use the following datasets in this work.
Whenever there is a mismatch between the resolution of images in in-distribution and out-of-distribution (OOD) data, we appropriately scale the OOD images with bilinear scaling.
When there is an overlap between the classes of the in-distribution and OOD dataset, we remove the common classes from the OOD dataset.
• CIFAR-10 (Krizhevsky et al., 2009) .
It consists of 50,000 training images and 10,000 test images from 10 different classes.
Each image size is 32 × 32 pixels.
• CIFAR-100 (Krizhevsky et al., 2009) .
CIFAR-100 also has 50,000 training images and 10,000 test images.
However, it has 100 classes which are further organized in 20 sub-classes.
Note that its classes aren’t identical to the CIFAR-10 dataset, with a slight exception with class truck in CIFAR-10 and pickup truck in CIFAR-100.
However, their classes share multiple similar semantics, making it hard to catch outliers from the other dataset.
• SVHN (Netzer et al., 2011) .
SVHN is a real-world street-view housing number dataset.
It has 73,257 digits available for training, and 26,032 digits for testing.
Similar to the CIFAR-10/100 dataset, the size of its images is also 32 × 32 pixels.
• STL-10 (Coates et al., 2011) .
STL-10 has identical classes as the CIFAR-10 dataset but focuses on the unsupervised learning.
It has 5,000 training images, 8,000 test images, and a set of 100,000 unlabeled images.
Unlike the previous three datasets, the size of its images is 96 × 96 pixels.
• DTD (Cimpoi et al., 2014) .
Describable Textures Dataset (DTD) is a collection of textural images in the wild.
It includes a total of 5,640 images, split equally between 47 categories where the size of images range between 300 × 300 and 640 × 640 pixels.
• ImageNet 3 (Deng et al., 2009) .
ImageNet is a large scale dataset of 1,000 categories with 1.2 Million training images and 50,000 validation images.
It has high diversity in both inter- and intra-class images and is known to have strong generalization properties to other datasets.
• Blobs.
Similar to Hendrycks et al.
(2019a)
, we algorithmically generate these amorphous shapes with deﬁnite edges.
• Gaussian Noise.
We generate images with Gaussian noise using a mean of 0.5 and a standard deviation of 0.25.
We clip the pixel value to the valid pixel range of [0, 1].
• Uniform Noise.
It refers to images where each pixel value is uniformly sampled from the [0, 1] range.
B A DDITIONAL EXPERIMENTAL RESULTS B. L IMITATIONS OF OUTLIER DETECTORS BASED ON SUPERVISED TRAINING
Existing supervised training based detector assumes that ﬁne-grained data labels are available for the training data.
What happens to the performance of current detectors if we relax this assumption by assuming that only coarse labels are present.
We simulate this setup by combining consecutive classes from the CIFAR-10 dataset into two groups, referred to as CIFAR-2, or ﬁve groups referred to as CIFAR-5.
We use CIFAR-100 as the out-of-distribution dataset.
We ﬁnd that the performance of existing detectors degrades signiﬁcantly when only coarse labels are present (Figure 4).
In contrast,
SSD operates on unlabeled data thus doesn’t suffer from similar performance degradation
[Figure] [Figure]
[Figure]
B. O N CHOICE OF NUMBER OF CLUSTERS
We ﬁnd that the choice of optimal number of clusters is dependent on which layer we use as the feature extractor in a Residual Neural network.
We demonstrate this trend in Figure 5, with CIFARas in-distribution dataset and CIFAR-100 as out-of-distribution dataset.
We extract features from the last layer of each block in the residual network and measure the SSD performance with them.
While for the ﬁrst three blocks, we ﬁnd an increase in AUROC with number of clusters, the trend is reversed for the last block (Figure 5).
Since last block features achieve highest detection performance, we model in-distribution features using a single cluster.
B. A BLATION STUDY FOR FEW - SHOT OOD DETECTION
For few shot OOD detection, we ablate along the number of transformations used for each sample.
We choose CIFAR-100 as in-distribution and CIFAR-10 as OOD dataset with SSD k , set k to ﬁve, and choose ResNet-18 network architecture.
When increasing number of transformations from 1, 5, 10, 20, 50 the AUROC of detector is 74.3, 75.7, 76.1, 76.3, 76.7.
To achieve a balance between the performance and computational cost, we use ten transformations for each sample in our ﬁnal experiments.
B. P ERFORMANCE OF SSD IMPROVES WITH AMOUNT OF UNLABELED DATA
With easy access to unlabeled data, it is compelling to develop detectors that can beneﬁt from the increasing amount of such data.
We earlier demonstrated this ability of SSD for the CIFAR-10 dataset in Figure 2.
Now we present similar results with the STL-10 dataset.
We ﬁrst train a self-supervised network, and an equivalent supervised network with 5,000 training images from the STL-10 dataset.
We refer to these networks by SSD-5k and Sup-5k, respectively.
Next, we include additional 10, images from the available 100k unlabeled images in the dataset.
As we show in Figure 6, SSD is able to achieve large gains in performance with access to the additional unlabeled training data.
B. R ESULTS WITH DIFFERENT PERFORMANCE METRICS
We provide our detailed experimental results for each component in the SSD framework with three different performance metrics in Table 6, 7,.
[Table]
T a b l e 7 : E xp e r
i m e n t a l r e s u lt s o f
SS D d e t ec t
o r w it
h m u lti p l e m e t r i c s ov e r
C
I F
A R10 ,
C
I F
A R100 , a nd
S TLd a t a s e t .
I nd i s t
r
i bu ti on
OOD
FP R ( T
P R = 95 % ) ↓
A U R
O C ↑
AU P R ↑
SS D
S up e r v
i s e d
SS D k
SS D +
SS D k +
SS D
S up e r i v s e d
SS D k
SS D +
SS D k +
SS D
S up e r v
i s e d
SS D k
SS D +
SS D k + k =
k = k =
k = k =
k = k =
k = k =
k = k =k =
C
I F
A
RC
I F A
R50 .47 .
S
VHN 2 .1 .0 .1 .0 .0 .
T e x t u r e 14 .
B l ob s 4 .0 .4 .0 .0 .0 .
C
I F
A
RC
I F A
R89 .96 .85 .69 .89 .
S
VHN 20 .28 .2 .4 .
T e x t u r e 65 .
B l ob s 1 .95 .1 .0 .3 .0 .
[Table]
S TLC
I F A
R29 .73 .32 .
[Table]
S
VHN 6 .26 .5 .
[Table] [Table]
T e x t u r e
B l ob s 16 .88 .14 .