S IMULTANEOUS C LASSIFICATION AND O UT - OF -D ISTRIBUTION D ETECTION U SING D EEP N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep neural networks have achieved great success in classiﬁcation tasks during the last years.
However, one major problem to the path towards artiﬁcial intelligence is the inability of neural networks to accurately detect samples from novel class distributions and therefore, most of the existent classiﬁcation algorithms assume that all classes are known prior to the training stage.
In this work, we propose a methodology for training a neural network that allows it to efﬁciently detect outof-distribution (OOD) examples without compromising much of its classiﬁcation accuracy on the test examples from known classes.
Based on the Outlier Exposure (OE) technique, we propose a novel loss function that achieves state-of-the-art results in out-of-distribution detection with OE both on image and text classiﬁcation tasks.
Additionally, we experimentally show that the combination of our method with the Mahalanobis distance-based classiﬁer achieves state-of-the-art results in the OOD detection task.
I NTRODUCTION
Modern neural networks have recently achieved superior results in classiﬁcation problems (Krizhevsky et al., 2012;
He et al., 2016).
However, most of the classiﬁcation algorithms proposed so far make the assumption that data generated from all the class conditional distributions are available during training time i.e., they make the closed-world assumption.
In an open world environment (Bendale & Boult, 2015), where examples from novel class distributions might appear during test time, it is necessary to build classiﬁers that are able to detect OOD examples while having high classiﬁcation accuracy on known class distributions.
It is generally known that deep neural networks can make predictions for out-of-distribution (OOD) examples with high conﬁdence (Nguyen et al., 2015).
High conﬁdence predictions are undesirable since they consist a symptom of overﬁtting (Szegedy et al., 2015).
They also make the calibration of neural networks difﬁcult.
Guo et al.
(2017) observed that modern neural networks are miscalibrated by experimentally showing that the average conﬁdence of deep neural networks is usually much higher than their accuracy.
A simple yet effective method to address the problem of the inability of neural networks to detect
OOD examples is to train them so that they make highly uncertain predictions for examples generated by novel class distributions.
In order to achieve that, Lee et al.
(2018a) deﬁned a loss function based on the Kullback-Leibler (KL) divergence metric to minimize the distance between the output distribution given by softmax and the uniform distribution for samples generated by a GAN (Goodfellow et al., 2014).
Using a similar loss function, Hendrycks et al.
(2019) showed that the technique of Outlier Exposure (OE) that draws anomalies from a real and diverse dataset can outperform the
GAN framework for OOD detection.
Using the OE technique, our main contribution is threefold:
• We propose a novel loss function consisting of two regularization terms.
The ﬁrst regularization term minimizes the l 1 norm between the output distribution given by softmax and the uniform distribution which constitutes a distance metric between the two distributions (Deza & Deza, 2009).
The second regularization term minimizes the Euclidean distance
between the training accuracy of a DNN and its average conﬁdence in its predictions on the training set.
• We experimentally show that the proposed loss function outperforms the previous work of Hendrycks et al.
and achieves state-of-the-art results in OOD detection with OE both on image and text classiﬁcation tasks.
• We experimentally show that our proposed method can be combined with the Mahalanobis distance-based classiﬁer (Lee et al., 2018b).
The combination of the two methods outperforms the original Mahalanobis method in all of the experiments and to the best of our knowledge, achieves state-of-the-art results in the OOD detection task.
R ELATED W ORK
Yu et al.
(2017) used the GAN framework (Goodfellow et al., 2014) to generate negative instances of seen classes by ﬁnding data points that are close to the training instances but are classiﬁed as fake by the discriminator.
Then, they used those samples in order to train SVM classiﬁers to detect examples from unseen classes.
Similarly, Kliger & Fleishman (2018) used a multi-class GAN framework in order to produce a generator that generates a mixture of nominal data and novel data and a discriminator that performs simultaneous classiﬁcation and novelty detection.
Hendrycks & Gimpel (2017) proposed a baseline for detecting misclassiﬁed and out-of-distibution examples based on their observation that the prediction probability of out-of-distribution examples tends to be lower than the prediction probability for correct examples.
Recently, Corbi ` ere et al.
(2019) also studied the problem of detecting overconﬁdent incorrect predictions.
A single-parameter variant of Platt scaling (Platt, 1999), temperature scaling, was proposed by Guo et al.
(2017) for calibration of modern neural networks.
For image data, based on the idea of Hendrycks & Gimpel (2017), Liang et al.
(2018) observed that simultaneous use of temperature scaling and small perturbations at the input can push the softmax scores of in- and out-of-distribution images further apart from each other, making the out-of-distribution images distinguishable.
Lee et al.
(2018a) generated
GAN examples and forced the neural network to have lower conﬁdence in predicting their classes.
Hendrycks et al.
(2019) substituted the GAN samples with a real and diverse dataset using the technique of OE.
Similar works (Malinin & Gales, 2018; Bevandi ´ c et al., 2018) also force the model to make uncertain predictions for OOD examples.
Using an ensemble of classiﬁers, Lakshminarayanan et al.
(2017) showed that their method was able to express higher uncertainty in OOD examples.
Liu et al.
(2018) provided theoretical guarantees for detecting OOD examples under the assumption that an upper bound of the fraction of OOD examples is available.
Under the assumption that the pre-trained features of a softmax neural classiﬁer can be ﬁtted well by a class-conditional Gaussian distribution, Lee et al.
(2018b) deﬁned a conﬁdence score using the
Mahalanobis distance that can efﬁciently detect abnormal test samples.
As also mentioned by Lee et al. (2018b), Euclidean distance can also be used but with less efﬁciency.
We prefer to call these methods Distance-Based Post-Training (DBPT) methods for OOD detection.
S IMULTANEOUS C LASSIFICATION AND O UT - OF -D ISTRIBUTION D ETECTION
We consider the multi-class classiﬁcation problem under the open-world assumption (Bendale &
Boult, 2015), where samples from some classes are not available during training.
Our task is to design deep neural network classiﬁers that can achieve high accuracy on examples generated by a learned probability distribution called D in while at the same time, they can effectively detect examples generated by a different probability distribution called D out during the test phase.
The examples generated by D in are called in-distribution while the examples generated by D out are called outof-distribution (OOD).
Adopting the idea of Outlier Exposure (OE) proposed by Hendrycks et al.
(2019), we train the neural network using training examples sampled from D in and D OEout .
During the test phase, we evaluate the OOD detection capability of the neural network using examples sampled from D testout , where D OEout and D testout are disjoint.
Lee et al.
(2018a)
and Hendrycks et al.
(2019) used the KL divergence metric in order to minimize the distance between the output distribution produced by softmax for the OOD examples and the uniform distribution.
In our work, we choose to minimize the l 1 norm between the two distributions which has shown great success in machine learning applications.
Viewing the knowledge of a model as the class conditional distribution it produces over outputs given an input (Hinton et al., 2015), the entropy of this conditional distribution can be used as a regularization method that penalizes conﬁdent predictions of a neural network (Pereyra et al., 2017).
In our approach, instead of penalizing the conﬁdent predictions of posterior probabilities yielded by a neural network, we force it to make predictions for examples generated by D in with an average conﬁdence close to its training accuracy.
In such a manner, not only do we make the neural network avoid making overconﬁdent predictions, but we also take into consideration its calibration (Guo et al., 2017).
Let us consider a classiﬁcation model that can be represented by a parametrized function f θ , where θ stands for the vector of parameters in f θ .
Without loss of generality, assume that the cross entropy loss function is used during training.
We propose the following constrained optimization problem for ﬁnding θ :
[Equation]
where L CE is the cross entropy loss function and K is the number of classes available in D in .
Even though the constrained optimization problem (1) can be used for training various classiﬁcation models, for clarity we limit our discussion to deep neural networks.
Let z denote the vector representation of the example x ( i ) in the feature space produced by the last layer of the deep neural network (DNN) and let A tr be the training accuracy of the DNN.
Observe that the optimization problem (1) minimizes the cross entropy loss function subject to two additional constraints.
The ﬁrst constraint forces the average maximum prediction probabilities calculated by the softmax layer towards the training accuracy of the DNN for examples sampled from D in , while the second constraint forces the maximum probability calculated by the softmax layer towards 1 K for all examples sampled from the probability distribution D OEout .
In other words, the ﬁrst constraint makes the DNN predict examples from known classes with an average conﬁdence close to its training accuracy, while the second constraint forces the DNN to be highly uncertain for examples of classes it has never seen before by producing a uniform distribution at the output for examples sampled from the probability distribution D OEout .
It is also worth noting that the ﬁrst constraint of (1) uses the training accuracy of the neural network A tr which is not available in general.
To handle this issue, one can train a neural network by only minimizing the cross entropy loss function for a few number of epochs in order to calculate A tr and then ﬁne-tune it using (1).
Because solving the nonconvex constrained optimization problem described by (1) is extremely difﬁcult, let us introduce Lagrange multipliers (Boyd & Vandenberghe, 2004) and convert it into the following unconstrained optimization problem: minimize
[Equation]
where it is worth mentioning that in (2), we used only one Lagrange multiplier for the second set of constraints in (1) instead of using one for each constraint in order to avoid introducing a large number of hyperparameters to our loss function.
This modiﬁcation is a special case where we consider the Lagrange multiplier λ 2 to be common for each individual constraint involving a different x ( i ) ∼ D OEout .
Note also that according to the original Lagrangian theory, one should optimize the objective function of (2) both with respect to θ ,λ 1 and λ 2 but as it commonly happens in machine learning applications, we approximate the original problem by calculating appropriate values for λ 1 and λ 2 through a validation technique (Hastie et al., 2001).
After converting the constrained optimization problem (1) into an unconstrained optimization problem as described by (2), it is possible that at each training epoch, the maximum prediction probability produced by softmax for each example drawn from D OEout changes, introducing difﬁculties in making the DNN produce a uniform distribution at the output for those examples.
For instance, assume that we have a K -class classiﬁer with K = 3 and at epoch t n , the maximum prediction probability produced by softmax for an example x ( i ) ∼ D OEout corresponds to the second class.
Then, the last term of (2) will push the prediction probability of example x
( i ) for the second class towards 13 while concurrently increasing the prediction probabilities for either the ﬁrst class or the third class or both.
At the next epoch t n +1 , it is possible that the prediction probability for either the ﬁrst class or the third class becomes the maximum among the three and hence, the last term of (2) will push that one towards 13 by possibly increasing again the prediction probability for the second class.
It becomes obvious that this process introduces difﬁculties in making the DNN produce a uniform distribution at the output for examples sampled from D OEout .
However, this issue can be resolved by concurrently pushing all the prediction probabilities produced by the softmax layer for examples drawn from
D OE out towards 1 K .
Additionally, in order to prevent the second and the third term of (2) from taking negative values during training, let us convert (2) into the following:
[Equation]
(cid:33)(cid:35)(cid:33)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
The second term of the the loss function described by (3) minimizes the squared distance between the training accuracy of the DNN and the average conﬁdence in its predictions for examples drawn from D in .
Additionally, the third term of (3) minimizes the l 1 norm between the uniform distribution and the distribution produced by the softmax layer for the examples drawn from D OEout .
While converting the unconstrained optimization problem (2) into (3), one could use several combinations of norms to minimize.
However, we found that minimizing the squared distance between the training accuracy of the DNN and the average conﬁdence in its predictions for examples drawn from
D in and the l 1 norm between the uniform distribution and the distribution produced by the softmax layer for the examples drawn from D OEout works best.
This is because l 1 norm uniformly attracts all the prediction probabilities produced by softmax to the desired value 1 K , better contributing to producing a uniform distribution at the output of the DNN for the examples drawn from D OEout .
On the other hand, minimizing the squared distance between the training accuracy of the DNN and the average conﬁdence in its predictions for examples drawn from D in emphasizes more on attracting the maximum softmax probabilities that are further away from the average conﬁdence of the DNN, making the neural network better detect in- and out-of-distribution examples at the low softmax probability levels.
E XPERIMENTS
During the experiments, we observed that if we start training the DNN with a relatively high value of λ 1 , the learning process might slow down since we constantly force the neural network to make predictions with an average conﬁdence close to its training accuracy.
Therefore, it is recommended to split the training of the algorithm into two stages where in the ﬁrst stage, we train the DNN using only the cross entropy loss function until it reaches the desired level of accuracy A tr
and then using a ﬁxed A tr , we ﬁne-tune it using the combined loss function given by (3).
4. C OMPARISON WITH S TATE - OF - THE -A RT IN OE
The experimental setting is as follows.
We draw samples from D in and we train the DNN until it reaches the desired level of accuracy A tr .
Then, drawing samples from D OEout , we ﬁne-tune it using the combined loss function given by (3).
During the test phase, we evaluate the OOD detection capability of the DNN using examples from D testout which is disjoint from D OEout .
We demonstrate the effectiveness of our method in both image and text classiﬁcation tasks by comparing it with the previous OOD detection with OE method proposed by Hendrycks et al.
A part of our experiments was based on the publicly available code of Hendrycks et al.
4.1. E VALUATION M ETRICS
Our OOD detection method belongs to the class of Maximum Softmax Probability (MSP) detectors (Hendrycks & Gimpel, 2017) and therefore, we adopt the evaluation metrics used in Hendrycks et al.
Deﬁning the OOD examples as the positive class and the in-distribution examples as the negative class, the performance metrics associated with OOD detection are the following:
• False Positive Rate at N % True Positive Rate ( FPRN ):
This performance metric (Balntas et al., 2016; Kumar et al., 2016) measures the capability of an OOD detector when the maximum softmax probability threshold is set to a predeﬁned value.
More speciﬁcally, assuming N % of OOD examples need to be detected during the test phase, we calculate a threshold in the softmax probability space and given that threshold, we measure the false positive rate,
i
e. the ratio of indistribution examples that are incorrectly classiﬁed as OOD.
• Area Under the Receiver Operating Characteristic curve (AUROC): In the out-of-distribution detection task, the ROC curve (Davis & Goadrich, 2006) summarizes the performance of an OOD detection method for varying threshold values.
• Area Under the Precision-Recall curve (AUPR): The AUPR
(Manning & Sch ¨ utze, 1999) is an important measure when there exists a class-imbalance between OOD and in-distribution examples in a dataset.
As in Hendrycks et al.
(2019), in our experiments, the ratio of OOD and in-distribution test examples is 1:5.
4.1. I MAGE C LASSIFICATION E XPERIMENTS
Results.
The results of the image classiﬁcation experiments are shown in Table 1.
In Figure 1, as an example, we plot the histogram of softmax probabilities using CIFAR-10 as D in and
Places365 as D testout .
The detailed description of the image datasets used in the image OOD detection experiments is presented in Appendix A.2.
Network Architecture and Training Details.
Similar to
Hendrycks et al.
(2019), for CIFAR 10 and CIFAR 100 experiments, we used 40-2 wide residual networks (WRNs) proposed by
Zagoruyko & Komodakis (2016).
We initially trained the WRN for 100 epochs using a cosine learning rate (Loshchilov & Hutter, 2017) with an initial value 0.1, a dropout rate of 0.3 and a batch size of 128.
As in Hendrycks et al.
(2019), we also used Nesterov momentum and l 2 weight regularization with a decay factor of 0.0005.
For CIFAR 10, we ﬁne-tuned the network forepochs minimizing the loss function given by (3) using a learning rate of 0.001, while for the CIFAR 100 the corresponding number of epochs was 20.
For the SVHN experiments, we trained 16WRNs using a learning rate of 0.01, a dropout rate of 0.4 and a batch size of 128.
We then ﬁne-tuned the network for 5 epochs using a learning rate of 0.001.
During ﬁne-tuning, the 80 Million
Tiny Images dataset was used as D OEout .
The values of the hyperparameters λ 1 and λ 2 were chosen in the range [0 . 03 , 0 . 09] using a separate validation dataset D valout similar to Hendrycks et al.
We note that D valout and D testout are disjoint.
The data used for validation are presented in Appendix A.3.
[Table]
Contribution of each regularization term.
To demonstrate the effect of each regularization term of the loss function described by (3) in the OOD detection task, we ran some additional image classiﬁcation experiments which are presented in Table 2.
For these experiments, we incrementally added each regularization term to the loss function described by (3) and we measured its effect both in the OOD detection evaluation metrics as well as in the accuracy of the DNN on the test images of
D in .
The results of these experiments validate that the combination of the two regularization terms of (3) not only improves the OOD detection performance of the DNN but also improves its accuracy on the test examples of D in compared to the case where λ 1 = 0 .
Table 2 also demonstrates that our method can signiﬁcantly improve the OOD detection performance of the DNN compared to the case where only the cross-entropy loss is minimized at the expense of only an insigniﬁcant degradation in the test accuracy of the DNN on examples generated by D in
[Table]
in
[Table]
4.1. T EXT C LASSIFICATION E XPERIMENTS
Results.
The results of the text classiﬁcation experiments are shown in Table 3.
The detailed description of the text datasets used in the NLP OOD detection experiments is presented in Appendix B.1.
FPR90 ↓
[Table]
↑
[Table]
D in +
[Table]
Network Architecture and Training Details.
For all text classiﬁcation experiments, similar to
Hendrycks et al.
(2019), we train 2-layer GRUs (Cho et al., 2014) for 5 epochs with learning rate 0.01 and a batch size of 64 and then we ﬁne-tune them for 2 epochs using the loss function given by (3).
During ﬁne-tuning, the WikiText-2 dataset was used as D OEout .
The values of the hyperparameters λ 1 and λ 2 were chosen in the range [0 . 04 , 0 . 1] using a separate validation dataset as described in
Appendix B.2.
4. A C OMBINATION OF OE AND DBPT M ETHODS FOR OOD D ETECTION
Lee et al.
(2018b) proposed a DBPT method for OOD detection that can be applied to any pretrained softmax neural classiﬁer.
Under the assumption that the pre-trained features of a DNN can be ﬁtted well by a class-conditional Gaussian distribution, they deﬁned the conﬁdence score using the Mahalanobis distance with respect to the closest class-conditional probability distribution, where its parameters are chosen as empirical class means and tied empirical covariance of training samples (Lee et al., 2018b).
To further distinguish in- and out-of-distribution examples, they proposed two additional techniques.
In the ﬁrst technique, they added a small perturbation before processing each input example to increase the conﬁdence score of their method.
In the second technique, they proposed a feature ensemble method in order to obtain a better calibrated score.
The feature ensemble method extracts all the hidden features of the DNN and computes their empirical class mean and tied covariances.
Subsequently, it calculates the Mahalanobis distance-based conﬁdence score for each layer and ﬁnally calculates the weighted average of these scores by training a logistic regression detector using validation samples in order to calculate the weight of each layer at the ﬁnal conﬁdence score.
Since the Mahalanobis distance-based classiﬁer proposed by Lee et al.
(2018b) is a post-training method, it can be combined with our proposed loss function described by (3).
More speciﬁcally, in our experiments, we initially trained a DNN using the standard cross entropy loss function and then we ﬁne-tuned it with the proposed loss function given by (3).
After ﬁne-tuning, we applied the Mahalanobis distance-based classiﬁer and we compared the obtained results against the results presented in Lee et al. (2018b).
The simulation experiments on image classiﬁcation tasks show that the combination of our method
which
belongs to the OE
“family”
of methods and the Mahalanobis
distance-based classiﬁer which
belongs
to the
“family”
of
DBPT methods
achieves state
-of-
the
art results in the
OOD detection task.
A part of our experiments was based on the publicly available code of Lee et al.
(2018b).
4.2. E VALUATION M ETRICS
To demonstrate the adaptability of our method, in these experiments, we adopt the OOD detection evaluation metrics used in Lee et al.
(2018b).
• True Negative Rate at N % True Positive Rate ( TNRN ):
This performance metric measures the capability of an OOD detector to detect true negative examples when the true positive rate is set to 95% .
• Area Under the Receiver Operating Characteristic curve (AUROC): In the out-of-distribution detection task, the ROC curve (Davis & Goadrich, 2006) summarizes the performance of an OOD detection method for varying threshold values.
• Detection Accuracy
(DAcc): As also mentioned in Lee et al.
(2018b), this evaluation metric corresponds to the maximum classiﬁcation probability over all possible thresholds (cid:15) : 1 − min (cid:15) { D in ( q ( x ) ≤ (cid:15) )
P ( x is from D in )
+ D out ( q ( x ) > (cid:15) )
P ( x is from D out ) } , where q ( x ) is a conﬁdence score.
Similar to Lee et al.
(2018b), we assume that P ( x is from D in ) =
P ( x is from D out ) .
4.2. E XPERIMENTAL S ETUP
To demonstrate the adaptability and the effectiveness of our method, we adopt the experimental setup of Lee et al.
(2018b).
We train ResNet (He et al., 2016) with 34 layers using CIFAR-10, CIFARand SVHN datasets as D in .
For the CIFAR experiments, SVHN, TinyImageNet (a sample of 10, images drawn from the ImageNet dataset) and LSUN are used as D testout .
For the SVHN experiments,
CIFAR-10, TinyImageNet and LSUN are used as D testout .
Both TinyImageNet and LSUN images are downsampled to 32 × 32 .
Similar to Lee et al.
(2018b), for the Mahalanobis distance-based classiﬁer, we train the ResNet model for 200 epochs with batch size 128 by minimizing the cross entropy loss using the SGD
[Table]
algorithm with momentum 0.9.
The learning rate starts at 0.1 and is dropped by a factor of 10 at 50% and 75% of the training progress, respectively.
Subsequently, we compute the Mahalanobis distancebased conﬁdence score using both the input pre-processing and the feature ensemble techniques.
The hyper-parameters that need to be tuned are the magnitude of the noise added at each test input example as well as the layer indexes for feature ensemble.
Similar to Lee et al.
(2018b), both of them are tuned using a separate validation dataset consisting of both in- and out-of-distribution data.
Since the Mahalanobis distance-based classiﬁer belongs to the “family” of DBPT methods for OOD detection tasks, it can be combined with our proposed method.
More speciﬁcally, we initially train the ResNet model with 34 layers for 200 epochs using exactly the same training details as mentioned above.
Subsequently, we ﬁne-tune the network with the proposed loss function described by (3) using the 80 Million Tiny Images as D OEout .
During ﬁne-tuning, we use the SGD algorithm with momentum 0.9 and a cosine learning rate (Loshchilov & Hutter, 2017) with an initial value 0.
using a batch size of 128 for data sampled from D in and a batch size of 256 for data sampled from D OEout .
For CIFAR-10 and 100 experiments, we ﬁne-tuned the network for 30 and 20 epochs respectively, while for SVHN the corresponding number of epochs was 5.
The values of the hyperparameters λ [-
-]1 and λ 2 were chosen using a separate validation dataset consisting of both in- and out-of-distribution images similar to Lee et al.
(2018b).
The results are shown in Table 4.
Discussion.
The results in Table 4 demonstrate the effectiveness of our method when combined with the Mahalanobis distance-based classiﬁer since it outperforms the original version of the Mahalanobis method proposed by Lee et al.
(2018b) in all of the experiments.
This result validates the contribution of our technique further, since it does not only achieve state-of-the-art results in
OOD detection with OE, but it can be additionally combined with DBPT methods like the Mahalanobis distance-based classiﬁer to achieve state-of-the-art results in the OOD detection task.
The superior performance of our method when combined with the Mahalanobis distance-based classiﬁer can be justiﬁed by the fact that the latter extracts the learned features from the layer(s) of the DNN and it subsequently uses those features to deﬁne a conﬁdence score based on the Mahalanobis distance.
The simulation results presented in Table 1 and Table 3 showed that our method can teach the
DNN to learn feature representations that can further distinguish in- and out-of distribution data and therefore, the combination of the two methods improves the OOD detection capability of a DNN.
C ONCLUSION
In this paper, we proposed a method for simultaneous classiﬁcation and out-of-distribution detection.
The proposed loss function includes two regularization terms where the ﬁrst minimizes the l 1 norm between the output distribution of the softmax layer of a DNN and the uniform distribution, while the second minimizes the Euclidean distance between the training accuracy of a DNN and its average conﬁdence in its predictions on the training set.
Experimental results showed that the proposed loss function achieves state-of-the-art results in OOD detection with OE (Hendrycks et al., 2019) in both image and text classiﬁcation tasks.
Additionally, we experimentally showed that our method can be combined with DBPT methods for OOD detection like the Mahalanobis distance-based classiﬁer (Lee et al., 2018b) and achieves state-of-the-art results in the OOD detection task.
A E XPANDED I MAGE OOD DETECTION R ESULTS AND D ATASETS USED FOR C OMPARISON WITH S TATE - OF - THE -A RT IN OE A. I MAGE OOD DETECTION R ESULTS
[Table]
A. D in , D OEout AND D testout FOR I MAGE E XPERIMENTS
SVHN:
The Street View House Number (SVHN) dataset (Netzer et al., 2011) consists of 32 ×color images out of which 604,388 are used for training and 26,032 are used for testing.
The dataset has 10 classes and was collected from real Google Street View images.
Similar to Hendrycks et al.
(2019), we rescale the pixels of the images to be in [0 , 1] .
CIFAR 10:
This dataset (Krizhevsky & Hinton, 2009) contains 10 classes and consists of 60, 32 × 32 color images out of which 50,000 belong to the training and 10,000 belong to the test set.
Before training, we standardize the images per channel similar to Hendrycks et al.
CIFAR 100: This dataset (Krizhevsky & Hinton, 2009) consists of 20 distinct superclasses each of which contains 5 different classes giving us a total of 100 classes.
The total number of images in the dataset are 60,000 and we use the standard 50,000/10,000 train/test split.
Before training, we standardize the images per channel similar to Hendrycks et al.
80 Million Tiny Images: The 80 Million Tiny Images dataset (Torralba et al., 2008) was exclusively used in our experiments in order to represent D OEout .
It consists of 32 × 32 color images collected from the Internet.
Similar to Hendrycks et al.
, in order to make sure that D OEout and D testout are
Places365: Places365 dataset introduced by Zhou et al.
(2018) was exclusively used in our experiments in order to represent D testout .
It consists of millions of photographs of scenes.
Gaussian: A synthetic image dataset created by i.i.d.
sampling from an isotropic Gaussian distribution.
Bernoulli
: A synthetic image dataset
created by sampling from a Bernoulli distribution.
Blobs
A
synthetic dataset of images with deﬁnite edges
Icons-50: This dataset intoduced by Hendrycks & Dietterich (2018) consists of 10,000 images belonging to 50 classes of icons.
As part of preprocessing, we removed the class “Number” in order to make it disjoint from the SVHN dataset.
Textures: This dataset contains 5,640 textural images (Cimpoi et al., 2014).
LSUN: It consists of around 1 million large-scale images of scenes (Yu et al., 2015).
Rademacher: A synthetic image dataset created by sampling from a symmetric Rademacher distribution.
A. V ALIDATION D ATA FOR I MAGE E XPERIMENTS
Uniform Noise: A synthetic image dataset where each pixel is sampled from U [0 , 1] or U [ − 1 , 1] depending on the input space of the classiﬁer.
Arithmetic Mean
A synthetic image dataset created by randomly sampling a pair of in-distribution images and subsequently taking their pixelwise arithmetic mean.
Geometric Mean: A synthetic image dataset created by
randomly
sampling
a
pair of
in-
distribution images and
subsequently taking their pixelwise geometric mean.
Jigsaw:
A synthetic image dataset created by partitioning an image sampled from D in into
equally sized patches and by subsequently permuting those patches.
Speckle Noised: A synthetic image dataset created by applying speckle
noise
to
images sampled
from D in .
Inverted Images: A synthetic image
dataset
created by shifting
and reordering the color channels of
images
sampled
from D in
RGB Ghosted:
A synthetic image dataset created by inverting the color channels of images sampled from D in .
B E XPANDED T EXT OOD DETECTION R ESULTS AND D ATASETS USED FOR C OMPARISON WITH S TATE - OF - THE -A RT IN OE B. D in , D OEout AND D testout FOR NLP E XPERIMENTS
20 Newsgroups: This dataset contains 20 different newsgroups, each corresponding to a speciﬁc topic.
It contains around 19,000 examples and we used the standard 60/40 train/test split.
TREC:
A question classiﬁcation dataset containing around 6,000 examples from 50 different classes.
Similar to Hendrycks et al.
(2019), we used 500 examples for the test phase and the rest for training.
SST:
The Stanford Sentiment Treebank (Socher et al., 2013) is a binary classiﬁcation dataset for sentiment prediction of movie reviews containing around 10,000 examples.
WikiText-2: This dataset contains over 2 million articles from Wikipedia and is exclusively used as
D OEout in our experiments.
We used the same preprocessing as in Hendrycks et al.
in order to have a valid comparison.
SNLI:
The Stanford Natural Language Inference (SNLI
) corpus
is
a
collection of
000 human
- written English sentence pairs (Bowman et al., 2015).
IMDB:
A sentiment classiﬁcation dataset containing movies reviews.
Multi30K: A dataset of English and German descriptions of images (Elliott et al., 2016).
For our experiments, only the English descriptions were used.
WMT
: A dataset used for machine translation tasks.
For our experiments, only the English part of the test set was used.
Yelp
: A dataset containing reviews of users for businesses on Yelp.
EWT: The English Web Treebank (EWT) consists of 5 different datasets: weblogs (EWT-W), newsgroups (EWT-N), emails (EWT-E), reviews (EWT-R) and questions-answers (EWT-A).
B. V ALIDATION D ATA FOR NLP E XPERIMENTS
The validation dataset D valout used for the NLP OOD detection experiments was constructed as follows.
For each D in dataset used, we used the rest two in-distribution datasets as D valout .
For instance, during the experiments where 20 Newsgroups represented D in , we used TREC and SST as D valout making sure that D valout and D testout are disjoint.
B. T EXT OOD DETECTION R ESULTS
[Table]
EWT-W 2.
Mean 4.
T R E
C
SNLI 4.
IMDB 0.
Multi30K 0.
WMT 0.
Yelp 0.
EWT-A 0.
EWT-E 0.
EWT-N 0.
EWT-R 0.
EWT-W 0.
Mean 0.
SS T
SNLI 33.
IMDB 32.
Multi30K 33.
WMT 17.
Yelp 11.
[Table]
C T RAINING D ETAILS FOR THE E XPERIMENTAL R ESULTS FOR C OMPARISON WITH M AHALANOBIS DISTANCE - BASED CLASSIFIER
During ﬁne-tuning with our proposed loss function given by (3)
[Table]
we used the training details presented in Table 7.
{+The values of the hyper-parameters λ 
+}
and λ 2 were chosen using a separate validation dataset consisting of both in- and out-of-distribution images similar to Lee et al.
(2018b).
[Table]