N ATURAL A TTRIBUTE - BASED S HIFT D ETECTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite the impressive performance of deep networks in vision, language, and healthcare, unpredictable behaviors on samples from the distribution different than the training distribution cause severe problems in deployment.
For better reliability of neural-network-based classifiers, we define a new task, natural attributebased shift (NAtS) detection, to detect the samples shifted from the training distribution by some natural attribute such as age of subjects or brightness of images.
Using the natural attributes present in existing datasets, we introduce benchmark datasets in vision, language, and medical domain for NAtS detection.
Further, we conduct an extensive evaluation of prior representative out-of-distribution (OOD) detection methods on NAtS datasets and observe an inconsistency in their performance.
To understand this, we provide an analysis on the relationship between the location of NAtS samples in the feature space and the performance of distanceand confidence-based OOD detection methods.
Based on the analysis, we split NAtS samples into three categories and further suggest a simple modification to the training objective to obtain an improved OOD detection method that is capable of detecting samples from all NAtS categories.
I NTRODUCTION
Deep learning has significantly improved the performance in various domains such as computer vision, natural language processing, and healthcare (Bojarski et al., 2016; Mikolov et al., 2013; Esteva et al., 2016).
However, it has been reported that the deep classifiers make unreliable predictions on samples drawn from a different distribution than the training distribution (Amodei et al., 2016;
Hendrycks & Gimpel, 2017; Nguyen et al., 2015).
Detection of unreliable predictions is important to build a robust model but it is relatively hard when the test distribution is gradually shifting due to a natural attribute since it is difficult to determine whether the classifier fails when the shift is not significant.
These shifts occur in the real-world as a result of a change in specific attribute.
For example, a clinical text-based diagnosis classifier trained in 2021 will gradually encounter increasingly shifted samples as time flows, since writing styles change and new terms are introduced in time.
Detection of such samples is a vital task especially in safety-critical systems, such as autonomous vehicle control or medical diagnosis, where wrong predictions can lead to dire consequences.
To this end, we take a step forward by proposing a new task of detecting samples shifted by a natural attribute ( e.g. , age, time) that can easily be observed in the real-world setting.
We refer to such shifts as N atural At tribute-based S hifts (NAtS) , and the task of detecting them as NAtS detection.
Detection of NAtS is both different from, and also more challenging than out-of-distribution (OOD) detection (Hendrycks & Gimpel, 2017; Liang et al., 2018; Lee et al., 2018; Chandramouli & Sageev, 2020), which typically evaluates the detection methods with a clearly distinguished in-distribution (ID) samples and OOD samples ( e.g. , CIFAR10 as ID and SVHN as OOD, which have disjoint labels).
In contrast, we aim to detect samples from a natural attribute-based shift within the same label space.
Since NAtS samples share more features with the ID than the typical OOD samples do, identifying the former is expected to be more challenging than the latter.
Although OOD detection has some relevance to NAtS detection, comprehensive evaluation of the existing OOD detection methods on the natural attribute-based shift is an unexplored territory.
Therefore, in this paper, we perform an extensive evaluation of representative OOD methods on NAtS samples.
Depending on the task environment, NAtS detection can be pursued in parallel to domain generalization (Seo et al., 2020; Gulrajani & Lopez-Paz, 2020;
Carlucci et al., 2019), which aims to overcome domain shifts ( e.g. , image classifier adapting to sketches, photos, art paintings, etc.).
For example, an X-ray-based diagnosis model should detect images of unusual brightness so that the X-ray ma- chine can be properly configured, and the diagnosis model can perform in the optimal setting.
In other cases, domain generalization can be preferred, such as when we expect the classifier to be deployed in a less controlled environment ( e.g. , online image classifier) for non-safety critical tasks.
In this paper, we formalize NAtS detection to enhance the reliability of real-world decision systems.
Since there exists no standard dataset for this task, we create a new benchmark dataset in the vision, text, and medical domain by adjusting the natural attributes ( e.g. , age, time, and brightness) of the ID dataset.
Then we conduct an extensive evaluation of representative confidence- and distance-based
OOD methods on our datasets and observe that none of the methods perform consistently across all NAtS datasets.
After a careful analysis on where NAtS samples reside in the feature space and its impact on the distance- and confidence-based OOD detection performance, we identify the root cause of the inconsistent performance.
Following this observation, we define three general NAtS categories based on two criteria: the distance between NAtS samples and the decision boundary, the distance between NAtS samples, and the ID data.
Finally, we conduct an additional experiment to demonstrate that a simple modification to the negative log-likelihood training objective can dramatically help the Mahalanobis detector (Lee et al., 2018), a distance-based OOD detection method, generalize to all NAtS categories.
We also compare our results with various baselines and show that our proposed modification outperforms the baselines and is effective across the three NAtS datasets.
In summary, the contributions of this paper are as follows:
• We define a new task, N atural
At tribute-based S hift detection (NAtS detection) , which aims to detect the samples from a distribution shifted by some natural attribute.
We create a new benchmark dataset and provide them to encourage further research on evaluating NAtS detection.
• To the best of our knowledge, this is the first work to conduct a comprehensive evaluation of the OOD detection methods on shifts based on natural attributes, and discover that none of the OOD methods perform consistently across all NAtS scenarios.
• We provide novel analysis based on the location of shifted samples in the feature space and the performance of existing OOD detection methods.
Based on the analysis, we split NAtS samples into three categories.
• We demonstrate that a simple yet effective modification to the training objective for deep classifiers enables consistent OOD detection performances for all NAtS categories.
N ATURAL A TTRIBUTE - BASED S HIFT D ETECTION
We now formalize a new task, NAtS detection, which aims to enhance the reliability of real-world decision systems by detecting samples from NAtS. We address this task in the classification problems.
Let D I =
{X , Y} denote the in-distribution data, which is composed of N training samples with inputs X = { x 1 , ..., x N } and labels Y =
{ y 1 , ..., y N } .
Specifically, x i ∈ R
d represents a d -dimensional input vector, and y i ∈ K represents its corresponding label where K = { 1 , ..., K } is a set of class labels.
The discriminative model f θ : X → Y learns with ID dataset D
I to assign label y i for each x i .
In the NAtS detection setting, we assume that an in-distribution sample consists of attributes, and some of the attributes can be shifted in the test time due to natural causes such as time, age, or brightness.
When a particular attribute A ( e.g. , age), which has a value of a ( e.g. , 16), is shifted by the degree of δ , the shifted distribution can be denoted as D A = a + δ S =
{X ′ , Y ′ } .
X ′
= { x ′ 1 , ..., x ′ M } and
Y ′ =
{ y ′ 1 , ..., y ′ M } represents the M shifted samples and labels respectively.
Importantly, in the
NAtS setting, although the test distribution is changed from the ID, the label space is preserved as K , which is the set of class labels in D I .
In the test time, the model f θ might encounter the sample x ′ from a shifted data D A
= a + δ S , and it should be able to identify that the attribute-shifted sample is not from the ID.
NA T S D ATASET D ESCRIPTION
In this section, we describe three benchmark datasets which have a controllable attribute for simulating realistic distribution shifts.
Since there exists no standard dataset for NAtS detection, we create new benchmark datasets using existing datasets by adjusting natural attributes in order to reflect real-world scenarios.
We carefully select datasets from vision, language, and medical domains containing natural attributes ( e.g. , year, age, and brightness), which allows us to naturally split the
- Brightness + - Age +
[Figure]
samples.
By grouping samples based on these attributes, we can induce natural attribute-based distribution shifts as described below.
Image.
We use the UTKFace dataset (Zhang et al., 2017) which consists of over 20 , 000 face images with annotations of age, gender, and ethnicity.
As shown in Figure 1, we can visually observe that the facial images vary with age.
Therefore, we set the 1 , 282 facial images of 26 years old age as D I .
For creating the NAtS dataset, we vary the age of UTKFace dataset.
To obtain an equal number of samples in the NAtS dataset, the age groups that has less than 200 images are merged into one group until it has 200 samples.
Finally, 15 groups D ageS are produced for the NAtS datasets, varying the ages from 25 to 1 ( i.e. , D age =25 S , D age =24 S , . . . , D age =1 S ).
Text.
We use the Amazon Review dataset (He & McAuley, 2016; McAuley et al., 2015) which contains product reviews from Amazon.
We consider the product category ”book” and group its reviews based on the year to reflect the distributional shift across time.
We obtain 9 groups with each group containing reviews from the year between 2005 and 2014 .
Then, the group with 24 , 000 reviews posted in 2005 is set as D I , and the groups with reviews after 2005 as D yearS ( i.e. ,
D year =2006 S , D year =2007 S , . . .
, D year =2014 S ).
Each D yearS group contains 1500 positive reviews andnegative reviews.
We observed that as we move ahead in time, the average length of a review gets shorter and it uses more adjectives than previous years.
Due to the space constraint, we provide a detailed analysis of the dataset in the Section B of the Appendix.
Medical.
We use the RSNA Bone Age dataset (Halabi et al., 2019), a real-world dataset that contains left-hand X-ray images of the patient, along with their gender and age ( 0 to 20 years).
We consider patients in the age group of 10 to 12 years for our dataset.
To reflect diverse X-ray imaging set-ups in the hospital, we varied the brightness factor between 0 and 4.5 and form 16 different dataset D
brightnessS ( i.e. , D brightness =0 .
0 S , D brightness =0 .
2 S , . . .
, D brightness =4 .
5 S ), and each group contains Xray images of 200 males and 200 females.
Figure 1 presents X-ray images with different levels of brightness with realistic and continuous distribution shifts.
In-distribution data D I is composed of 3 , 000 images of brightness factor 1 .
(unmodified images).
C AN OOD D ETECTION M ETHODS ALSO D ETECT NA T S?
In this section, we briefly discuss about OOD detection methods and conduct an extensive evaluation of OOD detection methods on our proposed benchmark datasets.
4. OOD D ETECTION M ETHODS
In this work, we use three widely-used post-hoc and modality-agnostic OOD detection methods.
We use maximum of softmax probability (MSP) (Hendrycks & Gimpel, 2017) and ODIN (Liang et al., 2018) as confidence-based OOD detection baselines, and Mahalanobis detector (Lee et al., 2018) as distance-based OOD detection baseline.
Note that ODIN and Mahalanobis detector assume the availability of OOD validation dataset to tune their hyperparameters.
However, for all our experiments, we use variants of the above methods that do not access the OOD validation dataset as
Hsu et al.
The exact equations and details of how each OOD detection method assigns
an
OOD score to a given sample is provided in Section A of the Appendix.
4. E XPERIMENTS AND R ESULTS
We now systematically evaluate the performance of the three OOD detection methods under NAtS.
We report the AUROC of all OOD detection methods averaged across five random seeds, evaluated for all NAtS datasets.
The details of the implementation are described in Section C of the Appendix.
Experimental Settings.
In image domain, we train a gender classification model on our UTKFace
NAtS dataset using ResNet18 model and the cross-entropy loss.
We use our Amazon Book Review
[Figure] NAtS dataset in text domain and train a 4-layer Transformer with the cross-entropy loss for the sentiment classification task.
Lastly, in medical domain, we use our RSNA Bone Age NAtS dataset and train a ResNet18 with cross-entropy loss to predict the gender given the hand X-ray image of the patient.
We then evaluate the trained model on the corresponding test set in image, text, medical domains, respectively.
Further, we evaluate the NAtS detection performance of representative OOD detection methods in image, text, and medical domains on their corresponding NAtS datasets, which gradually shift with age, year, and brightness, respectively.
Results.
We present the classification accuracy of the trained models on the ID test set in Table 1.
We observe that the models trained using the cross-entropy loss obtain high accuracy and perform well on their corresponding tasks.
We further demonstrate the effectiveness of the existing representative OOD detection methods on our benchmark datasets in Figure 2.
The higher AUROC indicates that more NAtS samples are distinguished from the ID samples using OOD detection methods.
To provide the reference point, we calculate the AUROC values where there is no shift in the attribute.
We observe that in the UTKFace NAtS dataset, samples are detected by ODIN and MSP, which are confidence-based methods, but not by Mahalanobis detector (Figure 2a).
In the Amazon Review dataset, NAtS samples are detected only by the Mahalanobis detector, while MSP and ODIN fail (Figure 2b).
Moreover, the scores of confidence-based methods are lower than 50 in AUROC.
Lastly, Figure 2c shows that inputs from NAtS in the RSNA Bone Age dataset are detected well by all three methods.
A NALYZING I NCONSISTENCY OF OOD D ETECTION M ETHODS
In this section, we first study the behavior of NAtS samples in three datasets using PCA visualization.
Then we analyze the inconsistent performance of the OOD detection methods by considering them in two categories, namely confidence-based and distance-based methods.
Lastly, based on the analysis, we conclude this section by defining three NAtS categories.
5. A NALYSIS OF THE LOCATION OF NA T S SAMPLES
As illustrated in Figure 3, we apply principal component analysis (PCA) on the feature representations obtained from the penultimate layer of the models to visualize the movement of NAtS samples as we monotonically increase the degree of attribute shift ( i.e. , age, year, and brightness).
Further,
Figure 4 presents the model’s prediction confidence across varying degrees of the attribute shift.
Image.
By gradually changing the age, NAtS samples move toward the space between the two clusters of ID samples ( i.e. , the decision boundary) as can be seen in Figure
[Top].
Further, Figure 4a demonstrates that confidence decrease as we increase the degree of attribute shift, which indicates that NAtS samples move close to the decision boundary.
Note that the majority of the NAtS samples still overlap with ID sample clusters as we change the age.
Text.
As shown in Figure
[Middle], NAtS samples gradually move away from the ID samples (and away from the decision boundary) as the year changes.
In contrast to the UTKFace dataset, the confidence gradually increases, as shown in Figure 4b, since the NAtS samples are getting far away from the decision boundary. [Table] [Figure] [Figure]
Medical.
Figure
[Bottom] demonstrates that when we increase the brightness, NAtS samples move to the middle of the two classes and also move towards the outer edge of the ID sample clusters.
Furthermore, as shown in Figure 4c, the relatively decreased confidence indicates that NAtS samples are placed near the decision boundary as we increase the brightness of the images.
5. C OMPARISON BETWEEN C ONFIDENCE - BASED AND D ISTANCE - BASED OOD D ETECTION Confidence-based methods.
Figure 2 and Figure 3 illustrate that confidence-based methods achieve high AUROC when the NAtS samples are near the decision boundary due to their low confidence.
In the image and medical domain, we observed the NAtS samples moving towards the decision boundary with increasing attribute shift, thus they are detected by the confidence-based methods.
In contrast, in the text domain, the high prediction confidence of the shifted NAtS samples causes the degradation in the AUROC of the confidence-based methods.
Therefore, we conclude that to effectively utilize the confidence-based methods in all three NAtS datasets, it is necessary to reduce the confidence of samples outside the ID, namely, enforce NAtS samples to move near the decision boundary, which is not always possible ( e.g., Amazon Review dataset).
Distance-based methods.
From Figure 2 and Figure 3, we observe that the distance-based OOD detection method ( i.e. , Mahalanobis Detector) achieves high AUROC when NAtS samples are sufficiently away from the ID samples.
In the text and medical domains, the Mahalanobis detector worked well since NAtS samples moved sufficiently away from the ID samples as the shift increased.
However, in the image domain, the method fails to detect NAtS samples because instead of deviating from the ID, they move intermediately between the classes.
Prior works (Luo et al., 2020; Liu et al., 2017) report that the cross-entropy loss cannot guarantee a sufficient inter-class distance.
In other words, representations do not need to be far from the decision boundary to lower the cross-entropy loss.
In this regard, we assume that the performance degradation of the Mahalanobis detector is caused by the cross-entropy loss learning latent features that are not separable enough to detect the NAtS samples located between the classes ( e.g. , Figure[Top]).
Specifically, if some classes are located nearby in the feature space, samples moving between classes ( i.e. , the case of the UTKFace dataset) will not be far from the ID.
Even though NAtS samples move away from one of the ID class cluster, they will gradually get closer to another ID class cluster.
[Figure]
Considering the performance of confidence-based and distance-based detection methods, we now divided NAtS into three categories based on two criteria: 1) whether the dataset is near the decision boundary or not; 2) whether they are close or far from the in-distribution dataset.
Since a dataset ( i e. , samples) far from the decision boundary and overlapping with the in-distribution dataset is not a distributionally shifted dataset, our work focuses on the remaining three cases that cover all possible scenarios of NAtS. Without loss of generality, we use a classification task with three classes as a motivating example depicted by Figure 5.
• NAtS category 1: This category comprises of NAtS samples that are located near the decision boundary, and between ID samples of different classes.
Purple Stars in Figure 5 represents samples from this category.
Such samples are easily detected by confidence-based methods, MSP and ODIN, but harder to be detected by Mahalanobis Detector, which is a distance-based method.
• NAtS category 2: This category consists of NAtS samples that are placed away from the decision boundary and the ID data.
For example, Gray Diamonds in Figure 5.
Mahalanobis detector regards such samples as NAtS, whereas confidence-based methods fail to detect them since NAtS samples have a higher prediction confidence ( i.e. , higher ID score) than ID samples.
• NAtS category 3: This category mainly comprises of NAtS samples located anywhere near the decision boundary but far away from ID data.
For example, Orange Triangles in Figure 5.
Such samples are easily detected by both distance-based and confidence-based OOD detection methods. [Figure]
[Table]
Decision Boundary Near Far Near In-distribution Near Far Far Confidence-based methods
[Table]
M ETHOD FOR C ONSISTENT NA T S D ETECTION P ERFORMANCE
In this section, we suggest a modification in the training objective for deep classifiers to encourage consistent NAtS detection performance on all NAtS categories.
Then we provide experiment results where the proposed method was compared against diverse OOD detection methods on the three NAtS datasets (UTKFace, Amazon Review, RSNA Bone Age).
6. M ETHOD
For a generally applicable OOD detection method to all NAtS categories, we suggest a new training objective for deep classifiers comprised of classification loss ( L CE ), distance loss ( L dist ) and entropy loss ( L entropy ) .
The proposed objective improves the performance of the Mahalanobis detector on NAtS samples from category 1 without sacrificing performance on NAtS samples from other categories.
We focus on improving the distance-based OOD detection method rather than the confidence-based method since it is not always possible to enforce NAtS samples to be embedded near the decision boundary, as discussed in Section 5.2.
The proposed training loss is defined as:
[Equation]
Note that for classification loss, we use the standard cross-entropy loss, but other losses such as focal loss can also be used.
The distance loss is used to increase the distance between distinct class distributions of ID samples so that NAtS samples have larger space to move around without overlapping with ID clusters, especially in NAtS category 1 :
[Equation]
where λ 1 < 0 is a hyperparameter, K is the number of target classes, D is the dimension of feature representation in the latent space (typically the penultimate layer), and µ l is the mean vector of the features of samples with label l .
Since the value of the vector norm increases as the dimension of the feature space increases, we normalize the distance by the square root of the feature dimension.
As will be discussed in Section D of Appendix, we discovered after some initial experiments that the distance loss often made the model use a very limited number of latent dimensions to increase the distance between class mean vectors, which degraded the NAtS detection performance.
In other words, adding only L dist to L CE caused the latent feature space to collapse into a very small number of dimensions ( i.e. rank-deficient), which caused all NAtS samples to be embedded near the
ID samples.
Therefore, we add entropy loss to increase the number of features used to represent samples.
The entropy loss is defined as:
[Equation]
where λ 2 > 0 and λ 3 > 0 are hyperparameters, z ∈ R D is the feature representation in the latent feature space, Var( · ) is variance, and C ij is the correlation coefficient between i th and j th dimension of the feature space.
Specifically, C ij is described as:
[Equation]
where Cov( · ) and σ ( · ) are covariance and the standard deviation, respectively.
Intuitively, the first term in equation 3 encourages each latent dimension to have diverse values, therefore preventing the latent feature space from collapsing into a confined space.
With the first term alone, however, all latent dimensions might learn correlated information, thus making the latent space rank-deficient.
Therefore, we use the second term in equation 3 to minimize the correlation between different latent dimensions.
Note that minimizing the feature correlation was also used in previous works under different contexts such as self-supervised learning (Zbontar et al., 2021).
6. R ESULTS AND D ISCUSSION
To demonstrate the effectiveness of the suggested method, we train all classifiers using the standard cross-entropy loss and our modified loss and compare post-hoc OOD detection methods across three
NAtS datasets.
Specifically, we present the results of the confidence-based ( i.e. , MSP and ODIN) and the distance-based methods ( i.e. , Mahalanobis distance).
We also include a recently proposed OOD detection method (Chandramouli & Sageev, 2020) which computes channel-wise correlations in
CNN with the gram matrix and estimates the deviation of the test samples from the training samples to detect the OOD samples.
1 Although this method uses the distance in the channel correlation space, we expect it to behave more similarly to the Mahalanobis detector than confidence-based methods, namely MSP and ODIN.
We also compare with another recent baseline which exploits the energy score to detect OOD samples (Liu et al., 2020).
As the method leverage the logit layer to calculate the energy score, and the softmax score is based on the logit values, we conjecture that the energy score demonstrates detection ability similar to the confidence-based methods.
[Table]
For a fair comparison, we use only the penultimate layer for evaluating the Mahalanobis detector and Gram matrix since our method is developed based on the analysis of the NAtS samples in the penultimate layer feature space.
We also provide results of the ensemble version that utilize all layers in Section E of the Appendix.
We describe the details of experiment setups and selecting the values for λ 1 , λ 2 and λ 3 in the Appendix Section C, where the three values can be reasonably chosen without any explicit NAtS validation datasets.
We also present an ablation study to investigate the effect of different terms in the proposed loss in Section D of the Appendix.
We also compare with other recent baselines that suggest other training objective for OOD detection in Section E of the
Appendix.
As shown in Table 1, the in-distribution classification accuracy of the model trained with our suggested loss is comparable to that of the model trained with the cross-entropy loss.
Further, we present the NAtS detection performance of the baselines and our method in Table 3.
As we expected above, the gram matrix achieves performance similar to the Mahalanobis detector, demonstrating the low
AUROC in the UTKFace dataset.
Interestingly, even though the energy score is calculated based on the logit values, which are more tempered than the softmax score, it shows NAtS detection performance similar to that of MSP, which is a confidence-based method.
These results show that similar to MSP, ODIN, and Mahalanobis, Gram matrix and Energy-based methods also have performance inconsistency on NAtS datasets.
We report the additional NAtS detection performance on four other metrics, which are often used in the OOD detection community in Section F of Appendix.
Also, it is readily visible that our proposed training objective makes the Mahalanobis detector a robust NAtS detection method for all NAtS categories.
In UTKFace, we can see the dramatic NAtS detection performance increase for Mahalanobis detector.
And for the other two datasets, the proposed loss does not decrease the NAtS detection performance of Mahalanobis detector.
Note that
ODIN is more sensitive to OOD samples than Mahalanobis detector for some brightness levels in the RSNA Bone Age dataset, but it shows inconsistent performance across all three datasets.
R ELATED W ORK
Extensive evaluation on distribution shifts.
Recent works have leveraged the distribution shifts for OOD detection and uncertainty estimation.
Rabanser et al.
(2019) primarily focus on detecting the entire shifted distribution, not on detecting a single shifted sample.
Yaniv et al.
(2019) quantify the predictive uncertainty and investigate the quality of the calibration under the dataset shift.
Engkarat & Takayuki (2019) aims to predict the classification accuracy of a shifted distribution by utilizing the average OOD score of the distribution.
Hsu et al.
(2020) also consider the distribution shift with preserved label space.
However, Hsu et al.
(2020) focus on the shifted distribution from different domain ( e.g. , real images vs. sketches ).
To the best of our knowledge, none of these works conduct an in-depth analysis on the performance of existing OOD detection methods on samples shifted based on natural attributes.
OOD detection methods.
Post-hoc OOD detection methods (Hendrycks & Gimpel, 2017; Liang et al., 2018; Lee et al., 2018; Chandramouli & Sageev, 2020) that utilize the classification models to obtain the OOD scores have achieved remarkable performance.
All of these works aim to detect shifted sample which might affect decision making system in terms of confidence and hidden representation.
Other OOD detection approach is to train the generative model (Ren et al., 2019; Choi et al., 2018; Mahmood et al., 2020) on the training distribution and estimate the density of OOD samples in test time.
While these approaches are viable, it is not directly related with the downstream task but aims to detect features which is different from training distribution.
In this paper, we mainly focus on methods that utilize the classification models to detect OOD samples since we are mainly interested in samples that affect decision-making systems.
Model uncertainty.
A number of previous works measure model uncertainty using various methods such as Bayesian neural networks (Blundell et al., 2015), Monte Carlo dropout (Gal &
Ghahramani, 2016), and deep ensembles (Lakshminarayanan et al., 2017).
Note that technically, model uncertainty can be used to detect NAtS samples, especially for NAtS categories 1 and 3, since sampling model weights from the function space can be seen as redrawing the decision boundary, and NAtS samples in categories 1 and 3 will be affected heavily by this process.
Model uncertainty, however, aims to capture the uncertainty in the model weights rather than detecting OOD samples, making the two rather independent research directions.
C ONCLUSIONS
To enhance the reliability of decision-making systems, we define a new task, Natural Attributebased Shift (NAtS) detection, that aims to detect the samples shifted by a natural attribute.
We introduce NAtS detection benchmark datasets by adjusting the natural attributes present in the existing datasets.
Through extensive evaluation of existing OOD detection methods on NAtS datasets, we observe inconsistent performance depending on the nature of NAtS samples.
Then, we analyze the inconsistency by probing the relationship between the location of NAtS samples and the performance of existing OOD detection methods.
Based on this observation, we suggest a simple remedy to help Mahalanobis OOD detection method to have consistent performance across all NAtS categories.
We hope our dataset and task inspire fellow researchers to investigate practical methods for identifying NAtS, which is crucial for deploying the prediction models in real-world systems.
E THICS S TATEMENT
We believe this is not applicable to us.
R EPRODUCIBILITY S TATEMENT
We describe the implementation and training details along with the details on computation resources in Section C of the Appendix.
We submit our source codes along with the instructions to reproduce our results in the Appendix.
I
A D ETAILS OF OOD DETECTION METHODS
In this section, we describe the three post-hoc and task-agnostic OOD detection methods in detail, focusing mainly on their formulation and how each method assigns an OOD score to an input sample.
A. M AXIMUM OF S OFTMAX P ROBABILITY (MSP)
In this method, the maximum of softmax probability is considered as confidence score (Hendrycks & Gimpel, 2017).
Formally, we calculate the maximum softmax probability as follows:
[Equation]
where C is the number of target classes, c is the index of a class, and z ( j ) denotes j th attribute of the feature in the logit layer.
A. ODIN
ODIN (Liang et al., 2018)
utilized two well established techniques, namely temperature scaling and input preprocessing to increase the difference between softmax scores of in-distribution and OOD samples.
Temperature scaling was originally proposed in Hinton et al.
(2015) to distill the knowledge in neural networks and was later adopted widely in classification tasks to calibrate confidence of prediction(Guo et al., 2017).
In addition to temperature scaling, the input is preprocessed in order to increase the softmax score of given input by adding small perturbations which are obtained by back-propagating the gradient of the loss with respect to the input.
More specifically, ODIN is computed as follows:
[Equation]
where T ∈ R
+ is the temperature scaling parameter, C is the number of target classes, c is the index of class, and z ( j ) denotes j th attribute of the logit layer features of input x .
During training, T is set to 1 .
For OOD detection, the input is first pre-processed as follows:
[Equation]
where ε represents the magnitude of perturbation.
Next, the network calculates the calibrated softmax score of the preprocessed input as follows:
[Equation]
where ˜ z ( j ) denotes j th attribute of the logit layer features of the preprocessed input ˜ x .
Lastly, the modified softmax score is compared to a threshold value δ .
If the score is greater than the threshold, then the input is classified as ID sample and otherwise OOD.
Originally, T , ε , and δ are hyperparameters and are selected such that the false positive rate (FPR) at true positive rate (TPR) 95% is minimized on the validation OOD dataset.
However, the performance saturates when T is greater than 1000 and therefore, in general, a large value of T is preferred.
Following this, in this paper, we fix T = 1000 for our experiments.
A. M AHALANOBIS D ETECTOR
To obtain Mahalanobis distance-based OOD score (Lee et al., 2018) of a sample, we calculate the mahalanobis distance from the clusters of classes to the sample.
Then, the distance from the closest class is chosen as the confidence score.
Specifically, the Mahalanobis score of an input x is defined as
[Equation]
where c and l are the class and layer index, respectively, f lθ is the l th layer’s feature representation of an input x , µ c,l and Σ l are their class mean vector and tied covariance of the training data, correspondingly.
Note that ODIN and Mahalanobis Detector assume the availability of OOD validation dataset.
However, some recent works (Shafaei et al., 2019; Techapanurak et al., 2020) report that this assumption limits the OOD detection generalizability since a model is biased towards an OOD validation set.
In response, this paper validate the performance of OOD methods in the version that does not require to tune with OOD validation dataset.
We perform the performance of ODIN as Hsu et al.
We do not perform Mahalanobis Detector ensembling over on all layers with the optimal linear combination which requires explicit OOD data.
Instead, we perform two version that use only the penultimate layer of hidden representation and sum uniformly over all layers.
Therefore, for all our experiments, we use modified OOD detection methods that do not require the OOD validation dataset.
B A NALYSIS ON THE T EXT D ATASET
In this section, we provide a detailed analysis of the text dataset.
We use the Amazon Review (He & McAuley, 2016; McAuley et al., 2015) dataset.
We consider the product category ”book” and conducted an analysis to see the impact of time on product reviews.
We then performed an analysis to see the impact of time on the length of the reviews.
Figure 6 presents a comparison between density plot of review length for each year from 2006 to 2014 .
We observe that as we move ahead in time, the length of the reviews gradually reduces.
Further, Figure 7 presents the distribution of the average ratio of important words in a sequence by year.
First, we train a model to classify sentiment polarity using Catboost (Prokhorenkova et al., 2017) using document term frequency vector.
We then extract top-100 important words using feature importance.
Lastly, we manually select important words over 100 words.
We find that the distribution of ratio of important words in a sequence gradually increase over time.
Based on this analysis, we figure out that the feature related with downtream task is shifted by time.
C I MPLEMENTATION D ETAILS
In this section, we describe the training details.
Followed by it, we describe the algorithm we used to select the hyperparameters in our suggested modification to the loss function.
Then, we provide the links to download the datasets.
C. T RAINING D ETAILS AND C OMPUTING I NFRASTRUCTURE
Image.
We used ResNet18 (He et al., 2016) that is pretrained with ImageNet (Deng et al., 2009) to train a gender classifier using a UTKFace NAtS dataset.
After the average pooling layer, we trained a fully-connected network composed of 2 hidden layers which have 128 and 2 units respectively with a relu activation.
The network is trained for 100 epochs with a batch size of 64.
We used stochastic gradient descent with Adam optimizer (Kingma & Ba, 2014) and set learning rate as 3 e − 5 .
For data augmentation, the technique of SimCLR (Chen et al., 2020) is used for all the experiments.
Text.
We perform experiment with Transformer network with 4 layers.
The network is trained for 10 epochs with batch size of 128 .
We used Adam as an optimizer and set the learning rate as 3 e − 6 .
Medical.
We use a ResNet18 (He et al., 2016) model, pretrained on ImageNet (Deng et al., 2009), and add two fully-connected layers containing 128 and 2 hidden units with a relu activation.
We train the network to classify gender given the x-ray image.
Each model is trained for 30 epochs using SGD optimizer with a learning rate of 0 .
01 and momentum of 0 .
9 , using a batch size of 64 .
All the experiments are conducted on a single GeForce RTX 3090 GPU with 24 GB memory.
The results are measured by computing mean and standard deviation across 5 trials upon randomly chosen seeds.
C. H YPERPARAMETER T UNING
Our suggested modification to the training loss mainly comprises of distance loss ( L dist ) and a entropy loss ( L entropy ).
More formally, the training loss is given by:
[Equation]
The entropy loss comprises of two terms and is defined as:
[Equation]
where λ 2 > 0 and λ 3 > 0 are hyperparameters, z ∈ R D is the feature representation in the latent feature space, and C ij is the correlation coefficient between i th and j th dimension of the feature space.
For more details, please refer to Section 6.1.
For simplicity, in this section, we will refer the first term of entropy loss as variance loss , and the second term as correlation loss .
To obtain the hyperparameters of different terms in our loss function, we explore the value of λ 2 in [0 . 01 , 0 . 1 , 1 . 0 , 10 . ] and λ 3 in [0 . 0001 , 0 .
The hyperparameter corresponding the distance loss , λ 1 , is set as 0.1, and then the hyperparameters of the variance loss and the correlation loss are chosen by a simple algorithm.
Algorithm: We now describe the algorithm we used to find the hyperparameters of variance and correlation loss .
First, we calculate the harmonic mean of the variance loss and correlation loss using our training dataset.
Next, we select the hyperparameters with the lowest value of harmonic mean.
Then, to ensure that entropy loss prevents the feature space collapsing problem, we apply the singular value decomposition (SVD) in the penultimate features and test if the sum of singular values except the two largest values is improved by the entropy loss or not.
Concretely, to prove the enhancement, we compare the values with those of the model trained with the L CE + λ 1 L dist .
If the selected hyperparameters do not improve the values, we reject them and investigate the hyperparameters that have the next lowest harmonic mean.
The hyperparameters satisfying the above steps are selected as the hyperparameters of our loss function.
Note that we reject the hyperparameters if they significantly degrade the classification accuracy.
In image domain, we applied the algorithm and obtained 0 .
as hyperparameters of λ 1 , λ 2 , and λ 3 , respectively.
In the medical domain, we obtain the hyperparameter
C. L INKS FOR D ATASETS
In our work, we use the openly available datasets, namely, UTKFace dataset 2 , Amazon Review dataset 3 , and RSNA Bone Age dataset 4 .
D A BLATION S TUDIES ON D ISTANCE AND E NTROPY L OSS
In this section, we conduct an ablation study to investigate the effect of each proposed term in the loss function in equation 5.
Table 4 presents an ablation study to find the effect of the entropy loss and distance loss in the proposed modification to training loss.
In UTKFace, when training with solely attaching L dist or L entropy to L CE , the NAtS detection performance using mahalanobis
OOD detection method improves compared to training with only L CE , but the performance does not monotonically increase in both cases along with the variation of Age.
However, when both L dist and
L entropy are used, there is a large improvement in performance with the results being monotonically increasing as we move from ID towards NAtS, which implies that these two losses are mutually beneficial.
In the text and medical domains, although there is a little degradation of AUROC when only L dist is added, the performance is improved by using L entropy in the training objective.
Further, to analyze the impact of each loss term to the feature-level representations, we perform a singular value decomposition (SVD) in the penultimate layer of ResNet18 trained on the UTKFace dataset, similar to Verma et al.
When trained with L CE only, the sum of the two largest singular values was 617 .
47 , while the sum of the remaining singular values was 785 .
When trained with L CE + L dist , the sum of the two largest singular values increased ( 4481 . 88 ), while the sum of the remaining singular values decreased ( 371 . 86 ).
As discussed in Section 6.1 in the main paper, this indicates that the addition of the distance loss is likely to collapse the latent feature space ( i.e. rank-deficient), thus possibly decreasing the model’s sensitivity to NAtS samples.
For example, if a model is trained to classify apples and banaNAtS based only on the color, it will not be able to tell that a firetruck is a NAtS sample.
When we train the classifier with L entropy added, the problem is effectively alleviated.
The sum of the two largest singular value is reduced from 4481 .
88 to 2960 .
63 , and the sum of the remaining singular values increased from 371 .
86 to 513 . 13 .
[Table] +
[Table]
A m az on
R e v i e w
[Table]
R S NA B on e A g e NAtS
[Table]
E C OMPARISON WITH A DDITIONAL B ASELINES
[Table]
[Table]
In this section, we investigate NAtS detection performance using additional baselines, including recent work, and compare them with those of the model trained with our modified loss on three
NAtS datasets.
Firstly, as we mentioned in the main paper, we evaluate the model trained with crossentropy loss with ensembled version of Mahalanobis detector (Lee et al., 2018) and the method that utilizes the gram matrix in OOD detection (Chandramouli & Sageev, 2020).
We also consider the other recent baselines for the comparison.
Recently, Sehwag et al.
(2021) propose a unified framework to leverage self-supervised learning for OOD detection.
We set the method as our baseline and denote this baseline as SSD + .
The experiment on SSD + requires a data augmentation strategy.
In the Image domain, we augment the training dataset as Chen et al.
We construct an augmented training dataset using a back-translation model for text experiments as (Fang et al., 2020; Gunel et al., 2020).
We train SSD + for 100 epochs for UTKFace and RSNA
Bone age dataset and 50 epochs for Amazon Review dataset.
Some recent methods (Hsu et al., 2020; Techapanurak et al., 2020) exploit the cosine similarity to detect OOD samples.
Hsu et al.
(2020) suggest to decompose confidence scores and to modify input preprocessing method in ODIN (Liang et al., 2018).
For OOD detection, they calculate the cosine similarity between the features from the penultimate layer and the class specific weights and use the maximum value as an OOD score.
Techapanurak et al.
(2020) also propose to compute cosine similarity between the class weight vector and the penultimate layer feature of each input to obtain
OOD scores.
Specifically, they train the model with cross entropy loss as standard classification model, but logit value is set by the scaled cosine similarity value.
After training, the cosine similarity values between the penultimate layer feature of the input and the weight vector for each class are calculated.
The maximum cosine similarity value is used as the OOD score for the samples.
We name the methods as GODIN and Scaled Cosine in Table 6, respectively.
NAtS detection performance of our method and other baselines are provided in Table 6.
In UTKFace dataset, GODIN, Scaled Cosine, and ensembled version of Mahalanobis detector have relatively low detection performance than the other baselines.
Although Gram matrix shows reasonable detection performance in UTKFace and RSNA Bone Age NAtS groups, this method is hard to applied to the
Amazon Book Review dataset because the notion of gram matrix is vague in the text domain.
In three NAtS categories, SSD+ has the most robust detection performance among the five baselines, however, our simple remedy in training loss help Mahalanobis detector to demonstrate improved performance particularly under the UTKFace NAtS dataset.
In conclusion, when compared with the five more recent baselines, our method shows robust and high detection performance in all the three NAtS datasets, regardless of the domains.
Scaled Cosine, which compute logit based on cosine similarity, have similar tendency with confidence-based methods.
F Q UANTITATIVE R ESULTS WITH O THER M ETRICS
We report the NAtS detection performance of baselines and our method based on four other metrics: Detection Accuracy, AUPR-In, AUPR-Out, and TNR@95%TPR, which are often used in the OOD detection community.
We present the detection performance on NAtS datasets in three domain measured by Detection Accuracy, AUPR-In, AUPR-Out, and TNR@95%TPR in Tables 7,8,9, and 10 respectively.
The results indicate that our simple modification in the training objective improves the performance of the Mahalanobis detector, thus making it a robust NAtS detection method for the three NAtS categories presented in the paper.
Specifically, in UTKFace, we observe that using our suggested training loss results in a significant increase in NAtS detection performance using the Mahalanobis detector.
At the same time, the suggested method effectively detects the NAtS samples in the other two datasets.
In summary, based on the results obtained using five different metrics (AUROC, Detection Accuracy, AUPR-In, AUPR-Out, and TNR@95%TPR), our suggested modification makes the Mahalanobis detector a general NAtS detection method for all three NAtS categories.
[Table]
R S
NA
B on e
A g e
[Table]
[Table]
[Table]
R S
NA
B on e
A g e
[Table] [Table]
R S
NA
B on e
A g e
[Table]
G A DDITIONAL PCA A NALYSIS
In this section
[Figure]
we provide the additional PCA visualization results.
Figure 8 presents PCA plots applied on the penultimate features obtained from the model trained with our loss.
We observe that the shifted samples in the image domain move in between the two ID classes, and the majority of the NAtS samples do not overlap with the ID clusters.
This helps in effectively detecting the shifted samples as NAtS.
Furthermore
[Figure]
we present the PCA visualization on a model trained with a cross-entropy loss in
Figure 9.
We categorize NAtS samples based on their grouth-truth labels as opposed to Figure 3 in which we do not segregate NAtS samples based on their ground truth labels.
In the main paper, we intentionally do not categorize (and colorize) the NAtS samples because the ground truth class labels are not available at the inference time.
In an ideal situation, the class labels should be predicted with the trained model only if the samples are not classified as NAtS, otherwise, they should be rejected for human assessment.
Note that the PCA visualization cannot be used to infer the real increase in the distance between two classes since PCA compresses the original feature space into 2 dimensions
[Figure]
[Figure]
H
Q UANTITATIVE R ESULTS
[Figure]
I E XPERIMENTS ON A M ULTI - CLASS S YNTHETIC D ATASET
We conduct additional experiments on a multi-class classification dataset to show that the three NAtS categories can exist in different tasks and datasets.
In order to determine the NAtS category to which a shifted sample belongs, we need to train the model for a task on a dataset, evaluate the performance of existing OOD detection methods, and analyze the location of samples in feature space.
However, we would have to conduct a series of experiments on numerous tasks and datasets until we encounter samples from each of the three NAtS categories.
(There is no way to know the NAtS category of a given dataset until we run experiments on it.)
To this end, we synthesized 2D multi-class dataset with 8 classes and conducted experiments on it.
We synthesized the 2-dimensional vectors for training dataset with 8 classes, and those samples are visualized as red points in the 2D plane in Figure 12.
In particular, we create samples in each class by randomly sampling from a gaussian distribution with a random mean and variance.
Deep neural networks are primarily useful in complex datasets with high dimensional inputs.
Therefore, for training, we increase the input dimensions by adding 8dimensional vector which are filled with the Gaussian noise which have mean of zero and variation of 0.1.
The network would focus on the 2 important input features which corresponds x and y coordinates to classify the classes and ignore others.
We train the feed-forward classifier using the standard cross-entropy loss and our modified loss.
The classifier comprises 9 linear layers, and each hidden layer consists of 10 units, and the output layer consists of 8 units (one for each class).
We then evaluate the performance of MSP, ODIN,
Mahalanobis under NAtS for model trained using CE loss in Figure 12 (a), (b), and (c) respectively.
Purple Stars , Gray Diamonds and Orange Triangles indicate samples from NAtS categories 1,and 3 respectively.
We observe that MSP and ODIN detects samples from NAtS category 1 andbut fails to detect from NAtS category 3.
Mahalanobis detects samples from NAtS category 2 ( Gray
Diamonds ) and 3 ( Orange Triangles ), but fails to detect samples from NAtS category 1 ( Purple
Stars ).
We then evaluate the performance of Mahalanobis under NAtS for model trained using our proposed loss in Figure 12 (d).
We observe that Mahalanobis (on model trained with our loss) effectively detects samples from all NAtS categories.
This demonstrates that our proposed training objective makes the Mahalanobis detector a robust NAtS detection method for all NAtS categories.
[Figure]