D ECOMPOSING T EXTURE AND S EMANTICS FOR O UT - OF - DISTRIBUTION D ETECTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Out-of-distribution (OOD) detection has made signiﬁcant progress in recent years because the distribution mismatch between the training and testing can severely deteriorate the reliability of a machine learning system.
Nevertheless, the lack of precise interpretation of the in-distribution limits the application of OOD detection methods to real-world system pipielines.
To tackle this issue, we decompose the deﬁnition of the in-distribution into texture and semantics, motivated by realworld scenarios.
In addition, we design new benchmarks to measure the robustness that OOD detection methods should have.
To achieve a good balance between the OOD detection performance and robustness, our method takes a divide-andconquer approach.
That is, the model ﬁrst tackles each component of the texture and semantics separately, and then combines them later.
Such design philosophy is empirically proven by a series of benchmarks including not only ours but also the conventional counterpart.
[Figure]
I NTRODUCTION
The out-of-distribution (OOD) detection is the task that recognizes whether the given data comes from the distribution of training samples (also known as in-distribution ) or not.
Any machine learning-based system could receive input samples that have a completely disparate distribution from the training environments ( e.g., dataset).
Since the distribution shift can severely degrade the model performance (Amodei et al., 2016), it is a potential threat for a reliable real-world AI system.
However, an ambiguous deﬁnition of the “in-distribution” limits the feasibility of the OOD detection method in real-world applications, considering the various OOD scenarios.
For example, subtle corruption is a clear signal of the OOD in the machine vision ﬁeld while a change in semantic information might not be.
On the other hand, an autonomous driving system may assume the in-distribution from the semantic-oriented perspective; e.g., an unseen trafﬁc sign is the OOD.
Interestingly, the interpretations of the in-distribution described in the above scenarios are not correlated; rather, they are contradicted.
Unfortunately, most of the conventional OOD detection methods (Zhang et al.,
2021; Tack et al., 2020;
Ren et al., 2019) assume the in-distribution as a single-mode; thus are not able to handle other aspects of the OOD (Figure 1a).
To tackle this issue, we revisit the deﬁnition of the in-distribution by decomposing it into two different factors: texture and semantics (Figure 1b).
For the texture OOD case, we deﬁne the OOD as the textural difference between the in- and out-of-distribution datasets.
On the contrary, the semantic
OOD focuses on the class labels that do not exist in the in-distribution environment.
Note that the two aspects have a trade-off relationship, thus detecting both OOD problems with a single model is challenging with the (conventional) entangled OOD point of view.
Similar to ours, Geirhos et al.
(2018) investigated the texture-shape cue conﬂict in the network, and a series of following studies (Hermann et al., 2019; Li et al., 2020; Ahmed & Courville, 2020) explored the way to ﬁnd a balance between these perspectives.
However, aforementioned works utilize texture-shape to analyze the bias inherited in deep networks.
In this work, instead, we focus on analyzing the texture and semantic nature underlying the in-distribution to build a more practically applicable OOD detection method.
However, to the best of our knowledge, none of the studies on the OOD detection benchmark have thoroughly investigated the deﬁnition of the in-distribution.
This can be problematic when the OOD detection method judges the image corrupted by minor distortion as OOD, even when the environment is tolerant to the small changes in texture.
Because of such a complicated scenario, it is important to evaluate the OOD detection method in a comprehensive way that goes beyond the simple benchmarks.
In this work, we propose a new approach to measuring the performance of the method according to the decomposed deﬁnition of the in-distribution.
One notable observation in our benchmark is that most of the previous OOD detection methods are highly biased to the texture information and ignore the semantic clues in many cases.
To mitigate this issue, our method tackles the texture and semantic information separately and aggregates these at the ﬁnal module (Figure 2).
To effectively extract the texture information, we use the 2D Fourier transform motivated by the recent frequency domain-driven deep method (Xu et al., 2020).
For the semantic feature, we design an extraction module upon the Deep-SVDD (Ruff et al., 2018) with our novel angular distance-based initialization strategy.
We then combine two features using the normalizing ﬂow-based method (Dinh et al., 2016), followed by our factor control mechanism.
This control module provides the ﬂexibility to handle various OOD scenarios by choosing which decomposed feature is more important in the given surrounding OOD circumstance.
The main contributions of this work are as follows:
• We decompose the “unclear” deﬁnition of the in-distribution into texture & semantics .
To the best of our knowledge, this is the ﬁrst attempt to clarify the OOD itself in this ﬁeld.
• Motivated by real-world problems, we create new OOD detection benchmark scenarios to evaluate the models based on the decomposed in-distribution factors.
• We propose a novel OOD detection method that is effective on both texture & semantics as well as the conventional benchmark setups.
In addition, our method does not require any auxiliary datasets or class labels unlike the previous models.
R ELATED WORK
In this section, we brieﬂy overview the notable studies on the OOD detection ﬁeld.
We categorize the deep learning-based OOD detection methods into three groups based on the characteristics of the information that they used.
Class labels of the in-distribution.
Early studies on deep OOD methods rely on class supervision.
ODIN, Generailized ODIN (Liang et al., 2017; Hsu et al., 2020) use the uncertainty measure derived by the Softmax output.
It determines the given sample as the OOD when the output probability of all classes is less than the predeﬁned threshold.
Sastry & Oore (2020); Lee et al.
(2018) utilize the extracted feature map ( e.g., gram matrix) from the pre-trained networks to calculate the OOD score.
Auxiliary distribution.
Outlier exposure (OE) (Hendrycks et al., 2018) exploits additional datasets that are disjointed from the test dataset to guide the network to better representations for OOD
PCA
PCA x
E S
E T
ResNet
Multi-SVDD 2D DFT
S ( x )
T ( x ) λ l og p θ ( x | T ( x ))
l
o g p ϕ
( x |
S ( x ))
RealNVP
RealNVP detection.
Papadopoulos et al.
(2021) further improves the performance of OE by regularizing the network with the total variation distance of the Softmax output.
Data augmentation.
Recently, contrastive learning-based methods have shown remarkable success on the tasks related to visual representation (He et al., 2020; Chen et al., 2020).
Motivated by this, several studies employ data augmentation methods such as image transformation or additional noise on the OOD detection task (Hendrycks et al., 2019; Tack et al., 2020).
Unlike the prior studies that exploit additional information other than the in-distribution, we only utilize the given (in-distribution) training dataset.
In addition, we separate and clarify the assumption of the OOD as texture and semantics to improve the practicability in the real world.
M ETHOD
In this section, we present an overview of our proposed method (Section 3.1), and the feature extraction modules of the model (Section 3.3 and 3.2).
Finally, we introduce the normalizing ﬂow-based conditional probabilistic modeling component (Section 3.4).
Conventional OOD detection has an assumption that in-distribution data are sampled from the distribution of the training dataset, x ∼ p data .
We decompose the image with two factors and calculate the anomaly score based on each factor’s likelihood.
The texture information T(x) extracted rigorous Fourier analysis process from input x.
The semantic information S(x) extracts the content label features such as shape.
Our framework calculates the likelihood of these two factors and then combines these likelihoods.
Since we use the
Normalizing ﬂow model that trains the exact likelihood, the extracted information is adjusted using a lambda that the user can control.
3. M ODEL O VERVIEW
We aim to train our method with the decomposed in-distribution likelihood, p ( T ( x ) | x ) and p ( S ( x ) | x ) (Figure 2).
With the given input image x , we extract the features for each variable with different approaches.
In detail, we distil the information from the texture and semantic information with T ( x ) and S ( x ) , respectively.
The extracted features are combined by the controllable normalizing ﬂow method.
Since our normalizing ﬂow-based model explicitly calculates the negative log-likelihood, we model each extracted information as log p θ ( T ( x ) | x ) and log p φ ( S ( x ) | x ) , where θ and φ are trainable parameters of the networks.
In addition, we introduce the control parameter λ ∈ [0 , 1] to model the ﬁnal probability as λ · log p ( x | T ( x ))
+ (1 − λ ) · log p ( x | S ( x )) .
With this control mechanism, a user can determine the appropriate model “mode” by referring to the prior knowledge.
For example, in the case where the texture information overwhelms the semantic one for detecting OOD, we can overweight λ for better performance.
By default, we use the λ value of 0.5 (no prior knowledge).
[Figure]
3. E XTRACTING THE S EMANTIC I NFORMATION
Multi-SVDD.
Beyond the one-class anomaly detection that considers the normal data as a single class ( e.g., DeepSVDD (Ruff et al., 2018)), recent studies have viewed the normal data as the union of the multiple hidden semantic information (Ghafoori & Leckie, 2020; Park et al., 2021).
Inspired by this idea, we use the multi-SVDD method to extract the semantic information in an unsupervised manner for the OOD detection task.
Multi-SVDD embeds the samples to the multiple center vectors as close as possible.
Suppose the set of center vectors C
= { c 1 , ..., c K } is initialized via K-means and the radius of each center is r =
[ r 1 , ..., r K ] .
In multi-SVDD, the objective function is deﬁned as follows.
[Equation]
Here, φ ( x i ; W ) is the deep network with a set of weight parameters W and c j is assigned to φ ( x i ; W ) .
As the set r is decreased, the samples are condensed into the center vectors.
By using the distance between the center vectors and the samples, we get an anomaly score.
Angular distance initialization.
The SVDD method is originally introduced for the anomaly detection task.
Because of the disparity between the OOD and the anomaly detection scenarios, direct application of the SVDD-based model to OOD detection causes unexpected performance degradation.
In anomaly detection, even though the abnormal samples lie in the in-distribution manifold, it is possible to detect them as abnormal unless they are close to the center vectors c j .
For example, as shown in Figure 3a, the OOD samples (red) that are located inside of the in-distribution manifold (light blue shade) can be detected as abnormal since they are outside of the tight cluster boundary (dark blue shade).
Because of such characteristics, a mixture of Gaussian probability density is a reasonable density space for the anomaly detection model.
Unlike the anomaly detection task, the deﬁnition of the OOD detection task is to ﬁnd the samples that are not the “in-distribution”.
In Figure 3a, all the OOD samples placed in the in-distribution manifold (light dark shade) are recognized as the in-distribution.
To tackle this issue, we propose an angular distance-based center vector initialization strategy:
[Equation]
where h is the dimension of the embedding space and γ is the hyper-parameter for the radius of the sphere.
After φ ( x i ; W ) is trained based on angular initialization, semantic features are extracted through this model; S ( x i ) =
φ ( x i ; W ) .
By setting the γ value as large enough, we ensure that all sample data are within a radius of the sphere as illustrated in Figure 3b.
While Equation 1 drives the training samples to be embedded around the center vectors on the sphere, the OOD samples remain near the origin.
This embedding space may be weak to recognize the semantic label of a given sample, but it is sufﬁcient to identify whether the sample is OOD or not.
3. E XTRACTING THE T EXTURE I NFORMATION
To effectively extract the texture property of the in-distribution, we interpret the image in the frequency space.
With a given input image x ∈ (cid:82) 3 × h × w , we ﬁrst convert it into the frequency domain using Discrete Fourier Transform (DFT) F as shown below.
[Equation]
Here, I ( p, q ) denotes the pixel value of the image at the ( p, q ) -coordinate and F ( f x , f y ) is the output of the DFT at the Cartesian coordinate ( f x , f y ) in the frequency space.
In order to construct a scale and rotation invariant frequency information in 2D image, we modify the coordinate system from
Cartesian ( f x , f y ) to polar ( f r , θ ) , following Dzanic et al.
[Equation]
Since directly computing the polar coordinate is computationally expensive and tricky, we iteratively calculate the rotation invariant frequency feature.
To do that, we only utilize the ﬁrst channel of the image as x ∈ (cid:82) 1 × h × w and assume that the image is square ( i.e., h = w ).
Let T ( x ) ∈ (cid:82) 2 /w be the texture feature vector of the image x .
Then, the i -th element of the feature T i ( x ) is calculated as:
[Equation]
y ) ,
R
[Figure]
=
[Equation] [Equation]
[Figure]
Discussion.
We compare the power spectrum density (PSD) of CIFAR-10 (Krizhevsky et al., 2009), distorted CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2010)
datasets (Figure 4).
The PSD discrepancy between the corrupted CIFAR-10 and the vanilla one indicates that the image feature acquired from the frequency domain is adequate to represent the texture cue.
In contrast, CIFAR-10 and CIFAR100 are not distinguishable in the frequency domain since they have very similar image texture due to the small resolution.
These observations support our assumption that the OOD detection model should decouple the texture and semantic information to properly handle the uncorrelated cues.
3. F EATURE C OMPOSITION VIA N ORMALIZING F LOW
Since we design our framework to directly sample the probability, any ad-hoc scoring functions are not required.
Instead, we use a normalizing ﬂow-based method (Dinh et al., 2014; Rezende &
Mohamed, 2015; Dinh et al., 2016) that uses the probability of given samples as a loss function.
In the following, we will describe how to get the probability of samples from the prior probability (Normal distribution) using normalizing ﬂow.
Given sample x , a normal prior probability distribution p Z on a latent variable z ∈ Z , and a bijection f : X → Z (with g = f − 1 ), the change of variables deﬁnes a model distribution on X by
[Equation]
(6) where ∂f ( x ) ∂x T is the Jacobian of f at x .
The bijection function f can be decomposed as f = f 1 ◦· · ·◦ f k .
We use a ﬂow-based RealNVP with coupling layers (Dinh et al., 2016).
To provide the input to
RealNVP, we apply PCA for the dimension reduction of each extracted features.
Finally, given the
[Figure]
The training dataset is in-distribution (ID), while test datasets are out-of-distribution (OOD).
Note that texture and semantic OODs are entangled in the conventional benchmark.
(c, d) Evaluation on the robustness.
Unlike (a, b), it provides the ID samples at test time.
An oracle in this scenario produces AUC as 50.0 (cannot distinguish as OOD at all) since the given samples are from ID.
in-distribution training dataset D , the objective of the RealNVP model with the trainable parameters θ and φ is to minimize the following equations.
[Equation]
By decomposing to the texture and semantic components based on probabilistic modeling, one nice side-effect is that we can adjust the contribution of these components by referring to the given OOD environment.
Since we designed the feature components to be separate, we combine them with a simple linear interpolation as follows.
[Equation]
where λ ∈ [0 , 1] is the control parameter (default is 0.5).
E XPERIMENTS
In this section, we demonstrate the effectiveness of our proposed method on various benchmark setups.
In Section 4.1, we report the performance on the conventional OOD detection task and discuss the limitation of the previous OOD detection studies.
We use the area under the curve (AUC) for the receiver operating characteristic (ROC) curve to evaluate the OOD detection performance.
Then, we will discuss on how to evaluate the robustness that the OOD detection method should have and show the robustness of the previous and ours models (Section 4.2).
4. C ONVENTIONAL O UT - OF - DISTRIBUTION D ETECTION
Setups.
We evaluate out-of-distribution detection methods on the widely used OOD detection benchmark.
Here, we use SVHN (Netzer et al., 2011), CIFAR-10 (Krizhevsky et al., 2009) and CIFAR100 (Krizhevsky et al., 2010) as the in-distribution dataset.
To simulate the OOD samples, LSUN (Yu et al., 2015) and Tiny-ImageNet (Torralba et al., 2008) datasets are additionally used.
Baselines.
We compare our approach to the methods belonging to three different OOD detection groups.
1) Methods that use class labels of the training samples.
Feature based methods such as
Maha (Lee et al., 2018) and Gram (Sastry & Oore, 2020) fall into this category.
2) Methods that utilize additional distribution (dataset) such as OE (Hendrycks et al., 2018) and OEC (Papadopoulos et al., 2021).
3) Self-supervised based methods.
Rotation-based (Rot) (Hendrycks et al., 2019),
SSL (Mohseni et al., 2020), and CSI (Tack et al., 2020) are in this group.
Results.
As shown in Table 1, our proposed method surpasses the competitors on the conventional
OOD detection task without using any extra information such as class labels, other datasets, or image transformation techniques that the other methods required.
In detail, ours with λ = 1 . 0 (semantic mode) achieves the best performance in all the cases with the exception of two scenarios.
Discussion.
As we have discussed, it is often risky to assume that the input samples of the realworld system have a single and entangled characteristic.
We argue that it is more natural to consider
[Table]
that the data samples can have multiple attributes such as texture and semantic information (Figure 5a and 5b).
For example, when the given environment requires to detect the textural discrepancy as OOD, then the detection method should concentrate on the textural side alone, not the semantic counterpart, and vice versa for the semantics driven OOD case.
Unfortunately, conventional OOD detection benchmarks are not developed to measure the disentangled view of the in-distribution.
Here, we dissect the traditional benchmark to quantify the effect of each property we decompose.
We ﬁrst categorized the datasets into the high- and low-resolution groups by the image resolution: C10, C100, SVHN, and TinyImageNet belong to the low-resolution (LR) group while LSUN is high-resolution (HR).
Table 1 shows that previous studies achieve high performance on the LR → HR scenarios and label-based methods produce outstanding results.
However, in Section 4.2.1, we will show that the superior performance of these methods is because they abuse the texture information ( e.g., detect as OOD by highly referring to the image resolution).
On the other hand, we argue that the semantic property is the key component to identity the OOD of the LR → LR scenarios.
For example, C10 ↔ C100 solely requires semantic information to detect
OOD since these datasets use very similar images (in terms of the texture and image resolution).
This is the reason why ours with λ = 0 .
(texture mode) is not able to detect OOD at all (55.7 and 49.4 AUC).
Note that other competitors also show the inferior performance, especially C10 ↔ C case, which demonstrate that these methods have weakness in handling semantic information.
4. E VALUATING THE R OBUSTNESS OF THE O UT - OF - DISTRIBUTION D ETECTION M ETHOD
In this section, we evaluate the robustness of the OOD detection method.
The following experiments are motivated by the real-world demands that the OOD detection framework focused on the semantic discrepancy should be tolerant to the minor changes in the image space (such as the texture shown in Figure 5c), and vice versa (Figure 5d).
However, how can we evaluate the robustness of the OOD detection method in texture and semantic perspectives?
To quantify the robustness of the OOD method, we introduce the AUC score-based measurement.
The evaluation protocol is as follow: 1) We ﬁrst train the OOD detection method with the indistribution training dataset.
We provide the in-distribution samples as test dataset.
Here, contrary to the conventional benchmark, AUC = 50% is the best performance.
This is because 50%
AUC indicates that the detection method cannot distinguish the test samples as OOD (determines as in-distribution).
In Section 4.2.1, we evaluate the robustness on the texture discrepancy by providing the textually different (but marginally) in-distribution dataset at test time (Figure 5c).
Then, in
Section 4.2.2, we benchmark the robustness on the semantic discrepancy (Figure 5d).
[Figure]
Level
ODIN 67.
[Table] [Table]
Maha
[Table] [Table]
[Table] [Table]
[Table]
Gram
[Table]
CSI 65.
[Table] [Table]
[Table] [Table]
λ = 0 .
[Table]
[Table] [Table]
Level
[Table]
ODIN
[Table] [Table]
Maha
[Table]
Gram
[Table]
CSI 54.
[Table] [Table]
[Table] [Table]
[Table] [Table]
λ
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
) Haze
Level
ODIN 63.
[Table]
4.2. R OBUSTNESS ON T EXTURE D ISCREPANCY OF THE I N - DISTRIBUTION
Setups.
Here, we experiment on two benchmark scenarios: 1) distortion and 2) image resolution.
Both cases assume that the deﬁnition of the OOD is in the disparity of semantic information.
That is, a mild change presented in the image space should not be considered as a signal of the OOD.
For the distortion scenario, we corrupt the CIFAR-10 dataset with frost, shot noise, haze, and motion blur distortions with diverse (but mild) corruption levels (Figure 6a).
To make the resolution change case, we ﬁrst perform center-crop and resize it back to the original resolution (32 × 32).
We carefully adjust the cropping operation not to harm the semantic information.
Result.
Table 2 shows the AUC score of the OOD detection.
Since this experiment focuses on detecting the test samples as the in-distribution, 50.0% AUC is the best score.
We observe that the deep feature-based methods (Maha and Gram) do not have robustness against the texture discrepancy at all.
They are not appropriate for real-world applications that do not treat a mild corruption as OOD.
On the contrary, our method with λ = 1 .
(semantics mode) achieves the best performance in all the scenarios because this concentrates on the semantic information alone.
[Table]
We use 32 × 32 CIFAR-10 as the ID and vary the resolution to make test datasets.
Note that 50.0 is the best score since the test datasets are from the in-distribution but with different image resolutions.
[Table]
In the resolution change scenario (Table 3), our method with λ = 1 .
(semantics mode) outperforms the others in all the settings.
All the methods except ours with semantics mode are extremely sensitive to the image resolution change, although no other information is modiﬁed.
4.2. R OBUSTNESS ON S EMANTIC D ISCREPANCY OF THE I N - DISTRIBUTION Setups
To evaluate the robustness that may arise from the semantic discrepancy, we use MNIST,
K-MNIST, and F-MNIST datasets.
K-MNIST comprises of Japanese characters while F-MNIST is a collection of fashion objects.
In this experiment, we set both MNIST and K-MNIST as the indistribution datasets since they are similar in terms of the texture aspect (only the character labels are different, as shown in Figure 6b).
On the other hand, MNIST ↔ F-MNIST is the conventional
OOD scenario since they have both disparate semantics and textures.
Result.
In Table 4, we compare our method with the OOD methods that use class labels or with a self-supervision based approach.
For MNIST → K-MNIST scenario, our method with λ = 0 . 0 successfully determines that K-MNIST is the in-distribution ( e.g., AUC is 50.6).
In contrast, our model with other λ values (0.5 and 1.0) cannot, since these mostly depend on the semantic information to detect the OOD.
This behavior also appears in the other OOD detection methods because they use entangled information.
Not surprisingly, all the methods identify that given test samples are OOD in the MNIST → F-MNIST case, where the texture information is the crucial component.
A series of experiments prove that the texture and semantic cues should be viewed as disentangled factors.
C ONCLUSION
In this work, we introduce a novel viewpoint of the in-distribution for the practically applicable
OOD detection in the real world.
To do that, we separate the deﬁnition of the “single-mode” indistribution to the “texture” or “semantics” factors by following the requirements of the real-world applications.
To effectively handle both aspects, we take a divide-and-conquer strategy that extracts the features using the appropriate method in each factor, then combines these with the normalizing ﬂow-based model.
By doing so, our method outperforms previous models on not only our newly proposed benchmark scenarios but also the conventional OOD detection cases.
We hope that our work provides useful guidance for future OOD detection work.
In the future, we aim to investigate more diverse factors beyond the textures and the semantics that complicated datasets can have, such as multi-object scenes.