O UT - OF - DISTRIBUTION D ETECTION WITH D IFFUSIONBASED N EIGHBORHOOD
Anonymous authors
Paper under double-blind review
A BSTRACT
Out-of-distribution (OOD) detection is an important task to ensure the reliability and safety of deep learning and the discriminator models outperform others for now.
However, the feature extraction of such models must compress the data and lose certain information, leaving room for bad cases and malicious attacks.
However, despite effectively ﬁtting the data distribution and producing high-quality samples, generative models lack suitable indicator scores to match with discriminator models in the OOD detection tasks.
In this paper, we ﬁnd that these two kinds of models can be combined to solve each other’s problems.
We introduce diffusion models (DMs), a kind of powerful generative model, into OOD detection and ﬁnd that the denoising process of DMs also functions as a novel form of asymmetric interpolation.
This property establishes a diffusion-based neighborhood for each input data.
Then, we perform discriminator-based OOD detection based on the diffusion-based neighborhood instead of isolated data.
In this combination, the discriminator models provide detection metrics for generation models and the diffusion-based neighborhood reduces the information loss of feature extraction.
According to our experiments on CIFAR10 and CIFAR100, our new methods successfully outperform state-of-the-art methods.
Our implementation is put in the supplementary materials.
I NTRODUCTION
Out-of-distribution (OOD) detection is an important task for deep models that helps the models determine their capability boundary and keep them from being fooled by OOD data.
It has a strong connection with many real-world machine-learning applications, such as cybersecurity (Xin et al., 2018), medical diagnosis (Latif et al., 2018; Guo et al., 2020) and autopilot (Geiger et al., 2012).
The existing methods for OOD detection can be generally categorized into discriminator-based and generation-based methods.
The discriminator-based methods (Wang et al., 2022) use the logit or the feature space to do that.
The generation-based methods (An & Cho, 2015; Nalisnick et al., 2019) use the reconstruction difference in data space or density estimation in latent space to do that.
The discriminator-based methods can extract useful features and make the detection faster and better in most cases.
However, such extraction and compression lose some information and leave room for bad cases and malicious attacks (Goodfellow et al., 2014; Amodei et al., 2016).
The generation-based methods can capture the whole data distribution but lack effective indicator scores to compete with the SOTA discriminator-based methods, partly because of the curse of dimensionality.
Previous works mostly concentrate on solving these challenges using only one kind of model.
For discriminator-based methods, Wang et al.
(2022) combine the information from both features and logits.
Sehwag et al.
(2020) use self-supervised learning to improve feature extraction.
For generation-based methods,
Nalisnick et al.
(2019) use the typicality set to design better indicator scores.
Jiang et al.
(2022) use statistical methods in the latent space, such as the Kolmogorov-Smirnov test.
In addition to overcoming the problems of each kind of model by itself, we ﬁnd that generative and discriminative models can be combined and solve each other’s problems.
Our solution is similar to explain our moti- vation the solution of water quality detection. They do not install detectors everywhere, but install a few ﬁxed detectors and then make the water ﬂow by stirring.
Then these detectors can monitor a large area of water.
For OOD detection, the discriminator is just the detector and we want to make it more sensitive to more situations. Therefore, we also want to ”stir” the input and analyze the potential new anomalies under such an operator. We ﬁnd that generation models are a good choice to ﬁnish this job.
The generation models can edit the inputs and interpolate between different inputs, which are ”stirring” operators we need.
Under such operators, the results of InD data should remain normal at all times and that of OOD data should expose anomalies.
[Figure]
To design suitable generation strategies that can enhance discriminator models, we introduce diffusion models (DMs), which play an important role in generation models, into OOD detection.
DMs have created many state-of-the-art generation results, including (Vahdat et al., 2021;
Ho et al., 2022).
We think their powerful data-ﬁtting ability can be beneﬁcial to OOD detection, too.
We dive into the structure of DMs and ﬁnd that the diffusion denoising process (DDP) of DMs can be an ideal choice for the ”stirring” operator we mentioned above.
Because it can adjust any level of feature space and provides tools to keep the adjustment under control by using the denoising and interpolation properties.
Such an operator ”stirs” the input and needs to be resampled several times to make the result convergent, which builds a neighborhood of input data, called the diffusion-based neighborhood (DiffNB).
According to our above analysis, the feature of OOD data explores anomalies, which means that the feature can change sharply.
We can detect such anomalies by simple Euclidean distance between several different features of DiffNB.
Our pipeline is in Figure 1.
We choose ten representative methods to compare with our methods on two representative datasets:
CIFAR10 and CIFAR100.
According to our experiments, our new methods outperform existing models and methods in most cases.
Our work has the following contributions:
• We ﬁnd that the diffusion denoising process of invertible diffusion models is a novel kind of asymmetric interpolation, which can keep the InD data relatively unchanged and provide tools to control the direction of the denoising process.
• We design a general strategy to combine the beneﬁt of discriminator and generation models.
Under the guidance of this strategy, we use a ResNet to extract features and use the diffusion denoising process of a diffusion model to reduce information loss.
• Our method is explainable and has a clear motivation and solid theoretical background, in the meanwhile, it gets competitive OOD detection results with SOTA methods.
B ACKGROUND
In this section, we ﬁrst introduce existing methods for OOD detection.
Then, we show the development of diffusion models related to our paper.
Because of the limited space in the main paper, more related works about diffusion models can be found in Appendix A.1.
2. O UT - OF - DISTRIBUTION D ETECTION
OOD detection is an important task that can help neural networks to determine their capability boundary.
More speciﬁcally, let X = { x 1 , . . .
, x n } ∼ p be a group of images from the in-distribution (InD) p .
We want to build a detector f that f ( x 1 , . . .
, x n ) = 1 , ∀ i, p ( x i ) ≥ σ and f ( x 1 , . . . , x n ) = 0 , ∀ i, p ( x i ) ≤ σ . Here, σ controls the decision boundary. When we get another group of data
Y = { y 1 , . . . , y n } , we decide whether this group is from (InD) p or an unknown distribution q based on the results of f .
If n = 1 , this is pointwise OOD detection, and if n ≥ 2 , this is group OOD detection.
In general, the existing OOD detection methods can be categorized into discriminator-based and generation-based methods.
Discriminator-based methods design indicator scores based on the output of discriminator models.
Some methods can be used without modifying the model.
ODIN (Liang et al., 2018) uses temperature scaling and the softmax results to detect OOD samples.
ViM (Wang et al., 2022) combines the information of features and logits.
KNN (Sun et al., 2022) includes the kth nearest neighbor of the input in feature space into the detection process.
Some methods try to improve the detection ability in the training process.
G-ODIN
(Hsu et al., 2020) designs a new loss function.
ConfGAN (Sricharan &
Srivastava, 2018) generates OOD data using GANs to help the discriminator models to determine the boundary.
PixMix (Hendrycks et al., 2022) uses data augmentation to improve the detection results.
SSD (Sehwag et al., 2020) uses self-supervised learning to improve feature extraction.
Generation-based methods use the reconstruction difference in the input space and the density estimation in the latent space to do OOD detection.
An & Cho (2015) use the reconstruction ability of VAEs.
Some methods assume that the generation models can reconstruct the in-distribution data better.
Some methods use the Distribution transformation capability of generation models and transfer the input distribution into simple Gaussian distribution.
The likelihood of the input becomes a direct choice, but Nalisnick et al.
(2018) ﬁnds that OOD data can also locate in the high-likelihood area.
Nalisnick et al.
ﬁnd the InD data is concentrated in the typical set instead of the high likelihood area and design new methods using the typical set.
Serr`a et al.
ﬁnd that we can use input complexity to correct the bias of likelihood.
In addition to the likelihood, many existing statistical methods can detect whether a distribution obeys standard Gaussian distribution.
Zhang et al.
(2020) uses KL-divergence to detect OOD data.
Jiang et al.
(2022) use a nonparametric statistics method called the Kolmogorov-Smirnov test.
2. D IFFUSION M ODEL
Classical diffusion model DMs build a transformation from Gaussian distribution to image distribution through a multistep denoising process.
Given a data distribution x 0 ∼ q ( x 0 ) , the diffusion process satisﬁes a Markov process as following Ho et al.
[Equation]
Here, T = 1000 , which is the max iteration step.
β t ∈ (0 , 1) , which controls the speed of adding noise.
Additionally, α t = [Equation]
−
[Equation]
[Equation] x t and ¯ [Equation]
The objective function is deﬁned by:
[Equation]
Here, (cid:15) θ is an estimate of the noise (cid:15) .
After we get well-trained (cid:15) θ , according to Song et al.
(2021a),
the denoising process of Denoising Diffusion Probabilistic Models (DDPMs) and Denoising Diffusion
Implicit Models (DDIMs) satisﬁes:
[Equation]
Here, δ is the iteration step size.
If σ t equals one, Equation (3) represents the denoising process of
DDPMs; if σ t equals zero, this equation represents the denoising process of DDIMs.
In Appendix
A.3, we will further describe how to make the iteration of the diffusion model fast and invertible.
Classiﬁer-free guidance Ho & Salimans (2021) show a simple and effective way to generate conditional samples called classiﬁer-free guidance.
It adds a condition embedding c into (cid:15) θ in the training process and changes the ﬁnal estimation of noise as:
[Equation]
Here, ω is the guidance weight, which controls the balance between realness and diversity.
D IFFUSION - BASED N EIGHBORHOOD FOR OOD D ETECTION
In this section, we ﬁrst theoretically deﬁne our problem, which is to design proper additional operators for a ﬁxed detector. We assume that the detector only concentrates on some subarea of the whole input. Then we use a toy example to show how to design proper additional operators and ﬁnd that reconstruction operators are potential candidates for general cases. After that, we use invertible diffusion models and the diffusion denoising process (DDP) to develop a new asymmetry interpolation to satisfy the requirements of the additional operators.
Then we use a new toy example to show what will happen under DDP for more complex situations and use the norm of the dynamic change under
DDP as our detection score.
Both DDP and the new score can be applied to black box detector f
and we introduce classiﬁer-free guidance for the multi-class condition.
In the end, we provide our general methods and visualize their actual effect when the detector is a pre-trained ResNet18.
3. T HEORETICAL S ETTING
We ﬁrst transfer the OOD detection task and the idea behind water quality detection mentioned in Section 1 into a theoretical problem.
Following the deﬁnitions in Section 2.1, the detector is a function f that can be an analytic function or a neural network.
The detection problem is that given a distribution p on a certain space U , let { x ∈ U | p ( x ) > σ } be W
and we want to ﬁnd a detector f that satisﬁes f − 1 (1) = W exactly. The idea behind water quality detection is to use a ﬁxed detector and make water ﬂow by stirring.
The ﬁxed detector means that the detector f may treat bad cases as normal and f − 1 i (1) is strictly bigger than W . The stirring operator means that we can add additional operators to help the detection. Therefore, we can deﬁne a group of operators { g i } i ∈ I g 1 satisfying f ◦
g i ( x ) = f
( x ) = 1 if x ∈ W and ∩ i ∈ I g
( f ◦ g i ) − 1 (1) ⊂ f − 1 (1) .
The ﬁrst condition means that all f ◦ g i can identify the InD data correctly and the second condition means that the ∩ i ∈ I g
( f ◦ g i ) − 1 (1) becomes smaller by adding more g i . Now we introduce the theoretical version of our problem: Given a ﬁxed subarea W ⊂ U , a single value function f satisfying W ⊂ f − 1 (1) , how can we design operators { g i } i ∈
I g to minimize ∩ i ∈
I g ( f ◦ g i ) − 1 (1) ?
For our task, f is a neural network and the difference between f − 1 (1) and W is the overconﬁdence area. Here, we assume all training data belong to one class. To design proper { g i } , we need to locate what causes such an overconﬁdence problem. We ﬁnd that discriminator models are similar to the ﬁxed water detector and only concentrate on some subareas of the whole input space.
Here, we provide the main assumption in our paper:
For a discriminator model, different channels concentrate on different subareas of the input space. Some subareas are more important than others. Here, we assume that even in the class related area, some subareas
are more important than others.
To show why this assumption is reasonable, We dive into the structure of a discriminator model and ﬁnd that the linear operators, including the convolutional layers and the following pooling layers, play an important role here. According to the singular value decomposition, every linear operator L A : R n → R m , m < n , can be transferred to a restriction operator 2 L A ∗ | S under proper change of basis and S is a m -dimensional subspace. The input data is bounded in our task, so the subspace S becomes a bounded area in the input space.
Different linear kernels in a layer concentrate on different
S i and the total sensitive area of this layer is ∪ i ∈ I S S i . Some subareas can be included by more S i , which are more important than the remaining area. 3. S INGLE -L AYER M ODEL
The whole input space may be covered by the support area of all kernels ∪ i ∈ I S S
i and some areas make the assump- tion clear are covered more times than others.
For simpliﬁcation, we can transfer the problem to a restriction operator.
The key idea is to set a threshold N and mask the area that is covered by less than N support areas S i . Therefore, the data pass through a layer of a neural network as if it had ﬁrst passed through a masking operator and some information is then ignored. For the convenience of theoretical analysis, we choose a simple neural network as f , which only contains one convolutional block and a full connection layer. The support area of each convolutional kernel in the input space is S i and the total support area is ∪ i ∈ I S S i .
Here, we simplify it to a simple mask operator, namely, each S i is a square and all S
i are disjoint with each other.
Our input space is the image space and our training set contains only one element zero (the empty image).
Our InD is
[Figure]
just a uniform distribution in a spherical neighborhood of zero and our target is to detect whether a new input is InD or OOD, namely, detect whether it falls in the neighborhood of zero.
We show this toy example on the left side of Figure 2 and the third row shows a bad case under this setting, which looks like zero after the mask operator.
On the right side of Figure 2, we use three kinds of additional operators moving, mixing and reconstruction to solve this problem.
The ﬁrst two can perfectly solve the toy example in Figure 2 and can be extended to any restriction operator support on different S .
We put the proof in Appendix A.5.
The key idea of these solutions is that they move or mix information from one place to another.
Then the restriction operator can get the whole information from a small support area S .
Such ideas can be generalized to more complex cases and the third reconstruction operator in Figure 2 plays a central role here.
We ﬁnd that the reconstruction strategy successfully contains the moving operator (the face of the cat moves to the left and down), the mix operator (the boundary of each small box becomes unclear) and the semantic level moving operator (the color of the cat is lighter).
We can move or mix the information at both the semantic and pixel level instead of only the pixel level now.
Therefore, it provides a candidate { g i } i ∈ I g with potential advantages for real detectors f , such as a ResNet.
Such a reconstruction strategy can be ﬁnished with any kind of generation model.
However, this strategy also has its weakness.
In Figure 2, we can detect the OOD examples more precisely using a reconstruction operator.
However, we also need to satisfy the condition W ⊂ f − 1 (1) , namely, to keep the pure white picture from being dirty.
This challenge tells us that we need to add more control to the reconstruction strategy.
In the following, we introduce the diffusion-denoising process (DDP) to solve this challenge.
We dive into the structure of diffusion models and show that DDP is a kind of interpolation under the invertible condition.
Therefore, DDP combines the beneﬁt of denoising and interpolation and provides many powerful tools to control the reconstruction process.
Due to space limitations, we put the analysis of enhancing invertibility in Appendix A.3 and directly analyze interesting applications of DDP under the invertible condition.
asym asym sym
Interpolation Many previous papers use diffusion models to interpolate two inputs, but our method is different from the existing symmetric one using spherical linear interpolation (Shoemake, 1985).
We combine the invertible diffusion models and the diffusion denoising process to get a new asymmetric interpolation.
Let us assume that x 0 is an image and (cid:15) is Gaussian noise, which is the reverse of an image x 1 , and we use x 0 and (cid:15) to get x t .
The diffusiondenoising process is shown in Algorithm 1 and we deﬁne DDP as Φ( x t , t, 0) .
When let t equals zeros, we do not add any noise to the images, we can get the original image x 0 , and when t equals T , we remove the total image and only leave the noise (cid:15) , we will do a full denoising process to this noise.
Because this denoising process is invertible, we can get the original image x 1 .
Therefore, the outputs of DDP gradually change from x 0 to x 1 .
Figure 3 shows the three different interpolation results.
The ﬁrst row uses the cat as x 0 and the second row uses the car as x 0 .
We can ﬁnd that x 0 can be better preserved with respect to the symmetric interpolation in the third row.
DDP is a kind of reconstruction strategy, in the meanwhile, it is also a denoising operator and an interpolation operator.
That DDP is a denoising operator means that it can keep the InD data relatively unchanged and pull the OOD data to the high-density area of InD, which is a distributionsensitive property.
That DDP is an interpolation operator means that even if we cannot reconstruct the input perfectly, we can at least control the direction of change.
Both these properties give us more possibilities to control DDP and solve the above weakness.
Finally, we get the following property:
The invertible diffusion-denoising process is also a type of distribution-sensitive interpolation.
3. M ULTI - LAYER M ODEL
In the above section, we show how to design { g i } i ∈ I g and our choice is invertible DDP. Here, we make the assump- tion clear show the effect of DDP in a more complex example to design a proper detection score. We consider the last convolutional layer of a multi-layer model.
In this setting, we face a major change. The input of this layer has many channels. To be more theoretical, the input feature is in R m × m × c , the input feature space is R m × m and the input feature is a combination of c elements in R m × m .
Each of them is the output of a previous convolutional kernel. Therefore, a single input becomes c points in the feature space instead of a single point. [Figure] For the convenience of theoretical analysis, we abstract the whole input space into 2-dimensional space and the ideal feature space belonging to the input data is the light blue square.
The point in each image represents c = 6 features of a single input and the corresponding arrow shows the movement of these features under DDP.
We use the dark blue area to show the features used by the convolutional kernels of the last convolutional layer. For better visualization, the output of each convolutional kernel is a single value, the number of points N that fall in its sensitive area. Again, we assume that all the sensitive areas form a mask. This example is just a simpliﬁcation of the process to compute conﬁdence.
For a normal InD input (the ﬁrst row), its features are uniformly distributed in the feature area, which ensures that N maintains a dynamic balance under the perturbation of DDP in the feature space.
For a normal OOD input (the second row), DDP pulls them to the high-density area and N increases at the same time.
For more challenging OOD data (the third row), it is not
InD and its features are relatively sparse in the ideal feature set.
However, all these features happen to fall in the dark blue area at the same time by coincidence or man-made, which causes the over-conﬁdence problem.
Such imbalance breaks the dynamic balance between the dark blue area and the remaining and causes a rapid decline of N under the perturbation of DDP.
Therefore, all two kinds of OOD input can be successfully detected by the change of conﬁdence.
These analyses can be extended to any middle layer by replacing the single value output with a vector output. 3. D IFFUSION - BASED N EIGHBORHOOD The ﬁrst toy example tells us that we can use DDP to enhance the detection process and the second solve the gap toy example tells us what will happen in feature spaces under DDP.
Detecting the dynamic change of conﬁdence still requires the output is single-value.
We can generalize this idea to detect the change in any feature space of the model by using the norm of the change as the score. After we remove the limitations of the form of the output, both DDP and the dynamic change score can be applied to any black box detector f , such as a ResNet.
The ﬁnal step is to take multiple categories into account and avoid category migration for InD input.
Here, we need the help of the interpolation property and there exist two choices.
First, we can search
Input: Images x 0 , generative interval [0 , T ] , generative gap δ 1: for t = T, · · · , δ do 2: x 0 t = 1 √ ¯ α t ( x t − √ 1 − ¯ α t (cid:15) θ ( x t , t )) 3: (cid:15) =
(cid:15) θ ( x t , t ) 4: x t − δ =
√ ¯
α t − δ x 0 t + √ 1 − ¯ α t − δ (cid:15) 5: end for 6: Return x
Input: Images x , timestep t 1: x knn = KNN( x , { training data } ) 2: (cid:15) =
Φ( x knn , 0 , T ) 3: x noise = √ ¯ α t x + √ 1 − ¯ α t (cid:15) 4: x neighbor =
Φ( x noise , t, 0) 5: Return x neighbor
Algorithm
Input: Images x , timestep t 1: y =
FC ( ResNet ( x )) 2: (cid:15) ∼ N (0 , 1) 3: x noise = √ ¯ α t x + √ 1 − ¯ α t (cid:15) 4: x neighbor =
Φ( x noise , y, t, 0) 5: Return x neighbor
Input: Images x , diffusion-based neighborhood x neighbor 1: fea
=
ResNet( x ) 2: fea neighbor =
ResNet ( x neighbor ) 3: score = (cid:80) | fea − fea neighbor | 4: if score > δ then 5:
Return OOD 6: end if
InD OOD (a)    2 1. (b) 0.
[Figure]
the kth nearest neighbor of the input in the input space and generate noises of them.
We interpolate these noises with the original input using DDP.
Another more interesting choice is that we can train a conditional diffusion model, and ﬁx the class condition to the class of input 3 .
Then all noise are corresponding to the images in the same class.
We can interpolate the input with any noise, instead of searching for it ﬁrst.
What’s more, we can choose several noises for each input and all the results of DDP become a neighborhood of the input.
We call this neighborhood the diffusion-based neighborhood.
Then we use a discriminator model to detect the difference in this neighborhood and determine the OOD samples based on that.
We put our algorithm in Algorithm 4.
In Figure 5, we show the diffusion-based neighborhood (DiffNB) of different inputs using DDP and the inﬂuence of DDP on the different feature spaces of a pre-trained ResNet18.
The DiffNB has a more obvious semantic change for OOD input.
For example, a rickshaw becomes a horse in Fig 5a.
On the other hand, the semantic information of DiffNB of the InD data is relatively unchanged.
Correspondingly, We can ﬁnd that the change in low-level features (the ﬁrst feature space, in Fig 5b) is similar but the change in high-level semantic features (the fourth feature space, in Fig 5c) becomes small when the input is InD and relatively big when the input is OOD.
E XPERIMENT
In this section, we ﬁrst show the detailed setting of our experiments.
Then we offer our OOD detection results, including our method and existing representative methods.
After that, we provide ablation study results to show the contributions of each item and hyperparameter in our new scores.
[Table]
4. S ETUP
We evaluate our methods on the most recent OOD detection benchmarks, OpenOOD benchmarks (Yang et al., 2022).
We use images from six different datasets, which are ﬁltered to ensure that the in-distribution and the OOD do not have overlapping data.
We use the Cifar10 (Krizhevsky et al., 2009) and Cifar100 as InD samples.
For the Cifar10 dataset, we use Cifar100, TinyImagenet (Krizhevsky et al., 2017), SVHN (Netzer et al., 2011), Texture and Places365 (Zhou et al., 2017) as
OOD data.
For the Cifar100 dataset, the OOD datasets are the same, except for swapping Cifar100 for
Cifar10 as the OOD dataset.
For a fair comparison, we ﬁrst train discriminator and generation models using the training set.
we evaluate the results by calculating the area under the receiver operating characteristic curve (AUROC) Fawcett (2006) between the test set of the InD dataset and the test set of others, to avoid the inﬂuence of model overﬁtting.
All images from different datasets are resized into 32 × 32 .
The discriminator models are pre-trained ResNet18 from OpenOOD.
The diffusion model used in this paper is just the classical model from DDPMs.
We use pre-trained unconditional models and train the conditional version by ourselves.
4. O UT - OF - DISTRIBUTION D ETECTION
We put the results in Table 1.
We choose ten representative baselines.
The ﬁrst seven methods do not adjust the discriminator model similar to our method.
ODIN (Liang et al., 2018) uses temperature scaling and gradient-based input perturbation.
EBO (Liu et al., 2020) uses an energybased function.
ReAct (Sun et al., 2021) uses rectiﬁed activation.
MLS (Hendrycks et al., 2019) uses maximum logit scores.
VIM (Wang et al., 2022) combines the information of feature space and logic space.
KNN (Sun et al., 2022) uses the nearest neighbor in the feature space.
All these methods are post-hoc methods and we outperform them in all cases of Cifar10 and two cases of Cifar100.
We also compare our methods with four methods with additional training on the discriminator model.
G-ODIN
(Hsu et al., 2020) decomposes the posterior to model the probability of InD.
[Figure]
CSI (Tack et al., 2020) explores the effectiveness of contractive learning objectives.
CutMix (Yun et al., 2019) and PixMix (Hendrycks et al. 2022) are two new kinds of data augmentation to improve the capability of models.
Our method outperforms them in three cases and gets competitive results in the others.
Our method performs worse than the SOTA methods when the test dataset is SVHN.
A performance bottleneck is that, in addition to density estimation, DDP also has a lazy strategy in the denoising process.
It tends to keep the smooth area unchanged.
In Figure 6, we show this phenomenon using a simple case.
We resize an InD image to r × r and then resize it back, which pulls the InD data away.
However, the reconstruction error
[Figure]
decrease instead of increase under this operator.
This phenomenon also occurs when the input is the relatively simple SVHN dataset.
4. A BLATION S TUDY
Here, we show the inﬂuence of each item and hyperparameter on our scores.
And all the setting is the same as the main experiments in Table 1 except for the ablation object.
[Figure]
Detection space
In Figure 7, we compare the results when we use different detection spaces, including the input image space, the different level feature spaces, and the logit space.
We get the best results when we use the high-level feature or the logit.
Here, the logit has 10 dimensions and is much smaller than the high-level feature space (512 dimensions).
This shows that DDP can reduce information loss successfully.
The low-level features and the image space get relatively bad results, the main reason is the information is still redundant at these levels.
Condition In Figure 8a, we compare our the unconditional and conditional methods.
The main problem is that Cifar100 has much more classes, which makes separating the feature space become more difﬁcult and unconditional diffusion models cannot keep the interpolation in a single class.
We also test different class weights ω , we ﬁnd that a higher class weight can get relatively better results.
This shows that realness is more important than diversity in the OOD detection task.
In addition, we also ﬁnd that the conditional version improves the detection results on Cifar100 more obviously, which means the conditional control is important especially when the number of classes is big.
Training In Figure 8b, we ﬁnd that although the training process of diffusion models is relatively computation-cost to achieve the best FID results, the OOD detection does not need the models to be 100 percent well-trained (200k epoch).
After 40k epoch training, we can get relatively good results and the improvement of FID does not help the OOD detection after that.
Timestep In Figure 8c, we determine how to choose the best t in DDP.
We ﬁnd that the best choice is t = 300 and this is consistent with our examples in Figure 4.
When t ≤ 300 , the difference caused by DDP is still not obvious enough.
When t ≥ 300 , the information start to lose because the noise item accounts for a larger and larger proportion, which limits the OOD detection results.
Resampling In Figure 8d, we determine the inﬂuence of the repeated sampling size.
According to our analysis, the consistent detection results are maintained by dynamic balance, therefore, we need to resample several times to remove the random error in DDP.
We ﬁnd that more is better and 4 times resampling is good enough.
D ISCUSSION
In this paper, we start with a toy example to show how to combine discriminator and generation models to solve the OOD detection task.
Under the setting of the ﬁrst toy example, such a strategy can perfectly solve the OOD detection problem.
Although we cannot say the general cases can also be perfectly solved, we show how to use this idea in the abstract feature space and get competitive results on Cifar10 and Cifar100 using the combination of a ResNet and a diffusion model.
Our approach has good interpretability and a solid theoretical background.
We believe that this strategy opens a new door to developing more powerful OOD detection methods and has the potential to be applied to OOD generalization and other related tasks.
R EPRODUCIBILITY S TATEMENT
Our implementation is put in the supplementary materials. We will publicize our method on GitHub once our paper is accepted.
A A PPENDIX A. R ELATED W ORK
Diffusion Model Denoising Diffusion Probabilistic Models (DDPMs)
Ho et al.
(2020) successfully generate high-quality images and make DMs become popular.
For now, DMs can not only generate unconditional high-quality images but also are applied to many different ﬁelds.
For the conditional generation task, DMs can do interpolation, manipulation, image-editing, style transformation and text-conditional generation Ramesh et al.
For different data types, DMs can do text Austin et al.
(2021), audio Kong et al.
; Lam et al.
(2021) and video generation.
The main challenge for DMs is that they require hundreds to thousands of iterations to produce highquality results, which limits the application of DMs.
After DDPMs, many works try to make DMs faster and better.
Some of them focus on the denoising equations of DMs.
Nichol & Dhariwal (2021) design a better time schedule for the denoising process.
Liu et al.
(2022) provide new numerical methods for the denoising process.
Bao et al.
ﬁnd analytic results for the variance of the denoising process.
Some of them try to design new training strategies and new models.
Salimans &
Ho (2022) use distillation to accelerate DMs.
Dhariwal & Nichol (2021) change the model structure from Unet to GAN to make each iteration step more powerful.
What’s more, Song et al.
(2021b)
ﬁrst shows that DMs can be rewritten as two neural differential equations Chen et al.
; Dupont et al.
Therefore, the numerical methods used to accelerate neural differential equations can also be used here.
A. S IMILARITY AND DIFFERENCE
Several baselines are similar to ours in some ways.
The ﬁrst one is KNN (Sun et al., 2022), which does a KNN search in the feature space.
However, KNN ignores the possibility that an OOD input may have a similar feature as an InD data, and all methods that only use the ﬁnal feature have this problem, too.
In addition, using KNN in the input space directly is also invalid, because of data sparsity and irrelevant information interference.
Another similar approach is data generation and augmentation.
These methods retrain the discriminator, but our generation and discriminator models are trained separately.
What’s more, existing methods use generation models to generate OOD data (Marek et al., 2021) to help the discriminator models to know the capability boundary.
Our methods use the generation models to do interpolation between the new input data and the training data, which do not need to carefully classify the training data and design new loss functions.
Some methods use data augmentation to help the training process of discriminator models.
Although classical data augmentation can enhance the richness of data and keep the data realistic, some new methods start to add complex and unreal augmentation (Hendrycks et al., 2022), which increases the burden on the models and lacks clear motivation.
Many methods combine a generation model and a discriminator model to do OOD detection, such more papers using generation and discriminator models as Ge et al.
; Neal et al.
; Lee et al.
; Du et al. (2022).
However, we ﬁnd these papers all try to use generation models to generate OOD data, which needs to design new object functions and combine the training process of the generation and discriminator models.
Our method use diffusion models to do interpolation and detect the dynamic change under DDP, which is simple and effective.
A. I NVERTIBLE D IFFUSION M ODEL
Here, we introduce more details of the diffusion model used in this paper.
In addition, we analyze how to make the iteration of diffusion models more invertible.
Score-based generation model Song et al.
(2021b) show that the diffusion-denoising process can also be treated as two differential equations:
[Equation]
This is called probability ﬂows (PFs).
The noise (cid:15) θ of DMs and the gradient of logic likelihood s θ are equivalent Bao et al.
More speciﬁcally, we have that s θ ( x, t )
= − 1 1 − ¯ α t (cid:15) θ ( x, t ) .
Pseudo numerical method Liu et al.
(2022) provide pseudo numerical methods for diffusion models (PNDMs) to accelerate DDIMs.
PNDMs deﬁne Equation (3) with σ t = 0 as transfer function:
[Equation]
PNDMs combine this transfer function with the noise estimated by classical numerical methods, like the linear multistep method, to get the new denoising equations:
[Equation]
Here, (cid:15) t = (cid:15) θ ( x t , t ) .
Both PFs and PNDMs accelerate the denoising process without loss of quality.
[Figure]
Invertibility We show the test results in Figure 9.
For DDIMs, the error occurs at the beginning, and the error accumulates with the increase of the total generation step.
For PFs, the initial error is not huge, but the cumulative error occurs when the number of the total generation steps is bigger than 500.
DDIMs are ﬁrstorder methods, and other methods are high-order methods.
We can say higher convergent order can increase the invertibility.
PFs use numerical methods of adaptive step size, and PNDMs use methods of ﬁxed step size.
Therefore, we think that ﬁxed step size can also beneﬁt invertibility.
To verify this, we replace the methods of adaptive step size used by PFs with the methods of ﬁxed step size and call it probability ﬂows plus (PFs+).
We ﬁnd that the errors decrease immediately.
The reason is that ﬁxed step size maintains consistency between the sampling locations of the forward and reverse processes, which beneﬁts the invertibility.
Combining the above analysis, we have the following property:
High convergent order and ﬁxed iteration step size can improve the invertibility of DMs under ﬁxed total iteration steps.
A. I NTERPOLATION
In Algorithm 5 and 6, we introduce two types of interpolation using diffusion models.
The positions of x 0 and x 1 are symmetric in the original interpolation and asymmetric for the asymmetric one.
According to our experiment, x 0 is more important in asymmetric interpolation.
The visualization of these interpolations can be found in Fig 3.
Algorithm
Input: Images x 10 , x 20 , generative gap δ , interpolation rate σ 1: x 1 T , x 2 T = Φ( x 10 , 0 , T, δ ) , Φ( x 20 , 0 , T, δ ) 2: x inter T =
Slerp ( x 1 T , x 2 T , σ ) 3: x inter= Φ( x inter T , T, 0 , δ ) 4: return x inter
Input: Images x 10 , x 20 , generative gap δ , interpolation timestep t 1: x 2 T =
Φ( x 20 , 0 , T, δ ) 2: x inter t
[Equation]
A. T HE PROOF OF TOY EXAMPLE
We ﬁnd that the ﬁrst two kinds of solutions can perfectly solve the problem of bad cases.
To make this claim strict, we need to make some deﬁnitions.
Each image can be represented as a function r on [0 , 1] 2 and the value of r ( x, y ) is the RGB value at position ( x, y ) .
And the images are continuous in most positions.
Therefore, we simplify the input space to C ([0 , 1] 2 ) the continuous function on [0 , 1] and then to one-dimensional C ([0 , 1]) for simplicity.
The mask operator is a restriction operator φ S ( r ) = r | S =[0 , 0 . 25] ∪ [0 . 5 , 0 . 75] here.
The InD is just { r ∈ C ([0 , 1]) | | r | ≤ σ } and | r | is the max absolute value of r on [0 , 1] .
The bad cases form a set that satisﬁes A σ ( φ S ) =
{ r | φ S ( r ) =
φ S (0) = 0 , | r | > σ } .
We can prove that A σ ( φ S )
= ∅ when we choose proper { g i } .
A straightforward solution is that for each r ∈ C ( R ) , let { g ( x ) i =
r ( x + i ) | i ∈ { 0 , ± 0 .
25 }} , which represents the moving operator.
The proof is that we can use g 0 | S to get the information about r on
S and use g ± 0 .
25 | S to get the remaining on [0 , 1] /S .
Then r mush satisﬁes | r ( x ) | < σ, ∀ x ∈ [0 , 1] .
There also exist other kinds of solutions.
For example, let { g ( x ) a,b ≡ 1 b − a (cid:82) b a r ( x ) dx | a, b ∈
0 , 1] } , which represents the mixing operator.
In addition, the mask operator is not necessary and we can generalize to any restriction extend it to any restriction operator.
The full proof can be found in Appendix A.5.
Here we prove that { g ( x ) i =
f
( x + i ) | i ∈ { 0 , ± 0 . 25 }} and { g ( x ) a,b ≡ 1 b − a (cid:82) b a f ( x ) dx | a, b ∈ [0 , 1] } can solve the problem:
Given the input space C ([0 , 1]) , the restriction operator φ S and a ﬁxed δ , how can we design additional operator set { g i } to minimize the annihilator set A δ ( { g i | S } ) ?
For { g ( x ) i = f
( x + i ) | i ∈ { 0 , ± 0 . 25 }} , g 0 | S = f | S
and it equals to zero means that f is zero on S.
And then we also know that g 0 .
25 | S = f | S − 0 .
25 and g − 0 .
25 | S = f | S +0 .
25 equals zero, which means that f is zero on S − 0 .
25 ∪ S +0 .
=
[0 , 1] /S .
Then the only choice is f is zero and the annihilator set is empty.
For { g ( x ) a,b ≡ 1 b − a (cid:82) b a f ( x ) dx | a, b ∈ [0 , 1] } .
If ∃ x 0 ∈ [0 , 1] , f ( x 0 ) > δ , then ∃ (cid:15) s.t. ∀ x ∈ [ x 0 − (cid:15), x 0 + (cid:15) ] , f ( x ) > δ because f is continuous.
Then we have that g ( x ) x 0 − (cid:15),x [Table] + (cid:15) ≡
[Table]
b
[Table]
a (cid:82) b a f ( x ) dx > δ and g ( x ) x 0 − (cid:15),x 0 + (cid:15) | S > δ , too.
This does not satisfy our condition, so the annihilator set is empty.