SIREN: Shaping Representations for Detecting Out-of-Distribution Objects
Abstract
Detecting out-of-distribution (OOD) objects is indispensable for safely deploying object detectors in the wild.
Although distance-based OOD detection methods have demonstrated promise in image classification, they remain largely unexplored in object-level OOD detection.
This paper bridges the gap by proposing a distancebased framework for detecting OOD objects, which relies on the model-agnostic representation space and provides strong generality across different neural architectures.
Our proposed framework SIREN contributes two novel components: (1) a representation learning component that uses a trainable loss function to shape the representations into a mixture of von Mises-Fisher (vMF) distributions on the unit hypersphere, and (2) a test-time OOD detection score leveraging the learned vMF distributions in a parametric or non-parametric way.
SIREN achieves competitive performance on both the recent detection transformers and CNN-based models, improving the AUROC by a large margin compared to the previous best method.
Code is publicly available at https://github.com/deeplearning-wisc/siren .
Introduction
Teaching object detectors to be aware of out-of-distribution (OOD) data is indispensable for building reliable AI systems.
Today, the mainstream object detection models have been operating in the closed-world setting.
That is, a model will match an object to one of the given class labels, even if it is irrelevant.
Instead, the open-world setting emphasizes that objects from the unknown classes can naturally emerge, which should not be blindly predicted into a known class.
In safety-critical applications, such as autonomous driving, failing to detect OOD objects on the road can directly lead to disastrous accidents [50].
The situation can be better avoided if the object detector recognizes the object as unfamiliar and appropriately cautions the human driver to take over.
In this paper we pioneer a distance-based framework for detecting OOD objects.
Currently, the distance-based method remains largely unexplored in object-level OOD detection.
In particular, by operating in the representation space, distance-based methods are model-agnostic and provide strong generality across neural architectures.
In contrast, existing approaches derive highly specialized OOD detection scores based on the outputs of the object detectors, which may not be seamlessly applicable across architectures.
For example, the classification output of the Faster R-CNN [52] is optimized by the multi-class softmax loss, whereas the recent transformer-based object detection networks such as DEFORMABLE - DETR [75] uses multi-label focal loss [36].
Thereby, while output-based OOD scoring functions may be limited to specific architectures, distance-based methods are not.
Although distance-based OOD scoring functions have been studied in image classification, they do not trivially transfer to object detection models.
For example, [33] modeled the feature embedding space as a mixture of multivariate Gaussian distributions and used the maximum Mahalanobis distance [41] to all class centroids for OOD detection.
However, we observe that the modern object detection
[Figure]
models such as DEFORMABLE - DETR [75] produce highly irregular embeddings (Figure 1 (a)), which do not fit the Gaussian distributional assumption.
As a result, the OOD detection score relying on such suboptimal embeddings can misbehave.
We propose a novel framework called SIREN , tackling two highly dependent problems—representation learning and OOD detection—in one synergistic framework.
Concretely, SIREN contributes two novel components: (1) We introduce an end-to-end trainable loss that enables S hap
I ng the R epres EN tations into a desired parametric form (Section 3.1).
In particular, we model the representations by the von Mises-Fisher (vMF) distribution, a classic probability distribution in directional statistics for hyperspherical data with the unit norm.
Our loss function encourages the normalized embedding to be aligned with its class prototype and shapes the overall representations into compact clusters for each class.
Compared to the Gaussian distribution, using the vMF distribution avoids estimating large covariance matrices for high-dimensional data that is shown to be costly and unstable [7, 65].
(2) We explore test-time OOD detection by leveraging the optimized embeddings in a parametric or non-parametric way (Section 3.2).
We propose a new test-time OOD score based on the learned classconditional vMF distributions.
The parameterization of the vMF distribution is directly obtainable after training, without requiring separate estimation.
Different from Mahalanobis distance [33], the proposed parametric score in principle suits the learned vMF distributions on the hypersphere.
Additionally, we explore a non-parametric nearest neighbor distance for OOD detection [60], which is agnostic to the type of distribution of the feature space.
Empirically, SIREN establishes superior performance on both transformer-based and CNN-based models.
On PASCAL - VOC , SIREN outperforms the latest baseline OW-DETR [18] by a significant margin ( ↑ 22.53% in AUROC).
Moreover, our framework is model-agnostic and does not incur changes to the existing network architecture.
The proposed loss can be flexibly added as a plug-in module on top of modern architectures, as we show in Section 4.
Our key contributions are summarized as follows:
To the best of our knowledge, SIREN pioneers a distance-based approach for object-level OOD detection.
Different from previous works, SIREN does not rely on specialized outputbased OOD scores, and can generalize across different architectures in a model-agnostic fashion.
SIREN establishes competitive results on a challenging object-level OOD detection task.
Compared to the latest method [18], SIREN improves the OOD detection performance by a considerable margin while preserving the mAP on the ID task.
We show that SIREN is effective for both recent transformer-based and classic CNN-based models.
We shape representations via a novel vMF-based formulation for object-level OOD detection.
We conduct in-depth ablations to understand how different factors impact the performance of SIREN (Section 5).
[Figure]
Preliminaries: Object-level OOD Detection
We start by introducing the OOD detection problem for object detection in the open-world setting, which has received increasing research attention lately [13, 18].
Our goal is to train object detection networks that can simultaneously: (1) localize and classify objects belonging to known categories accurately, and (2) identify unfamiliar objects outside the training categories.
Compared to imagelevel OOD detection, object-level OOD detection is more suitable for real-world machine learning systems, yet also more challenging as it requires reasoning OOD uncertainty at the fine-grained object level.
Since natural images are composed of multiple objects, knowing which regions of an image are anomalous allows for safe handling of unfamiliar objects.
Notations.
We denote the input and label space by X
= R q and Y = { 1 , 2 , ..., C } , respectively.
Let x ∈
X be the input image, b ∈ R 4 be the bounding box coordinates associated with objects in the image, and y ∈ Y be the semantic label of the object.
An object detection model is trained on ID dataset D intr =
{ ( x i , b i , y i ) } Mi =1 drawn from an unknown joint distribution P .
We use neural networks with parameters θ to model the bounding box regression p θ ( b | x ) and the classification p θ ( y | x , b ) .
Object-level OOD detection.
The OOD detection can be formulated as a binary classification problem, distinguishing between the in- vs. out-of-distribution objects.
Let P X denote the marginal probability distribution on X .
Given a test input x ′ ∼ P X , as well as an object b ′ predicted by the object detector, the goal is to predict a binary outcome g ( x ′ , b ′ ) .
We use g = 1 to indicate a detected object being ID, and g = 0 being OOD, with semantics outside the support of Y .
Proposed Method
Overview.
Our framework SIREN is illustrated in Figure 2, which trains an object detector in tandem with a representation-shaping branch.
The object detector backbone f : X (cid:55)→ R m maps an object to its feature embedding h ( x , b ) ∈ R m (often referred to as the penultimate layer).
In addition, we introduce a new MLP projection head ϕ : R m (cid:55)→ R d that maps the h ( x , b ) to a lower-dimensional embedding r ∈ R d ( d < m ) with unit norm ∥ r ∥ 2 = 1 .
The normalized embeddings are also referred to as hyperspherical embeddings , since they are on a unit hypersphere.
In designing SIREN , we address two key challenges: (1) How to shape the hyperspherical representations into desirable probability distributions during training time (Section 3.1)?
(2) How to perform test-time OOD detection by leveraging the learned distributions (Section 3.2)?
Our method does not incur any change to the object detection network backbone.
The proposed regularization can be flexibly used as a plug-in module on top of the modern architectures, as we will show in Section 4.
3.1 SIREN: Shaping Representations
Modeling the latent distributions.
We propose to model the latent representations by the von
Mises-Fisher (vMF) distribution [43], a probability distribution in directional statistics for spherical data with unit norm ∥ r ∥ 2 = 1 .
The probability density function for a unit vector r in R d is given as follows:
[Equation]
where κ ≥ 0 , ∥ µ ∥
= 1 , and the normalization factor Z d ( κ ) is defined as:
[Equation]
where I v is the modified Bessel function of the first kind with order v .
Z d ( κ ) can be calculated in closed form based on κ and the dimensionality d .
Importantly, the vMF distribution is characterized by two parameters: the mean vector µ and concentration parameter κ .
Samples that are more aligned with the center µ have a higher probability density, and vice versa.
Here κ indicates the tightness of the distribution around the mean direction µ .
The larger value of κ , the stronger the distribution is concentrated in the mean direction.
In the extreme case of κ = 0 , the sample points are distributed uniformly on the hypersphere.
When considering multiple classes, we can model the embedding space as a mixture of classconditional vMF distributions, one for each class c ∈ { 1 , 2 , ..., C } :
[Equation]
where κ c and µ c are class-conditional parameters.
Under this probability model, an embedding vector r is assigned to class c with the following normalized probability:
[Equation]
Shaping representations.
Our key idea is to design an end-to-end trainable loss function that enables S hap
I ng the R epres EN tations into a mixture of vMF distributions, which facilitates test-time
OOD detection in the representation space (Section 3.2).
We therefore name our method SIREN .
The learned mapping function projects an input to a point in the embedding space, where higher probability is assigned to the correct class in comparison to incorrect classes.
To achieve this, we can perform maximum likelihood estimation (MLE) on the training data:
[Equation]
where i is the index of the object embedding and M is the size of the training set.
By taking the negative log-likelihood, the objective function is equivalent to minimizing the following loss:
[Equation]
where y i is the ground truth label for the embedding r i .
In effect, L SIREN encourages the object embeddings to be aligned with its class prototype, which shapes the representations such that objects in each class form a compact cluster on the hypersphere; see Figure 1 (b).
Prototype estimation and update.
During training, SIREN estimates the class-conditional object prototypes µ c , c ∈ { 1 , 2 , ..., C } .
The conventional approach for estimating the prototypes is to calculate the mean vector of all training samples (or a subset of them) for each class, and update it periodically during training [74].
Despite its simplicity, this method requires alternating training and prototype estimation, which incurs a heavy computational toll and causes undesirable latency.
Instead, we update the class-conditional prototypes in an exponential-moving-average (EMA) manner [34, 63]:
[Equation]
while train do end where α is the prototype update factor, and r denotes the normalized object embeddings from class c .
The update can be done efficiently with negligible cost, enabling end-to-end training.
Overall training objective.
The overall training objective combines the standard object detection loss, along with our new representation shaping loss L SIREN :
[Equation]
8) where β is the weight of our representation shaping loss.
L cls and L loc are losses for classification and bounding box regression, respectively.
We provide extensive empirical evidence in Sectiondemonstrating the efficacy of our loss function.
To the best of our knowledge, our work makes the first attempt to explore vMF-based learning and inference for object-level OOD detection.
To highlight the novelty of the loss itself: we introduce a novel learnable { κ j } Cj =1 for each ID class in Equation (6), instead of using fixed values.
Our loss allows concentration parameter κ to adaptively and flexibly capture the class-conditional feature statistics during training.
This is desirable when each ID class may have its own concentration in the hypersphere.
We will later show that such learnable { κ j } Cj =1 enables better OOD detection performance (Section 5).
3.2 Test-time OOD Detection
During inference, we explore and contrast two types of uncertainty scores for detecting OOD objects.
Parametric vMF score.
We propose a new test-time OOD score based on the learned classconditional vMF distributions, parameterized by { (cid:98) κ c , µ c } Cc =1 .
Here (cid:98) κ c denotes the learned concentration parameter for class c , which captures the concentration of representations for class c .
For a test-time object ( x ′ , b ′ ) , we use the largest estimated class-conditional likelihood as the OOD score:
[Equation]
where r ′ =
ϕ ( h ( x ′ , b ′ )) is the normalized embedding from the MLP projection head.
Our OOD detection score thus in principle suits our learned embeddings and vMF distributions.
For OOD detection, one can use the level set to distinguish between ID and OOD objects:
[Equation]
The threshold γ can be chosen so that a high fraction of ID data (e.g., 95%) is correctly classified.
For objects classified as ID, one can obtain the bounding box and class prediction using the prediction head as usual.
Non-parametric KNN score.
To relax the distributional assumption on the learned embeddings, we additionally employ a non-parametric KNN distance for OOD detection, which performs well on compact and normalized feature space.
Following Sun et al.
[60], the KNN distance is defined as:
[Equation]
where r ( k ) denotes the normalized embedding of the k -th nearest neighbor (in the training data), for the test embedding r ′ .
Our algorithm is summarized in Algorithm 1.
[Figure]
Remark 1.
Different from Mahalanobis distance [33], our parametric vMF-based
OOD detection score operates under the same distributional model as the training process, and hence enjoys mathematical compatibility.
Computationally, we can directly use the parameters of vMF distributions (such as κ ) learned from training.
In contrast, the Mahalanobis distance requires a separate test-time estimation of feature statistics—which involves an expensive and numerically unstable step of calculating the covariance matrix.
The distinction of the uncertainty surface calculated by both our vMF score and the Mahalanobis distance is qualitatively demonstrated in Figure 3.
The data points are sampled from a mixture of three class-conditional vMF distributions (see details in Appendix F).
Experiments
In this section, we validate the effectiveness of SIREN on object detection models, including the latest transformer-based (Section 4.1) and flagship CNN-based models (Section 4.2).
Datasets.
Following [13], we use PASCAL - VOC 1 [14] and Berkeley DeepDrive ( BDD 100 K ) 2 [73] datasets as the ID training data.
For both tasks, we evaluate on two OOD datasets that contain a subset of images from: MS - COCO [37] and O PEN I MAGES (validation set) [30].
Extensive details on the datasets are in Appendix A.
Metrics.
For evaluating the OOD detection performance, we report: (1) the false positive rate (FPR95) of OOD objects when the true positive rate of ID samples is at 95%; (2) the area under the receiver operating characteristic curve (AUROC).
For evaluating the object detection performance on the ID task, we report the common metric mAP.
4.1 Evaluation on Transformer-based Model
Experimental details.
We adopt a very recent DEFORMABLE - DETR ( DDETR ) architecture [75].
DDETR introduces multi-scale deformable attention modules in the transformer encoder and decoder layers of DETR [4], and provides better convergence and lower complexity.
The multi-scale feature maps in DDETR are extracted from a ResNet-50 [21] pre-trained on ImageNet in a self-supervised fashion, i.e. , DINO [5].
We use the embeddings at the penultimate layer of the decoder in DDETR for projection.
For the projection head, we use a two-layer MLP with a ReLU nonlinearity, with dimensionality 256 → d → d .
The dimension d of the unit hypersphere is 16 for PASCAL - VOC and 64 for BDD 100 K .
The default weight β for the SIREN is 1.5 and the prototype update factor α is 0.95.
We initialize the learnable κ to be 10 for all classes.
The k in the KNN distance is set to 10.
Ablations on the hyperparameters are provided in Section 5 and Appendix C. Other hyperparameters are the same as the default ones in DDETR [75].
SIREN achieves superior performance.
In Table 1, we compare SIREN with competitive OOD detection methods in literature.
For a fair comparison, all the methods only use ID data for training.
SIREN outperforms competitive baselines, including Mahalanobis distance [33], KNN distance [60],
CSI [61], Gram matrices [53] and Dismax [39].
These baselines operate on feature embedding space, allowing a fair comparison.
Note that other common output-based methods (such as MSP [23],
ODIN [35], and energy [38]) are not directly applicable for multi-label classification networks in
DDETR .
For these methods relying on a multi-class classification model, we will later provide comparisons on the Faster R-CNN model in Section 4.2.
Implementation details and the training time for all the baseline methods are reported in Appendix D and E.
We highlight a few observations:
The comparison between SIREN vs
Mahalanobis highlights
precisely
the
benefits
of our embedding shaping
loss L SIREN .
As
shown in Figure 1, the
vanilla DETR
model produces ill-
[Table]
conditioned embeddings that do not conform to multivariate Gaussian distributions, rendering the
Mahalanobis approach ineffective (with AUROC around 50%–which is random guessing).
In contrast,
SIREN -vMF improves the OOD detection performance (AUROC) by 25.82 % on PASCAL - VOC ( MSCOCO as OOD).
Different from the Mahalanobis distance, our parametric vMF scoring function naturally suits the learned hyperspherical embeddings with vMF distributions.
The advantage of the vMF loss can be further verified by observing that SIREN -KNN outperforms directly applying KNN distance on the vanilla DDETR ( 16.08 % AUROC improvement on VOC with COCO as OOD).
2) SIREN outperforms the latest methods VOS [13] and OW-DETR [18], which are designed for object detection models and serve as strong baselines for us.
Compared with OW-DETR, SIREN -KNN substantially improves the AUROC by 22.53 % on PASCAL - VOC ( COCO as OOD).
OW-DETR uses the unmatched object queries with high confidence as the unknowns, and trains a binary classifier to separate ID and unknown objects.
However, the unmatched object queries might be distributionally too close to the ID classes and thus displays limited improvement for OOD detection.
In addition, VOS synthesizes virtual outliers from the class-conditional Gaussian distributions of the penultimate layer but fails to perform well due to the ill-conditioned embedding distribution in DDETR (non-Gaussian).
4.2 Evaluation on CNN-based Model
[Table]
Going beyond detection transformers, we show that SIREN is also suitable and effective on
CNN-based object-level OOD detection models, e.g. , Faster R-CNN [52].
Table 2 showcases the OOD detection performance with
SIREN trained on PASCAL - VOC dataset and evaluated on both MS - COCO and O PEN I MAGES datasets.
In addition to baselines considered in Table 1, we include common outputbased methods relying on multi-class classification, such as MSP, ODIN, and energy score.
In Table 2, we additionally report training time comparison.
SIREN consistently improves
OOD detection performance on both OOD datasets.
Notably, SIREN performs better than
40 Fixed kappa 74.
A U R O C 74.
(a) Different fixed kappa values 0.1 0.5 1.0 1.5 2.0 2.5 Loss weight 75.
(b) Different loss weight for SIREN    80 Hypersphere dimension 76.
(c) Different Hypersphere dimension 5 10 20 50 100 200 k in KNN score 77.
(d) Different k in the KNN score the previous best OOD detection approach VOS on Faster R-CNN while preserving the same training time as vanilla Faster R-CNN
Ablations and Discussions
In this section, we provide ablation results on how different factors impact the performance of SIREN .
For consistency, we present the analyses below based on the DDETR model.
Unless otherwise pointed, we use the KNN distance by default.
Ablations on learnable vs. fixed κ .
In this ablation, we show that our approach using learnable concentration parameters { κ c } Cc =1 is better than using the fixed ones.
Our method with learnable κ is desirable, since each ID class may have its own concentration in the hypersphere.
With fixed κ , the loss function can be simplified as follows:
[Equation]
Empirically, we indeed observe that employing learnable { κ c } Cc =1 achieves better OOD detection performance, with AUROC 78.23% on PASCAL - VOC model ( MS - COCO as OOD).
In Figure 4 (a), we show SIREN ’s performance when trained under different fixed κ values.
The best model under the fixed κ = 10 achieves an AUROC of 77.67%.
Too small of κ value ( e.g. , κ = 1 ) leads to almost uniform distributions and is therefore not desirable.
Ablations on the SIREN loss weight β .
Figure 4 (b) reports the OOD detection results as we vary the weight β for the representation shaping loss L SIREN .
The model is evaluated on the MS - COCO dataset as OOD.
Overall a mild weight works well.
Across all the β values considered, SIREN consistently outperforms the baseline OOD detection methods in Table 1.
One can use our SIREN loss as an easy plug-in module, with minimal hyperparameter tuning.
Ablations on the dimension d of hypersphere.
SIREN projects the object feature embeddings into a lower-dimensional hypersphere in R d , which allows tractable vMF estimation.
Figure 4 (c) shows the effect the embedding dimension d on the OOD detection performance.
We find that a lower dimension between 16 and 64 achieves favorable and stable performance.
In the extreme case with dimension d = 8 , the model suffers from considerable information loss and degraded performance.
On the other hand, too large of d causes training instability, which is not desirable either.
Ablations on the uncertainty score.
We perform ablation on two variants of the vMF-based OOD detection score ( c.f. Equation (9)): using the learned { κ c } C c =1 vs. approximately estimate the κ parameters directly from the converged embeddings.
In literature, there are several established methods for approximating κ [43, 55].
Denote r c the average of object embeddings for class c , the simplest approximate solution is given as follows:
[Equation]
[Figure]
The proof is given in Appendix B. We show in Table 3 that using learned { κ c } Cc =1 avoids the imprecision in approximating (cid:98) κ c directly from embeddings.
This affirms the importance of employing learnable concentration parameters in our SIREN loss.
Moreover, the non-parametric KNN density estimation provides stronger flexibility and generality, and leads to better performance.
[Table] [Table]
We ablate on the nonlinearity in the projection head by comparing with SIREN trained with a linear layer in Table 4.
The result shows using a nonlinear mapping for projection helps obtain a more expressive hypersphere, which improves OOD detection by 4.37% in terms of AUROC ( MS - COCO as OOD, vMF as the OOD score).
[Table]
vMF w
[Table]
o nonlinearity
[Table]
vMF
[Table]
Qualitative analysis
In Figure 5, we visualize the predictions on several OOD images, using object detection models trained without SIREN (top) and with SIREN (bottom), respectively.
The in-distribution data is
BDD 100 K .
SIREN better identifies OOD objects (in green) compared to a vanilla object detector
DDETR , reducing false positives.
Moreover, the confidence score of the false-positive objects of
SIREN is lower than that of the vanilla model (see the train/bicycle in the 3rd/5th column).
Related work
OOD detection for classification can be broadly categorized into post hoc and regularization-based approaches [72].
In [2], the OpenMax score is proposed for OOD detection based on the extreme value theory (EVT).
Subsequent work [23] proposed a simple baseline using maximum softmax confidence
(MSP).
Several improvements have been proposed, such as deep ensemble [31], ODIN [35], distancebased score [33, 51, 53, 60], energy-based score [38, 49, 64], DICE [59] and ReAct score [58].
While post hoc methods often considered OOD scoring function alone, our framework considers both representation learning (at training) and OOD detection (at testing).
Along the line of regularization-based approaches, most prior works focused on regularizing the model’s output [3, 16, 22, 26, 28, 42, 44, 47, 48, 62, 66, 71].
For example, the model is regularized to produce lower confidence [24, 32] or higher energy [13, 38] on the outlier data.
In contrast, SIREN regularizes the model by shaping the representations during training.
[39, 40, 61] employed selfsupervised learning and learnable prototypes to learn desirable representations for novelty detection, respectively, but they used complex output-based score ensembles during inference, which are not generalizable and perform sub-optimally (see Tables 1 and 2).
[57] shaped the latent representations as multivariate Gaussians but they trained a variational autoencoder and utilized the reconstruction error for open-set recognition.
In contrast, SIREN directly shapes the embeddings of a discriminative-based model , and does not require training a generative model.
OOD detection for object detection is a rising topic with very few existing works.
For Faster R-CNN,
Du et al.
[13] proposed to synthesize virtual outliers in the feature space for model regularization, which we compare in Section 4.
[12] explored unknown-aware object detection by leveraging videos in the wild, whereas we focus on settings with still images only.
For DETR , [18] adopted unmatched object queries that are with high confidence as unknowns, which did not focus on regularizing the model for desirable representations.
Several works [9, 11, 19, 45, 46] used approximate Bayesian methods, such as MC-Dropout [15] for OOD detection.
They require multiple inference passes to generate the uncertainty score, which are computationally expensive on larger datasets and models.
vMF distribution in ML has been adopted for supervised classification [29, 54], face verification [20], generative modeling [8], segmentation [25] and clustering [17], etc.
Some works exploited vMF distribution for anomaly detection, they employed generative models and used the vMF distribution as the prior for zero-shot learning [6] and document analysis [76].
Both the problem settings and approaches are orthogonal to SIREN .
To the best of our knowledge, our work makes the first attempt to employ vMF-based learning and inference with learnable parameters for object-level
OOD detection.
Metric learning has been explored for both classification [10, 67, 70, 74] and object detection [27, 56, 68, 69].
Unlike ours, they did not focus on the OOD detection problem.
Conclusion
In this paper, we propose a novel framework SIREN , which tackles object-level OOD detection with a distance-based approach.
SIREN mitigates the key shortcoming of the previous output-based OOD detection approach, and explores a new vMF loss to shape representations for OOD detection.
To the best of our knowledge, SIREN makes the first attempt to employ vMF-based learning and inference for OOD detection.
SIREN establishes competitive performance on challenging object-level OOD detection tasks, evaluated broadly under both the recent detection transformers and CNN-based models.
Our in-depth ablations provide further insights on the efficacy of SIREN .
We hope our work inspires future research on OOD detection with representation shaping.
Broader Impacts
Our project aims to improve the reliability and safety of modern machine learning models.
Our study can lead to direct benefits and societal impacts, particularly for safety-critical applications such as autonomous driving.
Our study does not involve any human subjects or violation of legal compliance.
We do not anticipate any potentially harmful consequences to our work.
Through our study and releasing our code, we hope to raise stronger research and societal awareness towards the problem of out-of-distribution detection in real-world settings.
Acknowledgement
We thank Yiyou Sun and Ziyang (Jack)
Cai for their valuable suggestions.
The authors would also like to thank NeurIPS anonymous reviewers for their helpful feedback.
The work is supported by the
AFOSR Young Investigator Program Award.
A Experimental Details
Following [13], we summarize the OOD detection evaluation task in Table 5.
The OOD test dataset is selected from MS - COCO and O PEN I MAGES dataset, which contains disjoint labels from the respective
ID dataset.
SIREN is trained for a total of 50 epochs on PASCAL - VOC , and trained for 30 epochs on
BDD 100 K using ADAM optimizer.
The initial learning rate is 2e-4 and decays at epoch 40 and 24 by 0.1 for PASCAL - VOC and BDD 100 K dataset, respectively.
We set the number of the object queries as 300, the batch size as 8, and the weight β in Equation (8) as 1 . 5 .
See detailed ablations on the hyperparameters on a validation OOD dataset in Appendix C .
[Table]
B Estimating (cid:98) κ
We provide the mathematical details for the estimation of κ in Equation (13).
Concretely, we frame the estimation problem as deriving maximum likelihood estimators (MLEs) for the vMF density function, given the training data { r i } Mi =1 .
Specifically, the maximum likelihood learning can be written as the following optimization problem:
[Equation]
[Equation]
where r = 1 M (cid:80) Mi =1 r i and L MLE is the optimization objective of the maximum likelihood estimation for the distributional parameters µ , κ .
In order to optimize this objective, we take the derivatives of the objective w.r.t.
the parameters µ , κ and set them to 0.
Then, the optimal parameters µ ∗ , (cid:98) κ should satisfy the following two conditions:
[Equation]
where Z ′ d ( (cid:98) κ )
=
dZ d ( (cid:98) κ ) d (cid:98) κ .
For the second condition in Equation (15), we let ξ =
(2 π ) d/ 2 and s = d/ 2 − 1 for notation simplicity.
Put them into the Equation (2), we get the following:
[Equation]
Divide Z d ( (cid:98) κ ) by Z ′ d ( (cid:98) κ ) , we get:
[Equation]
Note that the Bessel function holds a recursive property, which is given as follows:
[Equation]
Substitute I ′ s ( (cid:98) κ )
I s ( (cid:98) κ ) in Equation (17) with the formula in Equation (18) and integrate it with the second condition of Equation (15), we get:
[Equation]
Since there is no known closed-form solution to the Bessel ratio inversion problem as shown in the formula above, we adopt the approximation schemes based on the continued fraction form of the
Bessel ratio function [1], namely:
[Equation]
Replace R with ∥ r ∥ , we get (cid:98) κ
≈ d ·∥ r ∥ 1 −∥ r ∥ 2 .
Following [1], we further add a correction term −∥ r ∥ 3 to the numerator and we get the approximated estimation calculated as:
[Equation]
C Hyperparameter Analysis
Below we perform sensitivity analysis for each important hyperparameter.
Our sensitivity analysis uses the speckle-noised PASCAL - VOC validation dataset as OOD data, which is different from the actual OOD test datasets in use.
We use SIREN pre-trained with DINO [5] as the object detection backbone, trained on in-distribution dataset PASCAL - VOC .
We use the KNN score as the OOD score during inference.
Effect of the hypersphere dimension d .
SIREN projects the object feature embeddings into a lower-dimensional hypersphere in R d , which allows tractable vMF estimation.
A reasonable choice of the hyperspherical dimension d is able to preserve sufficient information for OOD detection while avoiding distributional parameter estimation in the high-dimension space.
As shown in Table 6, using a dimension d between 16 and 64 yields a desirable and stable performance on the OOD validation data while properly maintaining the ID performance (mAP).
We set d = 16 for PASCAL - VOC andfor BDD 100 K in Table 1.
[Table]
Effect of the SIREN loss weight β .
In Table 7, we show the sensitivity of the OOD detection performance of our SIREN w.r.t.
the weight β of the representation shaping loss.
Overall, we find that β = 1 . 5 achieves the best OOD detection and ID performance.
Effect of the k in the KNN score .
In Table 8, we show the sensitivity of the OOD detection performance of our SIREN w.r.t.
the k of the KNN distance during inference.
Overall, we find that k
= 10 achieves the best OOD detection performance.
D Baselines
To evaluate the baselines, we follow the original methods in MSP [23], ODIN [35], KNN [60], and CSI [61] and apply them accordingly on the classification branch of the object detectors.
The
[Table]
[Table]
Mahalanobis distance [33] and gram matrices [53] are calculated based on the penultimate-layer features of the decoder in DDETR .
For CSI [61], we use the rotation degree prediction (0 ◦ , 90 ◦ , 180 ◦ , 270 ◦ ) as the self-supervised task.
We set the temperature in the contrastive loss to 0.5.
We use the penultimate-layer features of the decoder (with dimensionality 256) to perform contrastive learning.
The weights of the losses that are used for classifying shifted instances and instance discrimination are both set to 1 following the original paper [61].
For OW-DETR [18], we follow the original paper and utilize the sigmoid probability of the additional unknown class for OOD detection.
For VOS [13], we use the same hyperparameters as those in the original paper and synthesize virtual outliers in the penultimate layer of the object detector.
Then we regard the virtual outliers as the negative samples during the object classification.
For Dismax [39], we add the dismax loss with learnable prototypes in the penultimate layer of DDTER and apply the same inference score as the original paper for OOD detection.
E Comparison of Training Time
We provide the comparison of the training time for various baselines in Table 9 on our reported hardware (Section G in the Appendix).
As the table shows, the training of our method SIREN incurs minimal computational overhead compared to the vanilla DDETR .
In contrast, other baselines such as
OW-DETR can be more than 2 times slower than SIREN .
[Table]
F Details of Visualization
For Figure 3 in the main paper
[Table]
we generate the toy data in the unit hypersphere by sampling from three vMF distributions in the 3D space.
We adopt a concentration parameter κ of 100 for all three classes.
The centroid vectors are set to [0 , 0 , 1] , [ √ 32 , 0 , − 12 ] and [ − √ 32 , 0 , − 12 ] , respectively.
The uncertainty surface is obtained by calculating the uncertainty score of 200 2 points in the surface of the 3D ball.
G Software and Hardware
We run all experiments with Python 3.8.5 and PyTorch 1.7.0, using 8 NVIDIA GeForce RTX 2080Ti
GPUs.