Delving into OOD Detection with Vision-Language Representations
Anonymous Author(s)
AffiliationAddress
Abstract
Recognizing out-of-distribution (OOD) samples is critical for machine learningsystems deployed in the open world.
The vast majority of OOD detection methodsare driven by a single modality ( e.g. , either vision or language), leaving the richinformation in multi-modal representations untapped.
Inspired by the recent successof vision-language pre-training, this paper enriches the landscape of OOD detectionfrom a single-modal to a multi-modal regime.
Particularly, we propose Maximum
Concept Matching (MCM), a simple yet effective zero-shot OOD detection methodbased on aligning visual features with textual concepts.
We contribute in-depthanalysis and theoretical insights to understand the effectiveness of MCM.
Extensiveexperiments demonstrate that our proposed MCM achieves superior performance ona wide variety of real-world tasks.
MCM with vision-language features outperformsa common baseline with pure visual features on a hard OOD task with semanticallysimilar classes by 56.60% (FPR95).
Introduction
Out-of-distribution (OOD) detection is critical for deploying machine learning models in the wild,where samples from novel classes can naturally emerge and should be flagged for caution.
Despiteincreasing attention, the vast majority of OOD detection methods are driven by single-modal learning [22, 80, 59, 78, 20, 83, 76, 27].
For example, labels are typically encoded as one-hot vectorsin image classification, leaving the semantic information encapsulated in texts largely unexploited.
OOD detection relying on pure visual information can inherit the limitations, e g. , when an OODinput may be visually similar to in-distribution (ID) data yet semantically different from any ID class.
In this paper we delve into a new landscape for OOD detection, departing from the classic singlemodal toward a multi-modal regime.
While the motivation is appealing, a core challenge remains: howto effectively utilize joint vision-language features for OOD detection?
In the visual domain, existingmethods typically require good feature representations [61, 57], and a distance metric under which
OOD data points are relatively far away from the in-distribution (ID) data [35].
These approaches,however, do not directly translate into the multi-modal regime.
On the representation learning side,recent vision-language pre-training schemes such as CLIP [51] and ALIGN [26] have emerged aspromising alternatives for visual representation learning.
The main idea is to align an image withits corresponding textual description in the feature space.
While the resulting representations arepowerful, OOD detection based on such aligned multi-modal features is still in its infancy.
We bridge the gap by exploring a distance-based OOD detection approach, leveraging the jointvision-language representations.
Our method capitalizes on the compatibility between visual featuresand textual features.
By defining the textual features as the “concept prototypes” for each ID class,we characterize OOD uncertainty by the distance from the visual feature to the closest ID prototype.
That is, images closer to one of the textual embeddings of ID classes are more likely to be ID and
[Figure]
“This is a photo of a ______
[Figure] [Figure]
[Figure] [Figure]
vice versa.
By a proper scaling of the distance, our proposed Maximum Concept Matching ( MCM )score achieves strong ID-OOD separability (see Figure 1).
MCM stands in contrast with the previousdistance-based approaches, such as Mahalanobis [35], which defines class prototypes based on purevisual embeddings.
Indeed, we show later in Section 5 that MCM (with multi-modal vision-languagefeatures) is far more competitive than Mahalanobis (with single-modal visual features).
Moreover,while prior works of CLIP-based OOD detection [14, 12] rely on a set of candidate OOD labels,
MCM is OOD-agnostic and alleviates the need for any prior information about test inputs.
Our work also advances the field by showcasing the promise of zero-shot OOD detection, whichoffers strong performance and generality without training on the ID samples.
In particular, classic
OOD detection methods often require training from scratch [21, 7] or fine-tuning [25, 14] on a given
ID dataset.
In this setting, a classifier and its companion OOD detector are good at only one task.
Every new task (ID dataset) requires additional training and brings additional computation and storagecosts.
In contrast, we show for the first time that: (1) MCM achieves superior performance across awide variety of real-world tasks—with just one single pre-trained model .
This is encouraging giventhat there is no training or any OOD information involved.
On the challenging ImageNet-1kbenchmark, MCM’s zero-shot OOD detection performance favorably matches and even outperformsstrong task-specific baselines fine-tuned on BiT [25] and ViT models [14].
(3) MCM remains robustagainst hard OOD inputs, including both semantically hard OODs [72] and spurious OODs [42].
We summarize our main contributions as follows:
We propose MCM, a simple yet effective OOD detection method based on aligned vision-
language features.
MCM offers several compelling advantages over other OOD detection
methods: generalizable (one model supports many tasks), OOD-agnostic (no information
required from OOD data), training-free (no downstream fine-tuning required)
and scalable
to large real-world tasks.
We conduct extensive experiments and show that MCM achieves superior performance on a
wide range of real-world tasks.
On ImageNet-1k, MCM achieves an average AUROC of91.78%, outperforming most recent methods that require training.
Moreover, we show that
MCM remains competitive under challenging hard OOD evaluation tasks.
We provide in-depth empirical and theoretical analysis, providing insights to understand the
effectiveness of MCM.
We hope that this work will serve as a springboard for future works
on OOD detection with multi-modal features.
Preliminaries
Contrastive vision-language pre
training.
Compared to visual representation learning modelssuch as ViT [11], vision-language representation learning demonstrates superior performance onimage classification tasks.
For instance, CLIP [51] adopts a self-supervised contrastive objective( i.e ., InfoNCE loss [64]) to align an image with its corresponding textual description in the featurespace.
Specifically, CLIP adopts a simple dual-stream architecture with one text encoder T : t → R d(e.g.
Transformer [66]) and one image encoder I : x → R d ( e.g ., ViT [11]).
After pre-training on adataset of 400 million text-image pairs, the joint vision-language embeddings of CLIP well associateobjects in different modalities.
Despite the promise, existing CLIP-like models perform zero-shotclassification in a closed-world setting.
That is, it will match an input into a fixed set of categories,even if it is irrelevant ( e.g ., a tree being predicted as a bird in Figure 1).
This motivates our work toleverage the multi-modal representation for OOD detection, which is largely unexplored.
Zero-shot OOD detection.
Given a pre-trained model, a classification task of interest is definedby a set of class labels/names Y in , which we refer to as the known (ID) classes.
Here ID classes aredefined w.r.t.
the classification task of interest, instead of the classes used in pre-training.
Accordingly,
OOD is defined w.r.t.
the ID classes, not the data distribution during pre-training.
The goal of OODdetection is to (1) detect samples that do not belong to any of the known classes; (2) otherwise, assigntest samples to one of the known classes.
Therefore, the OOD detector can be viewed as a “safeguard”for the classification model.
Formally, we denote the OOD detector as a binary function mapping:
G ( x ; Y in , T , I ) :
X → { in , out } , where x ∈ X denote a test image.
Our method is based on onlythe names of the given classes in Y in , and a pre-trained model.
Different from standard supervisedlearning, there is no training on the ID samples involved, hence zero-shot .
OOD Detection via Concept Matching
We illustrate our approach in Figure 1, which derives the OOD detector G ( · ) based on conceptmatching .
For a given task with label set Y in = { y 1 , y 2 , ..., y K } , we can construct a collectionof concept vectors T ( t i ) , i ∈ { 1 , 2 , ..., K } , where t i is the text prompt “ this is a photo of a⟨
y i ⟩ ” for a label
y i .
The concept vectors are represented by the embeddings of the text prompts.
For any test input image x ′ , we can calculate the label-wise matching score based on the cosinesimilarity between the image feature I ( x ′ ) and the concept vector T ( t i ) : s i ( x ′ ) =
[Equation]
Formally, we define the maximum concept matching ( MCM ) score as:
[Equation]
where τ is the temperature.
Intuitively, for ID data, it will be matched to one of the concept vectors(textual prototypes) with a high score; and vice versa.
Formally, our OOD detection function can beformulated as:
[Equation]
where by convention 1 represents the positive class (ID) and 0 indicates OOD.
λ is chosen so that ahigh fraction of ID data ( e.g ., 95%) is above the threshold.
For samples that are classified as ID, onecan obtain the class prediction based on the closest concept: ˆ y =
[Equation]
K ] s i .
Remark : (1) Our work differs from (and perhaps is complementary to) CLIP by focusing on OODdetection rather than (closed-world) zero-shot classification.
We show new theoretical insights nextthat softmax scaling plays a unique role in zero-shot OOD detection—improving the separabilitybetween ID and OOD data.
This role has not been studied rigorously for zero-shot OOD detection.
Readers familiar with CLIP may notice that MCM can be used for zero-shot classification in theclosed world.
This also makes MCM practically convenient for dual goals: detect OOD samples andassign ID data to one of the known classes.
(2) Our method in principle is not limited to CLIP; it canbe generally applicable for contrastive vision-language pre-training models that promote multi-modalfeature alignment.
New insights on softmax scaling for zero-shot OOD detection.
We provide theoretical justifications that softmax scaling improves the separability between ID and OOD data for CLIP-based
OOD detection, which is contrary to models trained with cross-entropy (CE) loss.
In particular,
CLIP-like models are trained with a multi-modal contrastive loss, which maximizes the cosinesimilarity between an image and its textual description in the feature space.
The resulting cosinesimilarity scores display strong uniformity 1 across labels, as evidenced in Figure 2 (right).
Comparedto OOD inputs, the gap between the maximum cosine similarity and the average is larger for IDinputs.
However, the gap can be small when the number of ID classes increases where ID samplesoccur with lower highest cosine similarity.
As a result, the highest cosine similarity for ID samplesand OOD samples can be highly close ( c.f. Figure 2 (left)).
[Figure]
Motivated by these observations, MCM employs softmax as a post hoc mechanism tomagnify the difference.
This is fundamentallydifferent from the softmax score derived from amodel trained with cross-entropy loss , whichinherently maximizes the posterior p ( y | x ) forthe ground-truth label, and minimizes theprobability for other labels.
Unlike CLIPlike models, logit scores displaying uniformitywould be heavily penalized by the CE loss.
Asa result, the maximum logit score corresponding to the ground truth label can already besignificantly higher than other labels.
Applying softmax on the logit scores can exacerbateoverconfident predictions, and reduce the separability between ID and OOD data [39].
Interestingly, for CLIP-like models, the trend is the opposite—applying softmax helps sharpen theuniform-like inner product scores, and increases the separability between ID and OOD data.
To helpreaders better understand the insights, we first formalize our observations that OOD inputs triggersimilar cosine similarities across ID concepts (Figure 2, right) as the following assumption:
Assumption 3.1.
Let z := 1 { y ∈ Y in } and Q x denotes the out-of-distribution P x | z
=0 (marginal distribution of x conditioned on z = 0 ).
Assume ∃ δ > 0 such that
[Equation]
where ˆ y :=
argmax i ∈ [ K ] s i ( x ) and ˆ y 2 :=
argmax i ̸
=ˆ y,i ∈
[ K ] s i ( x ) denote the indices of the largest andsecond largest cosine similarities for an OOD input x .
Now we provide formal guarantees that using softmax can provably reduce the false positive rate(FPR) compared to that without softmax.
Theorem 3.1.
Given a task with ID label set Y in = { y 1 , y 2 , ..., y K } and a pre-trained CLIP-like model ( T , I ) .
If Q x satisfies Assumption 3.1, then there exists a constant T = λ ( K − 1) ( λ wo + δ − s ˆ y 2 )
Kλ −such that for any temperature τ > T , we have
[Equation]
where FPR( τ, λ ) is the false positive rate based on softmax scaling with temperature τ and detectionthreshold λ ; FPR wo ( λ wo ) is the false positive rate without softmax scaling based on threshold λ wo .
This suggests that applying softmax scaling with a moderate temperature results in superior OODdetection performance compared to that without softmax scaling.
The proof is in Appendix A .
Laterin Section 5, we empirically verify on a real-world ImageNet dataset that our bound can indeed besatisfied in CLIP where the thresholds are chosen at 95% true positive rate.
What MCM offers: Beyond theoretical insights, we would like to highlight several compellingadvantages of our zero-shot OOD detection approach in practice:
• Generalizable to many tasks : Traditional OOD detection methods are based on a task-specific
model.
As a result, the OOD detector is not suitable for a realistic
online scenario
where the task
changes from one to another.
In contrast,
we will show in Section 4 that MCM can perform a widevariety of OOD detection tasks, with just one single
model.
For a
new
task, only
the
names
of the
task’s
visual
concepts Y in
are
required
• OOD-agnostic : Our method does not rely on any OOD information, and thus suits many real-
world scenarios where one cannot anticipate what the unknowns would be ahead of time.
This also
mitigates the shortcoming of a recent approach [14], which assumes that a set of unseen labels are
given as some weak information about OOD data.
• Training-free : MCM enables OOD detection in a completely zero-shot setting.
This stands in
contrast to the vast majority of OOD detection literature, which often requires training from scratch
or fine-tuning to achieve competitive performance.
• Scalable : The contrastive vision-language pre-training paradigm makes MCM scalable to a largenumber of class labels and realistic high-resolution images.
We now proceed to the experimental results, demonstrating these advantages on real-world tasks.
A Comprehensive Analysis of MCM
4.1 Datasets and Implementation Details
Datasets.
Most previous works on OOD detection only focus on small-scale datasets with blurryimages such as CIFAR [33] and TinyImageNet [34].
With pre-trained models such as CLIP, OODdetection can be extended to more realistic and complex datasets.
In this work, we scale up evaluationsin terms of (1) image resolution, (2) dataset variety, and (3) number of classes.
We consider thefollowing ID datasets: C UB -200 [69],
S TANFORD -C ARS [32], F OOD -101 [5], O XFORD -P ET [49]and variants of I MAGE N ET [9].
For OOD test datasets, we use the same ones in [25], includingsubsets of iNaturalist [65], S UN [73], P LACES [81], and T EXTURE [8].
For each OOD dataset,the categories are not overlapping with the ID dataset.
We also use subsets of ImageNet-1k forfine-grained analysis.
For example, we construct ImageNet-10 that mimics the class distributionof CIFAR-10 but with high-resolution images.
For hard OOD evaluation, we curate ImageNet-20,which consists of 20 classes semantically similar to ImageNet-10 ( e.g ., dog (ID) vs. wolf (OOD)).
Model.
In our experiments, we adopt CLIP [51] as the target pre-trained model, which is one of themost popular and publicly available vision-language pre-trained models.
Note that our method is notlimited to CLIP; it can generally be applicable for contrastive vision-language pre-training modelsthat promote multi-modal feature alignment.
Specifically, we mainly use CLIP-B/16, which consistsof a ViT-B/16 Transformer as the image encoder and a masked self-attention Transformer [66] as thetext encoder.
To indicate input patch size in ViT models, we append “/x” to model names.
We prepend-B, -L to indicate Base and Large versions of the corresponding architecture.
For instance, ViT-B/16implies the Base variant with an input patch resolution of 16 × 16 .
We also use CLIP-L/14 which isbased on ViT-L/14 as a representative of large models.
Unless specified otherwise, the temperature τis 1 for all experiments.
Details of the datasets, the experimental setup and hyper-parameters areprovided in Appendix B .
Metrics.
For evaluation, we use the following metrics: (1) the false positive rate (FPR95) of OODsamples when the true positive rate of in-distribution samples is at 95%, (2) the area under the receiveroperating characteristic curve (AUROC), and (3) ID classification accuracy (ID ACC).
4.2 Main Results
MCM supports a diverse collection of tasks while being zero-shot.
We first show that zero-shot
OOD detection with MCM is effective across a wide variety of tasks—with just one single pre-trainedmodel .
To showcase the versatility of MCM, we consider the seven ID datasets here.
To the bestof our knowledge, this is among the first attempts to showcase the efficacy under an expansive and
[Table] [Table]
diverse collection of ID datasets.
The zero-shot OOD detection performance is summarized in Table 1.
A salient observation is that MCM can achieve superior detection performance on many tasks.
Forexample, using S TANFORD -C ARS as ID, MCM yields an average FPR95 of 0.09 %.
Considering thatthere are no training samples or OOD information involved, these results are very encouraging.
It can be also seen from Table 1 that MCM is promising, especially when the number of samples per
ID class is limited in the training set.
For example, there are only around 40 samples per class for
Stanford-Cars, 100 for Oxford-Pet, and 30 for CUB-200.
The sample scarcity makes OOD detectionmethods that rely on fine-tuning difficult.
For example, after fine-tuning on Food-101, while the IDaccuracy is increased from 86 .
3% to 92 .
5% ↑ , OOD detection based on MSP is on par with MCM( 99 . 5% vs. 99 .
4% in AUROC).
MCM scales effectively to large datasets.
To examine the scalability of MCM, we compare itwith recent competitive OOD detection methods [25, 14] on the ImageNet-1k dataset (ID) in Table 2.
We observe the following trends:
• Larger model leads to superior performance.
Compared with CLIP-B, MCM based on CLIP-Lreduces FPR95 by 6 .
Zero-shot ID classification accuracy is also improved by 7% with thelarger model, reaching 75 .
53% (see Appendix D).
This suggests that larger models are endowedwith a better representation quality which benefits both ID classification and OOD detection with
MCM.
Our finding echos with the recent observations [67] that higher ID classification accuracy iscorrelated with stronger OOD detection performance.
• MOS [25] recently demonstrated competitive performance on ImageNet-1k, which requires modelfine-tuning based on BiT [31].
In contrast, we show that MCM (CLIP-L) outperforms MOS by1 .
67% in AUROC while being zero-shot (training-free).• MCM shares a softmax scaling function with the classic (visual) confidence-based score MSP [19].To implement MSP we adopt the commonly used linear
probe approach by fine-
tuning
a linear
layer on frozen visual features of CLIP.
After fine-tuning, ID accuracy significantly improves,reaching 84 .
12% (CLIP-L).
Interestingly, the OOD detection performance of MSP is worse than MCM by 16 .
52% in FPR95.
We investigate more in Section 5.
• Recently, Fort et al.
[14] explore small-scale OOD detection by fine-tuning the full ViT model.
When extended to large-scale tasks, we find that MCM still yields superior performance under
the same image encoder configuration (ViT-B or ViT-L).
This further highlights the advantage ofutilizing vision-language joint embeddings for large-scale visual OOD detection.
[Table]
MCM benefits hard OOD detection.
Going beyond, we investigate whether MCM is still effectivefor hard OOD inputs.
We consider the following two categories of hard OOD:
• Semantically hard OOD : OOD samples that are semantically similar to ID samples are particularly
challenging for OOD detection algorithms [72].
To evaluate hard OOD detection tasks in realisticsettings, here we consider ImageNet-10 (ID) vs. ImageNet-20 (OOD) and vice versa.
The pair
consists of high-resolution images with semantically similar categories such as
dog
versus wolf.
As shown in Table
3, MCM
outperforms Mahalanobis [35] by 56.50% in FPR95 for ImageNet-
(ID) vs. ImageNet-20 (OOD) and 38.95% vice versa.
• Spurious OOD : Modern neural networks can exploit spurious correlations for predictions [3].
For example, in the Waterbirds dataset [55], there are spurious correlations between the habitat(
e.g
water and bird types
A recent work [42]
proposes
a new type of hard OOD named spurious
OOD and shows that most OOD detection methods perform much worse for spurious OODinputs compared to non-spurious inputs.
The spurious OOD inputs are created to share the samebackground ( i.e ., water) as ID data but have different object labels ( e.g ., a boat rather than a bird).
See Appendix C for illustrations.
The results are shown in Table 3.
It has been shown that CLIPrepresentations are robust to distributional shifts [51].
Therefore, while prior works [42] show thatspurious OOD inputs are challenging for methods based on ResNet [18], MCM and Mahalanobisscores based on pre-trained CLIP perform much better.
On the other hand, fine-tuning exposes themodel to the training set containing spurious correlation.
As a result, MSP performs much worsethan MCM ( 39 .
57% vs. 5 .
85% in FPR95).
[Figure]
MCM outperforms CLIP-based baselines.
Two recent works also use CLIP embeddings for OOD detection [14, 12].
However, fundamental limitations exist forboth works.
Fort et al.
[14] assume that a candidate OODlabel set Y C is known, and used (cid:80) y ∈Y C ˆ p ( y | x ) for OODdetection.
Here the predictive probability ˆ p ( y | x ) is obtained by normalizing the inner products over |Y in | + |Y C |classes.
While applying softmax converts any vector toprobabilities, as we show in Section 3, the converted probabilities do not necessarily correspond to P ( OOD | x ) .
Moreover, obtaining such an OOD label set is typically notfeasible, which fundamentally limits the applicability.
Alatest work [12] realizes this idea by training an extra textdecoder on top of CLIP’s image encoder to generate candidate labels.
However, [12] cannot guarantee the generatedlabels are non-overlapping with ID labels.
We enhance the baseline with a stronger decoder and a filter module (see Appendix E.1).
As shownin Figure 3, MCM outperforms the enhanced baseline on all OOD datasets.
Moreover, MCM is muchsimpler to use—alleviating the need for an OOD label set or training an additional caption generator.
In contrast, the caption generator’s performance largely affects the OOD detection performance.
Poor caption quality degenerates the OOD detection performance of candidate label-based methods.
Moreover, obtaining a reliable caption generator for any input image can significantly increase thecomputational overhead.
[Figure]
Discussion: A Closer Look at MCM
Empirical verification on the role of softmax.
In Section 3, we prove that softmax scaling oncosine similarity scores with a moderate τ improves the ID-OOD separability.
Here we empiricallyverify our theoretical results.
As shown in Figure 4, compared to directly using the maximum cosinesimilarity without softmax (leftmost figure), softmax scaling with a temperature τ = 1 significantlyimproves the performance by 30 .
8% in FPR95, and further increasing τ ( e.g ., τ = 10 ) leads to similarperformance.
The results are based on ImageNet-100 (ID) versus SUN (OOD).
Now, we verify if our theoretical bound ( c.f. Theorem 3.1) is satisfied empirically as well.
From theleftmost of Figure 4, we can estimate λ wo ≈ 0 .
26 , δ ≈ 0 .
03 , and s ˆ y
≈ 0 .
By checking the thirdfigure ( τ = 1 is the temperature value we use for most experiments), we approximate λ ≈ 0 . 011 .
As K = 100 , we plug in the values and obtain the lower bound T =
[Equation]
≈
[Equation]
Since τ = 1 > 0 . 65 , by Theorem 3.1, applying softmax scaling with τ = 1 is provably superior thanwithout softmax scaling for OOD detection.
[Figure]
vision-language features better than visual feature alone?
MCM can be interpreted as a distance-basedapproach—images that are closer to one of the K classprototypes are more likely to be ID and vice versa.
Herethe class prototypes are defined based on a textual encoder.
Alternatively, one could define the class prototype basedon visual features.
For example, Mahalanobis [35] definesthe class prototype as the average of visual embeddings forimages belonging to the same class.
This raises the question of whether MCM (with multi-modal vision-languagefeatures) is better than Mahalanobis (with single-modalvisual feature).
For a fair comparison, we use the same
ViT image encoder from CLIP-B. Both MCM and Mahalanobis extract visual features from the penultimate layer.
On ImageNet-1k, Mahalanobis displays a limited performance, with 73 .
87% AUROC averaged across four OOD test datasets ( 90.62% for MCM), as shownin Figure 5.
From a practical perspective, Mahalanobis requires computing the inverse covariancematrix, which can be both computationally expensive and inaccurate when the number of samples isscarce and the number of ID classes grows.
In contrast, MCM is much easier to use and more robust.
[Table]
Effect of prompt ensembling.
We examine MCM’s per-
[Table]
formance with prompt ensembling.
For example, Radford et al.
[51] create 80 possible prompts according to the
[Table]
image modalities and nuances in ImageNet.
We experimentwith the two prompt sets, one of size 80 as in [51], andour own set of 5 prompts.
Ensembles are obtained by av-
[Table]
eraging the textual features.
As expected, using ensemblesincreases the ID classification accuracy on ImageNet-1k
[Table]
(2% with CLIP-B and 3% with CLIP-L).
For OOD detection, the average FPR95 is reduced from37.19% with the default prompt to 35.17% ↓ with an ensemble of five prompts shown in Table 4.
Inaddition, the detection performance with 5 prompts is slightly better than with 80 prompts.
Note that prompt ensembling does not increase the inference-time cost, as the textual embeddings (across manyprompts) can be pre-calculated and averaged into a single embedding.
Related Works
OOD detection in computer vision.
For open-world multi-class classification, the goal of OODdetection is to derive a binary ID-OOD classifier along with a multi-class classification model forvisual inputs.
A plethora of methods has been proposed for deep neural networks, including generativemodel-based methods [15, 46, 48, 29, 45, 53, 58, 75], and discriminative-model based methods.
Forthe latter category, an OOD score can be derived based on the softmax output or logit layer [4, 10,19, 22, 25, 39, 41, 60], gradient information [24], or the feature embeddings [35, 56, 57, 61, 72] of aclassifier.
Morteza et al.
[44] recently provided theoretical guarantees for existing OOD detectionscores.
Works insofar have mostly focused on OOD detection for a task-specific model using onlythe visual information.
In contrast, we explore a novel paradigm of zero-shot OOD detection thatincorporates rich textual information and can perform a wide variety of tasks.
OOD detection in natural language processing.
Distribution shifts can occur due to the change oftopics and domains, unexpected user utterances, etc .
Challenging benchmarks [30] and characterization of distributional shifts [1] have been proposed in recent years.
Compared to early languagemodels such as ConvNets and LSTMs, pre-trained language models are more robust to distributionshifts and more effective at identifying OOD instances [20, 50, 76].
Various algorithmic solutions areproposed to handle OOD detection, including outlier exposure [23], model ensembling [37], dataaugmentation [6, 80, 78], contrastive learning [83, 27], and an auxiliary module that incorporatesdomain labels [59].
Tan et al.
[62] also explore zero-shot OOD detection for text classification tasks.
However, prior works focus on pure NLP settings while we explore utilizing textual embeddings forzero-shot visual OOD detection.
Vision-language models.
Utilizing large-scale pre-trained vision-language models for multimodaldownstream tasks has become an emerging paradigm with remarkable performance [63, 17].
Ingeneral, two types of architectures exist: single-stream models like VisualBERT [36] and ViLT [28]feed the concatenated text and visual features into a single transformer-based encoder; dual-streammodels such as CLIP [51], ALIGN [26], and FILIP [77] use separate encoders for text and image andoptimize with contrastive objectives to align semantically similar features in different modality.
Inparticular, CLIP enjoys popularity due to its simplicity and strong performance.
CLIP-like modelsinspire numerous follow-up works [79, 38, 82], which aim to improve data efficiency and betteradaptation to downstream tasks.
This paper adopts CLIP as the target pre-trained model, but ourapproach can be generally applicable for contrastive models that promote vision-language alignment.
Multi-modal OOD detection.
Exploring textual information for visual OOD detection is a new areawith limited existing works.
Fort et al.
[14] propose to feed the potential OOD labels to the textualencoder of CLIP [51].
Recently, Esmaeilpour et al.
[12] propose to train a label generator basedon the visual encoder of CLIP and use the generated labels for OOD detection.
While both worksrely on a set of candidate OOD labels, MCM is OOD-agnostic and alleviates the need for any priorinformation about OOD.
Moreover, prior works [51, 12] only focus on small-scale inputs.
We largelyexpand the scope to a wide range of large-scale realistic datasets, and show new theoretical insights.
Conclusion
In this work, we delve into a new landscape for OOD detection, departing from the classic singlemodal toward a multi-modal regime.
By viewing the textual features as the “concept prototypes”, weexplore a new OOD detection approach MCM, based on the joint vision-language representations.
Unlike the majority of OOD detection methods, MCM offers several compelling advantages: trainingfree, generalizable to many tasks, scalable to hundreds of classes, and does not require any priorinformation on OOD inputs.
Moreover, we provide theoretical guarantees on how softmax scalingprovably improves zero-shot OOD detection.
We investigate the effectiveness of MCM on a widerange of large-scale realistic tasks, including several types of hard OOD datasets.
Lastly, wedemonstrate the advantage of vision-language features over pure visual features for OOD detection.
We hope our work will inspire future research toward multi-modal OOD detection.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope
[Yes
(b) Did you describe the limitations of
your work?
[N/A]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3and Section A
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section A3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Section B(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4 and Section B(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
There is no training involved in MCM.
For fair
comparison and reproducibility, we use the publicly available checkpoints of CLIPfrom OpenAI.
The test OODs do not have randomness either.(d)
Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section B4.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes] See Section B(b)
Did you mention the license of the assets?
[N/A](c)
Did you include any new assets either in the supplemental material or as a URL?
[Yes]See Section B(d)
Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Theoretical Justification: Softmax Scaling for Zero-Shot OOD Detection
In this section, we provide the proof for Theorem 3.1 in Section 3, which states the benefits ofapplying softmax scaling to inner products for OOD detection.
We begin with a review of notations.
Notations.
We denote the text encoder of a pre-trained CLIP-like model as T : t → R d and theimage encoder I : x → R d .
For a given task with label set Y in = { y 1 , y 2 , ..., y K } , we construct acollection of concept vectors T ( t i ) .
For a given input x ′ , we denote the cosine similarity w.r.t conceptvectors as s i ( x ′ ) =
I ( x ′ ) ·T ( t i ) ∥I ( x ′ ) ∥·∥T ( t i )
∥ ∀ i ∈
[ K ] , where | s i ( x ′ ) | ≤ B for all x ′ ∈ X . 2 .
We definethe maximum concept matching (MCM) score as: S MCM ( x ′ ;
Y in , T , I ) =
max i ∈
[ K ] e si ( x [Equation]
[Equation]
(cid:80)
Kj =1 e sj ( x ′ )
/τ .
We denote the maximum inner product without applying softmax scaling as S woMCM ( x ′ ; Y in , T , I )
=max i ∈
[ K ] s i ( x ′ ) .
By convention, the OOD detection functions are given by:
[Equation]
Remarks: By convention, 1 represents the positive class (ID) and 0 indicates OOD; λ and λ wo aretypically chosen such that the true positive rate is at 95% .
For convenience, we paste the assumptions and the theorem in Section 3 below,
Assumption A.1.
Let z := 1 { y ∈ Y in } and Q x denotes the out-of-distribution P x | z
=0 (marginal distribution of x conditioned on z = 0 ).
Assume ∃ δ > 0 such that
[Equation]
where ˆ y :=
argmax i ∈ [ K ] s i ( x ) and ˆ y 2 :=
argmax i ̸
=ˆ y,i ∈
[ K ] s i ( x ) denote the indices of the largest andsecond largest cosine similarities for an OOD input x .
Theorem A.1.
Given a pre-trained CLIP-like model ( T , I ) and a task with label set Y in = { y 1 , y 2 , ..., y K } .
If Q x satisfy Assumption A.1, Then there exists a constant T = λ ( K − 1) ( λ wo + δ − s ˆ y 2 )
Kλ −such that for any temperature τ > T , we have:
[Equation]
where FPR( τ, λ ) is the false positive rate based on softmax scaling with temperature τ and thresholdλ ; FPR wo ( λ wo ) is the false positive rate without softmax scaling based on threshold λ wo .
This suggests that applying softmax scaling with temperature results in superior OOD detection performancecompared to without softmax scaling.
Proof.
By definition, we express the false positive rate FPR( τ, λ ) as follows,
[Equation]
By inequality e x ≥ 1 + x , we have,
[Equation]
This indicates
[Equation]
Since
K (cid:88) i =
[Equation] [Equation]
s i
[Equation]
x ′ ))
By Assumption 3.1, we have
[Equation]
Therefore,
Q
[Equation]
where λ ′ = (cid:0) K − 1 λ (cid:1) τK − 1 is a monotonic function of λ ( i.e ., minimizing false positive rate w.r.t.
λis equivalent to minimizing w.r.t λ ′ .)
For τ > 0 , we can rewrite the MCM score as
[Equation]
As ˆ y :
= argmax i ∈
[ K ] s i ( x ) , s j ( x ′ )
− s ˆ y ( x ′ ) ≤ 0 , S MCM ( x ′ ;
Y in , T , I ) is a monotonically decreasing function of τ , we have:
[Equation]
Therefore by the definition of λ , we have λ > 1 K , λ ′
=
(cid:0) K − 1 λ (cid:1) τK − 1 >
For moderately large τ > T where T = λ ( K − 1) ( λ wo + δ − s ˆ y 2 )
Kλ −, we always have s ˆ y 2 ( x ′ )
− δ + λ ′ >
λ wo .
Therefore, we obtain the following inequality,
[Equation]
which means that the FPR without softmax scaling is larger than that with softmax scaling and amoderately large temperature.
We show in Section 5 that the bound is indeed satisfied in practicewith a large-scale ID dataset.
B Experimental DetailsB.1 Software and Hardware
All methods are implemented in Pytorch 1.10.
We run all OOD detection experiments on NVIDIA
GeForce RTX-2080Ti GPU and use NVIDIA A100 GPU for fine-tuning CLIP and ViT.
B.2 Hyperparameters
The only hyperparameter in MCM is the (temperature) scaling factor τ .
We use τ = 1 by defaultunless otherwise specified.
Our experiments suggest that MCM is insensitive to the scaling factor,where τ in a wide range of [0 . 5 , 100] shares similar performance.
B.3 Datasets
ImageNet-
We create ImageNet-10 that mimics the class distribution of CIFAR-10 but with highresolution images.
It contains the following categories (with class ID): warplane (n04552348), sportscar (n04285008), brambling bird, (n01530575), Siamese cat (n02123597), antelope (n02422699),
Swiss mountain dog (n02107574), bull frog (n01641577), garbage truck (n03417042), horse(n02389026), container ship (n03095699).
ImageNet-20 For hard OOD evaluation with realistic datasets, we curate ImageNet-20, whichconsists of 20 classes semantically similar to ImageNet-10 ( e.g ., dog (ID) vs. wolf (OOD)).
Thecategories are selected based on the distance in the WordNet synsets [13].
Specifically, it containsthe following categories: sailboat (n04147183), canoe (n02951358), balloon (n02782093), tank(n04389033), missile (n03773504), bullet train (n02917067), starfish (n02317335), spotted salamander (n01632458), common newt (n01630670), zebra ( n01631663), frilled lizard (n02391049),green lizard (n01693334), African crocodile (n01697457), Arctic fox ( n02120079), timber wolf(n02114367), brown bear (n02132136), moped (n03785016), steam locomotive (n04310018), spaceshuttle (n04266014), snowmobile (n04252077).
ImageNet-100 We randomly sample 100 classes from ImageNet-1k to curate ImageNet-100.
Itcontain the following categories: n01986214, n04200800, n03680355, n03208938, n02963159,n03874293, n02058221, n04612504, n02841315, n02099712, n02093754, n03649909, n02114712,n03733281, n02319095, n01978455, n04127249, n07614500, n03595614, n04542943, n02391049,n04540053, n03483316, n03146219, n02091134, n02870880, n04479046, n03347037, n02090379,n10148035, n07717556, n04487081, n04192698, n02268853, n02883205, n02002556, n04273569,n02443114, n03544143, n03697007, n04557648, n02510455, n03633091, n02174001, n02077923,n03085013, n03888605, n02279972, n04311174, n01748264, n02837789, n07613480, n02113712,n02137549, n02111129, n01689811, n02099601, n02085620, n03786901, n04476259, n12998815,n04371774, n02814533, n02009229, n02500267, n04592741, n02119789, n02090622, n02132136,n02797295, n01740131, n02951358, n04141975, n02169497, n01774750, n02128757, n02097298,n02085782, n03476684, n03095699, n04326547, n02107142, n02641379, n04081281, n06596364,n03444034, n07745940, n03876231, n09421951, n02672831, n03467068, n01530575, n03388043,n03991062, n02777292, n03710193, n09256479, n02443484, n01728572, n03903868.
Conventional (non-spurious)
OOD datasets Huang et al.
[25] curate a diverse collection ofsubsets from iNaturalist [65], SUN [73], Places [81], and Texture [8] as large-scale OOD datasets for
ImageNet-1k, where the classes of the test sets do not overlap with ImageNet-1k.
We provide a briefintroduction for each dataset as follows.iNaturalist contains images of natural world [65].
It has 13 super-categories and 5,089 sub-categoriescovering plants, insects, birds, mammals, and so on.
We use the subset that contains 110 plant classesnot overlapping with ImageNet-1k.
SUN stands for the Scene UNderstanding Dataset [73].
SUN contains 899 categories that cover morethan indoor, urban, and natural places with or without human beings appearing in.
We use the subsetwhich contains 50 natural objects not showing in ImageNet-1k.
Places is a large scene photographs dataset [81].
It contains photos that are labeled with scenesemantic categories from three macro-classes: Indoor, Nature, and Urban.
The subset we use issampled from 50 categories that are not present in ImageNet-1k.
Texture stands for the Describable Textures Dataset [8].
It contains images of textures and abstractedpatterns.
As no categories overlap with ImageNet-1k, we use the entire dataset as in [25].
C Spurious OOD Datasets
In general, spurious attributes refer to statistically informative features that co-exist with the majorityof ID samples but do not necessarily capture cues related to the labels such as color, texture,background, etc [3, 2, 84, 16, 74].
A recent work [42] investigated a new type of hard OOD samples(called spurious OOD) that contain spurious or environmental features, but no object features relatedto the ID classes.
A concrete example is shown in Figure 6, where images of birds co-occur frequentlywith either the land background or water background.
Modern neural networks can spuriously relyon the image background ( e.g ., water or land) for classification instead of learning to recognize theactual object [54].
Ming et al.
[42] show that spurious OOD samples remain challenging for mostcommon OOD detection methods based on pure vision models such as ResNet [18].
For ID dataset, we use Waterbirds [55], which combines bird photographs from CUB-200 [69] withwater or land background images from P LACES [81].
For the spurious OOD dataset, we use the onecreated in [42] consisting of land and water background from Places [81].
[Figure]
D ID Classification Accuracy
[Figure]
Table 5 shows the multi-class classification accuracy on ImageNet-1k for methods in Table 2.
[Table]
E Implementation of CLIP-Based Baselines
E.1 Overview of Baselines
We review two previous works on CLIP-based OOD detection [14, 12] in Figure 7, which derivethe scoring function based on candidate OOD labels.
For a given task with ID label set Y in =
TextEncoder
Image
Encoder ...
“This is a photo of a __” score distribution
Frog
Bird
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
{ y 1 , y 2 , ..., y K } and candidate labels Y C
=
{ y K +1 , y K +2 , ..., y K + L } , where ideally Y in ∩ Y C = ∅ ,they construct a collection of text embeddings T ( t i ) , i ∈ { 1 , 2 , ..., K + L } .
Here, t i is the textprompt “ this is a photo of a ⟨ y i ⟩ ” for a label
y i .
For any test input image x , we can calculatethe label-wise matching score based on the cosine similarity between the image and text features:s i ( x ) =
I ( x ) ·T ( t i ) ∥I ( x ′ ) ∥·∥T ( t i ) ∥ .
Therefore, a detection score can be derived as:
[Equation]
where τ > 0 is the temperature scaling hyperparameter.
E.2 Obtaining OOD Candidate Labels
For the baseline methods, obtaining OOD candidate labels is a major challenge and limitation.
Recently, [12] propose ZO-CLIP, where a transformer (decoder) based on the image encoder of
CLIP is used to generate candidate labels.
The transformer is trained from scratch on the COCOdataset [40] with simple teacher forcing algorithms.
Although the decoder trained on COCO maywork well on CIFAR (ID), it is does not scale up to large-scale datasets such as ImageNet [9] wherecategories are not covered in COCO.
As a result, [12] only test on small-scale datasets with commonclasses such as CIFAR (ID).
We improve the baseline by using a high-quality caption generator pre-trained on much larger datasets,which not only saves computational overhead but can potentially improve the quality of generatedlabels.
The pipeline involves three components (see Figure 8):
• A caption generator.
Given an input image, it generates a caption serving as the textual
description of the input.
In this work, we consider ClipCap [43], which uses GPT-2 [52] togenerate captions based on CLIP’s image encoder.
ClipCap is pre-trained on a much larger
dataset Conceptual Captions [47] compared to COCO, which can be viewed as an enhanced
version of the ZO-CLIP baseline [12].
The checkpoints are publicly available 3 .
• A syntactic parser.
Given a caption, we extract noun objects using a parsing toolkit released
by spaCy 4 .
Those nouns can be used as candidate labels Y C of the input image.
• A filter module.
Unlike [12], we further enhance the baseline by adopting a filtering
technique to remove overlapping categories in Y C with ID labels Y in , which we detail below.
E.3 Label Filtering
Example.
To illustrate the effects of filtering, we begin with a concrete example where ID labelsare [“frog”,“bird” . . .
“truck”], as shown in Figure 8.
The generated labels (without filtering) ofan ID input of a bird sitting on a tree are [“bird”, “tree”].
Therefore, Y in ∪ Y C
=
[“frog”,“bird” . .
.“truck”,“bird”, “tree”].
Ideally, the softmax probability distribution over the concatenated labels wouldbe [0 , 0 . 5 , 0 , . .
. , 0 . 5 , 0] and by definition S ( x )
≈ 0 .
However, if we filter the generated labelsto eliminate nouns with similar meaning as ID, our concatenated labels would be [“frog”,“bird” . . .“truck”,“tree”] and the probability vector would be [0 , 1 , 0 , . . .
, 0] , which leads to a much higherscore S ( x )
= 1 .
In contrast, the generated labels for an OOD input with a man in the desert wouldbe [“man”, “desert”].
The resulting probability vector would be [0 , 0 , 0 , . .
, 1 , 0] and the score
S ( x ) = 0 .
Therefore, filtering makes it easier to separate ID inputs from OOD inputs ( c.f . Figure 9).
String-based filtering.
To implement the idea of filtering, we need a measurement of similaritybetween the generated labels and ID labels.
The simplest way is string-based filtering where agenerated label is filtered if it matches any ID labels (in the string format), as in the case above.
[Figure]
ID
F Alternative Scoring Functions
F.1 Does Energy-Based
Score Work Well on Inner Products?
In the visual domain, denote the ID label set Y in = { 1 , 2 , ..., K } and a discriminative neural classifier f : x (cid:55)→ f ( x ) ∈ R K that maps the input to K logits.
After the model is trained with the softmax cross-entropy loss, the logits can be converted to a categorical distribution:
[Equation]
p [Equation] where f j ( x ) is the j -th index of f ( x ) .
Given the softmax activations, Liu et al.
[41] propose to use energy score ( i.e ., free energy) for OOD detection, defined as:
[Equation]
Energy score is shown to better align with the logarithm of the in-distribution density P
x (i.e. P x | z =1 )compared to the classic MSP score and is supported by empirical evidence [41].
As we show in Section 3, although MCM applies the softmax function and converts the cosine similarity between the image and text features to a categorical distribution, it cannot be interpreted as ˆ p ( y | x ) .
Next, we empirically verify whether applying the energy function on the inner products s ( x ) leads to improved OOD detection performance, where the detection score is defined as:
[Equation]
The results are shown in Table 6.
We use CLIP-B/16 on ImageNet1k (ID).
The trend is contrary tothe observations of pure vision models [41].
With the default temperature τ = 1 , the performance ofthe energy score is significantly worse compared to MCM (AUROC 50 .
74% vs. 90 .
With asmall temperature ( τ = 0 . 005 ), the performance of the energy score is largely improved.
However,it is still less competitive compared to MCM (AUROC 84 .
42% vs. 90 .
This further suggeststhat while the (softmax) scaling function of MCM is similar to the classic MSP score, they have verydifferent properties.
with the energy score [41] ( to inner products) on ImageNet1k (ID). We use CLIP-B/16 as the backbone. The results are contrary to the observations for pure vision models where the energy score (applied to logits) demonstrates competitive performance.
[Table]
F.2 Other Scoring Functions
In addition to the energy score, we explore the effectiveness of several alternative scoring functions:
• The (negative) entropy of softmax scaled cosine similarities S entropy := − H ( s ) =(cid:80)
Ki
=1 s i ( x ) log s i ( x ) ;
• The variance of the cosine similarities denoted as S var ;
• The scaled difference between the largest and second largest cosine similarities S scaled :=
e s ˆ y ( x ) − s ˆ y 2 ( x ) where ˆ
y := argmax i ∈
[ K ] s i ( x ) and ˆ y 2 :
=
argmax i ̸
=ˆ y,i ∈
[ K ] s i ( x ) .
Similar to the energy score, for S entropy , we select the hyperparameter τ ∈ { 0 . 005 , 0 . 01 , 0 .
1 , 1 , 10 }and report the performance with the best τ .
As shown in Table 7, MCM still gives the most promisingresults compared to the other three alternative scores across most OOD test sets.
G MCM for ResNet-based CLIP Models
In the main paper, we demonstrated the effectiveness of the MCM score across a wide range of tasksfor CLIP-B and CLIP-L, both of which use ViT as the image encoder.
In this section, we investigate
[Table]
[Table]
the effectiveness of MCM on ResNet-based CLIP.
Specifically, we use RN50x4 (178.3M)
[Table]
whichshares a similar number of parameters as CLIP-B/16 (149.6M).
The results are shown in Table 8.
It can be seen that MCM still shows promising results with ResNet-based CLIP models, and theperformance is comparable between RN50x4 and CLIP-B/16 ( 89 . 97 vs. 90 . 62 in AUROC).
H MCM Without Softmax Scaling
In Section 3, we provide theoretical justifications on the necessity of softmax scaling for CLIP-likemodels and we further verify our theoretical insights based on ImageNet-100 (ID) in Section 5(Figure 4).
To further verify our observations empirically, we show OOD detection performancefor a wide range of datasets based on the maximum cosine similarity score S woMCM ( x ′ ;
Y in , T , I ) =
max i ∈
[ K ] s i ( x ′ ) .
The results are shown in Table 9.
For easy tasks such as Food-101 [32], StanfordCars [32], and Oxford-Pet [49] as ID, the performance of maximum cosine similarity score is similarto MCM (see Table 1 and Table 2).
However, for more challenging tasks such as ImageNet-20 and
ImageNet-1k, MCM significantly outperforms that without softmax scaling.
For example, the average
FPR95 is improved by 11.33% on ImageNet-20 and 22.02% on ImageNet-1k, which highlights thenecessity of a proper scaling function for CLIP-based OOD detection.
[Table]