Paper: Negative Label Guided OOD Detection with Pretrained Vision-Language Models (ICLR-2024)

Reviewer comment (Writing):
I enjoy reading how you move from basic data modelling to a more realistic data modelling in B.3. However, it would be better to point out why the contents in B.3 is more general, which can increase the interests of readers for understanding the proposed score.

Author response:
**A2:** Thanks for your comments! In the theoretical analysis of the main text, to simplify the theoretical model and improve the readability of the article, we assume that the probability of a sample $x$ matching with each negative label is $p$. Therefore, for $M$ negative labels, we consider the number of matches between an image $x$ and them to follow a binomial distribution $B(M, p)$. However, due to the large number of negative labels, covering a wide semantic space, their affinity with the sample $x$ is variable.

Specifically, in Appendix B.3, we assume that the probability of an image $x$ matching with a negative label is $p_i$, where different negative labels have different probabilities. This better reflects the real-world scenario, and thus we consider the number of matches between an image $x$ and them to follow a Poisson binomial distribution.

We will provide more details in the main text to help readers better understand the general case in B.3.

Reviewer comment (Presentation):
In Table 2, all methods can obtain good results (AUROC>90% or FPR95<10% in the most cases). I actually do not see the necessarity to put that table in the main content. It would be better to move some interesting parts in Appendix to the main content.

Author response:
**A3:** Thanks for your suggestions! We will reorganize our paper in the updated version.

Reviewer comment (Experiments):
What will happen if we select all different words from the ID classes?

Author response:
**A4:** Thanks for your comments! Here are the results (last row) that we select all different words from the ID classes.
| Negative Label Number | iNaturalist |        |   SUN  |        | Places |        | Textures |        | Average |        |
|-----------------------|:-----------:|:------:|:------:|:------:|:------:|:------:|:--------:|:------:|:-------:|:------:|
|                       |    AUROC↑   | FPR95↓ | AUROC↑ | FPR95↓ | AUROC↑ | FPR95↓ |  AUROC↑  | FPR95↓ |  AUROC↑ | FPR95↓ |
| 100                   |    53.18    |  99.30 |  53.34 |  99.44 |  53.09 |  99.05 |   53.04  |  99.18 |  53.16  |  99.24 |
| 200                   |    99.23    |  2.59  |  89.83 |  50.87 |  85.15 |  62.33 |   84.23  |  69.01 |  89.61  |  46.20 |
| 1,000                 |    99.62    |  1.35  |  92.42 |  36.96 |  88.24 |  49.44 |   86.11  |  56.79 |  91.60  |  36.14 |
| 2,000                 |    99.67    |  1.17  |  93.60 |  29.58 |  89.90 |  42.83 |   88.84  |  48.10 |  93.01  |  30.42 |
| 5,000                 |    99.64    |  1.39  |  94.83 |  23.28 |  91.28 |  37.58 |   89.97  |  44.26 |  93.93  |  26.63 |
| 10,000                |    99.49    |  1.91  |  95.49 |  20.53 |  91.64 |  35.59 |   90.22  |  43.56 |  94.21  |  25.40 |
| 20,000                |    99.18    |  3.16  |  95.24 |  21.47 |  91.16 |  37.23 |   89.83  |  44.40 |  93.85  |  26.56 |
| 30,000                |    98.90    |  4.42  |  94.98 |  22.40 |  90.75 |  39.11 |   89.12  |  47.00 |  93.44  |  28.23 |
| 50,000                |    98.39    |  6.89  |  94.48 |  25.15 |  90.20 |  41.62 |   87.71  |  53.01 |  92.70  |  31.67 |
| All                   |    97.95    |  9.18  |  94.16 |  27.26 |  89.93 |  42.78 |   86.67  |  56.44 |  92.18  |  33.91 |

Reviewer comment (Presentation):
In Table 2, all methods can obtain good results (AUROC>90% or FPR95<10% in the most cases). I actually do not see the necessarity to put that table in the main content. Can we move it to appendix?

Author response:
**A3:** Thanks for your suggestions! We will reorganize our paper in the updated version.

Reviewer comment (Experiments):
What will happen if we select all different words from the ID classes?

Author response:
**A4:** Thanks for your comments! Here are the results (last row) that we select all different words from the ID classes.
| Negative Label Number | iNaturalist |        |   SUN  |        | Places |        | Textures |        | Average |        |
|-----------------------|:-----------:|:------:|:------:|:------:|:------:|:------:|:--------:|:------:|:-------:|:------:|
|                       |    AUROC↑   | FPR95↓ | AUROC↑ | FPR95↓ | AUROC↑ | FPR95↓ |  AUROC↑  | FPR95↓ |  AUROC↑ | FPR95↓ |
| 100                   |    53.18    |  99.30 |  53.34 |  99.44 |  53.09 |  99.05 |   53.04  |  99.18 |  53.16  |  99.24 |
| 200                   |    99.23    |  2.59  |  89.83 |  50.87 |  85.15 |  62.33 |   84.23  |  69.01 |  89.61  |  46.20 |
| 1,000                 |    99.62    |  1.35  |  92.42 |  36.96 |  88.24 |  49.44 |   86.11  |  56.79 |  91.60  |  36.14 |
| 2,000                 |    99.67    |  1.17  |  93.60 |  29.58 |  89.90 |  42.83 |   88.84  |  48.10 |  93.01  |  30.42 |
| 5,000                 |    99.64    |  1.39  |  94.83 |  23.28 |  91.28 |  37.58 |   89.97  |  44.26 |  93.93  |  26.63 |
| 10,000                |    99.49    |  1.91  |  95.49 |  20.53 |  91.64 |  35.59 |   90.22  |  43.56 |  94.21  |  25.40 |
| 20,000                |    99.18    |  3.16  |  95.24 |  21.47 |  91.16 |  37.23 |   89.83  |  44.40 |  93.85  |  26.56 |
| 30,000                |    98.90    |  4.42  |  94.98 |  22.40 |  90.75 |  39.11 |   89.12  |  47.00 |  93.44  |  28.23 |
| 50,000                |    98.39    |  6.89  |  94.48 |  25.15 |  90.20 |  41.62 |   87.71  |  53.01 |  92.70  |  31.67 |
| All                   |    97.95    |  9.18  |  94.16 |  27.26 |  89.93 |  42.78 |   86.67  |  56.44 |  92.18  |  33.91 |

Reviewer comment (Evaluation):
OOD detection methods have different performance on different datasets. On some datasets, the performance is extremely good. Can we see the difference among these datasets? Why can OOD detection be easily addressed on some datasets?

Author response:
Thanks for your comments! We think that the performance differences mainly stem from the characteristics of the OOD datasets themselves. In  Figure 4 and Figure 5 of the Appendix, we provide a few sample images from the ID and OOD datasets for reference. The ID dataset ImageNet contains a large number of categories related to animals and food, while iNaturalist consists of images of natural plants. SUN and Places datasets contain images of natural landscapes. Therefore, compared to other OOD datasets, iNaturalist has the largest differences in category terms compared to ImageNet, such as animals vs. plants. Therefore, introducing negative labels can significantly improve the OOD detection performance. On the other hand, SUN and Places datasets often contain multiple elements from the natural world beyond their annotated ground truth, making them more prone to confusion with the ID data.

Furthermore, recent research trends have also shown that achieving high performance on iNaturalist is quite common. For example, ASH [1] and NNGuide [2] both achieve over 97% AUROC on iNaturalist, while GEN [3] achieves 99.13% AUROC.

Reviewer comment (Evaluation):
Especially for results based on CLIP-B/16 with various ID datasets, the performance is too high. More explanation is need.

Author response:
Thanks for your suggestions! We follow MCM to conduct experiments on the large-scale benchmark in Table 1 and small-scale benchmarks in Table 2. The performance of MCM in Table 2 is very high and our NegLabel outperforms MCM. Our opinion about the high performance is consistent to MCM, i.e., the small-scale ID datasets have less challenges for VLMs.

Reviewer comment (Presentation):
Figures might be better to illustrate the final performance in Section 4.

Author response:
Thanks for your suggestions! We will replace some tables with figures in Section 4 in the updated paper.

Reviewer comment (Theory):
What is the relationship between Bernoulli distribution and binomial distribution? This is a well-known result in statistics?

Author response:
The Bernoulli distribution is a special case of the binomial distribution, where n = 1. Symbolically, X ~ B(1, p) has the same meaning as X ~ Bernoulli(p). Conversely, any binomial distribution, B(n, p), is the distribution of the sum of n independent Bernoulli trials, Bernoulli(p), each with the same probability p.[4]

Reviewer comment (Theory):
Why is B.3 more general than things demonstrated in main context?

Author response:
Thanks for your comments! In the theoretical analysis of the main text, to simplify the theoretical model and improve the readability of the article, we assume that the probability of a sample $x$ matching with each negative label is $p$. Therefore, for $M$ negative labels, we consider the number of matches between an image $x$ and them to follow a binomial distribution $B(M, p)$. However, due to the large number of negative labels, covering a wide semantic space, their affinity with the sample $x$ is variable.

Specifically, in Appendix B.3, we assume that the probability of an image $x$ matching with a negative label is $p_i$, where different negative labels have different probabilities. This better reflects the real-world scenario, and thus we consider the number of matches between an image $x$ and them to follow a Poisson binomial distribution.

We will provide more details in the main text to help readers better understand the general case in B.3.

Reviewer comment (Presentation):
Figure 1 is clear and important. However, it makes me think about one question: in a statistical view, what should the right subfigure be?

Author response:
**A1:** Thanks for the comments! The actual subgraph would be very difficult to visualize, as the x-axis would contain 1,000 (ID labels) + 10,000 (negative labels) elements. We have provided the visualization results of the top 5 ID labels and negative labels similarities in Figures 4 and 5 in the appendix.

Reviewer comment (Presentation):
Why does grouping strategy appear in Section 3.2 instead of 3.1? It seems that Grouping Strategy is a more advanced way to pre-process negative labels?

Author response:
**A3:** Thanks for this question. Here we detail our proposed pipeline of Negative-Label-based OOD detection: corpus -> NegMining -> Grouping -> NegLabel Score. Specifically, our method requires first selecting appropriate negative labels through NegMining (as shown in Section 3.1), and then designing a NegLabel score using these negative labels (as shown in Section 3.2 part 1). The Grouping strategy is an optimization of the NegLabel score. By grouping the selected negative labels and calculating the NegLabel score for each group, the average of all groups is taken as the final OOD score. Therefore, the Grouping Strategy is not a more advanced way to preprocess negative labels. It does not involve the preprocessing process of negative labels. It is suitable to be placed in Section 3.2 part 2 as it optimizes and enhances the OOD score.

To better understand the grouping strategy, we provide a toy example. Let's assume for an ID image, there is a relatively high response in the ID labels, but coincidentally, there is also a relatively high response in the negative labels, with both responses being of comparable magnitude. Without grouping, after softmax function, the probabilities for ID and negative labels might both be around 0.5, resulting in a score of 0.5.

However, if we divide the negative labels into 10 groups, this misclassification would be included in one of the groups at most. The score for this group would be 0.5, while the rest of the groups would be close to 1.0. Thus, the final score would be calculated as 1 * 0.9 + 0.5 * 0.1 = 0.95. This illustrates how the Grouping Strategy helps mitigate the impact of such misclassifications.

Reviewer comment (Theory):
Can the authors explain why is the case considered in B.3 more general? More explanation can make readers know your analysis better.

Author response:
**A4:** Thanks for your comments! In the theoretical analysis of the main text, to simplify the theoretical model and improve the readability of the article, we assume that the probability of a sample $x$ matching with each negative label is $p$. Therefore, for $M$ negative labels, we consider the number of matches between an image $x$ and them to follow a binomial distribution $B(M, p)$. However, due to the large number of negative labels, covering a wide semantic space, their affinity with the sample $x$ is variable.

Specifically, in Appendix B.3, we assume that the probability of an image $x$ matching with a negative label is $p_i$, where different negative labels have different probabilities. This better reflects the real-world scenario, and thus we consider the number of matches between an image $x$ and them to follow a Poisson binomial distribution.

We will provide more details in the main text to help readers better understand the general case in B.3.

Reviewer comment (Presentation):
Figure 1 is clear and important. However, it makes me think about one question: in a statistical view, what should the right subfigure be?

Author response:
**A1:** Thanks for the comments! The actual subgraph would be very difficult to visualize, as the x-axis would contain 1,000 (ID labels) + 10,000 (negative labels) elements. We have provided the visualization results of the top 5 ID labels and negative labels similarities in Figures 4 and 5 in the appendix.

Reviewer comment (Presentation):
Why does grouping strategy appear in Section 3.2 instead of 3.1? It seems that Grouping Strategy is a more advanced way to pre-process negative labels?

Author response:
**A3:** Thanks for this question. Here we detail our proposed pipeline of Negative-Label-based OOD detection: corpus -> NegMining -> Grouping -> NegLabel Score. Specifically, our method requires first selecting appropriate negative labels through NegMining (as shown in Section 3.1), and then designing a NegLabel score using these negative labels (as shown in Section 3.2 part 1). The Grouping strategy is an optimization of the NegLabel score. By grouping the selected negative labels and calculating the NegLabel score for each group, the average of all groups is taken as the final OOD score. Therefore, the Grouping Strategy is not a more advanced way to preprocess negative labels. It does not involve the preprocessing process of negative labels. It is suitable to be placed in Section 3.2 part 2 as it optimizes and enhances the OOD score.

To better understand the grouping strategy, we provide a toy example. Let's assume for an ID image, there is a relatively high response in the ID labels, but coincidentally, there is also a relatively high response in the negative labels, with both responses being of comparable magnitude. Without grouping, after softmax function, the probabilities for ID and negative labels might both be around 0.5, resulting in a score of 0.5.

However, if we divide the negative labels into 10 groups, this misclassification would be included in one of the groups at most. The score for this group would be 0.5, while the rest of the groups would be close to 1.0. Thus, the final score would be calculated as 1 * 0.9 + 0.5 * 0.1 = 0.95. This illustrates how the Grouping Strategy helps mitigate the impact of such misclassifications.

Reviewer comment (Theory):
Can the authors explain why is the case considered in B.3 more general? More explanation can make readers know your analysis better.

Author response:
**A4:** Thanks for your comments! In the theoretical analysis of the main text, to simplify the theoretical model and improve the readability of the article, we assume that the probability of a sample $x$ matching with each negative label is $p$. Therefore, for $M$ negative labels, we consider the number of matches between an image $x$ and them to follow a binomial distribution $B(M, p)$. However, due to the large number of negative labels, covering a wide semantic space, their affinity with the sample $x$ is variable.

Specifically, in Appendix B.3, we assume that the probability of an image $x$ matching with a negative label is $p_i$, where different negative labels have different probabilities. This better reflects the real-world scenario, and thus we consider the number of matches between an image $x$ and them to follow a Poisson binomial distribution.

We will provide more details in the main text to help readers better understand the general case in B.3.

Reviewer comment (Theory):
**Assumption about CLIP's latent space** The assumption that ID image and corresponding ID label embeddings are close in CLIP's latent space is actually false. The contrastive learning strategy of CLIP is based on cosine similarity and thus textual and visual embeddings are actually on different manifolds see (Weixin Liang et al. 2022) for more details. Using ID labels as proxies for the ID images may thus produce unexpected results. In my understanding, the negative labels are compared to the images using cosine similarity, and thus having them close to ID textual embeddings still makes sense but the authors should be cautious here.

Author response:
**A1:** Thank you for this valuable comment. We apologize for the inappropriate description. As mentioned in the question, images and text belong to different manifolds in the embedding space of CLIP, so they cannot be considered *close*. We will revise this description in the last paragraph of Section 3.1.

Besides, we would reiterate our OOD detection setting based on CLIP-like VLM models. We only work with a pre-trained CLIP-like model and a specific set of class labels or names for a zero-shot classification task. And the ID images are not available under this zero-shot setting. Hence，using ID labels as the proxies of ID images for negative label selection is intuitive, and it is experimentally verified to be effective in our method. We believe that not relying on the accessibility of ID images allows our method to be used in a wider range of scenarios and is more consistent with the working conditions of CLIP-like models.

Reviewer comment (Experiments):
**Comparison with related work** The method is close in spirit to CLIPN (Wang et al. 2023) as well as ZOC (Esmaeilpour et al 2022). The former fine-tunes CLIP to incorporate negative prompts to assess the probability of a concept not being present in the image. An extensive comparison with CLIPN would be overkill but some discussion about it would be welcome. The second also performs zero-shot OOD using maximum softmax probability on the labels extended with the object detected in the images. Comparisons with ZOC are presented in the supplementary but would better belong in the main paper IMO as it is one of the few recent zero-shot OOD detection baselines. To be fair this comparison should be provided with the same set-up as in Table 2.

Author response:
**A2:** Thanks for your suggestions! We will add some discussions with CLIPN and ZOC in the updated version.

**Discussion about CLIPN:**

CLIPN utilizes additional datasets to train a text encoder that can understand negative prompts. It discriminates OOD data by comparing the similarity differences between the outputs of two text encoders and the image encoder. **On the other hand, our method does not require additional data or training.** It only introduces negative labels from the corpus to accomplish zero-shot OOD detection tasks. Therefore, in terms of the idea, both methods aim to introduce negative semantics of the ID category to reject inputs unrelated to the ID category. In terms of computational complexity, our method is slightly higher than CLIPN, but it is almost negligible compared to the computational complexity of CLIP itself. In terms of specific performance, our method has a higher average performance on the ImageNet-1k benchmark. Moreover, compared to CLIPN and NegLabel on CLIP-B/16 and CLIP-B/32, our method NegLabel is more robust to model architectures.

|        Method        | iNaturalist |       |  SUN  |       | Places |       | Textures |       | Average |       |
|:--------------------:|:-----------:|:-----:|:-----:|:-----:|:------:|:-----:|:--------:|:-----:|:-------:|:-----:|
|                      |    AUROC    | FPR95 | AUROC | FPR95 |  AUROC | FPR95 |   AUROC  | FPR95 |  AUROC  | FPR95 |
|   CLIPN (CLIP-B/32)  |    94.67    | 28.75 | 92.85 | 31.87 |  86.93 | 50.17 |   87.68  | 49.49 |  90.53  | 40.07 |
|   CLIPN (CLIP-B/16)  |    95.27    | 23.94 | 93.93 | 26.17 |  90.93 | 40.83 |   92.28  | 33.45 |  93.10  | 31.10 |
| NegLabel (CLIP-B/32) |    99.11    |  3.73 | 95.27 | 22.48 |  91.72 | 34.94 |   88.57  | 50.51 |  93.67  | 27.92 |
| NegLabel (CLIP-B/16) |    99.49    |  1.91 | 95.49 | 20.53 |  91.64 | 35.59 |   90.22  | 43.56 |  94.21  | 25.40 |

**Discussion about ZOC:**

As for ZOC, the authors do not report the performances on ImageNet-1k benchmark, and there is no third-party implementation. We think it is mainly because the COCO-pretrained image captioner can not handle the large-scale benchmark ImageNet-1k with 1,000 categories. We try to reproduce an upgraded version of ZOC on ImageNet-1k benchmark. We leveraged the capabilities of the multimodal model LLaVA [1] and ChatGPT to migrate ZOC to large-scale data sets. The idea comes from MCM's discussion and reproduction of ZOC.

We first used LLaVA to describe each sample and extract the entities contained in the description. Then we encountered a difficulty. LLaVA's descriptions of a large number of ID samples were slightly different from their corresponding ImageNet-1k labels, such as dog v.s. Alaskan Malamute. As a result, the ID label filtering policy cannot take effect. In order to overcome this problem, we collected all entities generated by LLaVA and used ChatGPT to filter words similar to the ImageNet-1k label. The remaining words are treated as candidate OOD labels of ZOC. The performances of the upgraded version of ZOC are shown in the table below.

|  Method  | iNaturalist |          |    SUN    |           |   Places  |           |  Textures |           |  Average  |           |
|:--------:|:-----------:|:--------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
|          |    AUROC    |   FPR95  |   AUROC   |   FPR95   |   AUROC   |   FPR95   |   AUROC   |   FPR95   |   AUROC   |   FPR95   |
|    ZOC   |    86.09    |   87.30  |   81.20   |   81.51   |   83.39   |   73.06   |   76.46   |   98.90   |   81.79   |   85.19   |
| NegLabel |  **99.49**  | **1.91** | **95.49** | **20.53** | **91.64** | **35.59** | **90.22** | **43.56** | **94.21** | **25.40** |

In addition, since the experimental settings of CLIPN and ZOC are different from NegLabel, we will update the comparison of their experimental results in Table 1 and some discussions in the Appendix A.

Reviewer comment (Experiments):
**Misleading baseline presentation** Even if the MSP, ODIN, or Energy Logit baseline are generally used on models fine-tuned on a downstream task, one could also use these detectors in a zero-shot setting. Classifying them in purely zero-shot can be misleading eg. MCM can be seen as zero-shot MSP on CLIP's output probabilities.

Author response:
**A3:** Thanks for your suggestions! We will add some zero-shot baselines in Table 1.

Reviewer comment (Presentation):
**Short Related Work** The related work section is relatively short, especially for such a prolific field as OOD detection. I don't think it is really detrimental to the paper but a discussion on the sense of OOD detection for foundation models trained on vast and various data would be welcome.

Author response:
**A4:** Thanks for your suggestions! We will add some related works in Section 5 in the updated paper (before the author-reviewer deadline).

Reviewer comment (Experiments):
I wonder why the authors did not add the simple and strong baseline Mahaloanobis (Lee et al. 2018) /SSD (Sehwag et al. 2022).

Author response:
**A5:** Thanks for your suggestions! We will add Mahaloanobis as the baseline in Table 1. 

SSD is based on self-supervised learning, and our baselines are based on the pretrained or finetuned CLIP. Therefore, they can not be compared in a fair way.

Reviewer comment (Experiments):
No ablations are provided on $n_g$. How do the authors set this parameter and what is its impact?

Author response:
**A6:** We apologize for this misunderstanding, and we have provided the ablation experiments on $n_g$ in Table 16 of the appendix.

Reviewer comment (Novelty):
Would the hard OOD detection task be better qualified as Open-Set Recognition?

Author response:
**A7:** This is a good question! The hard OOD detection task is set by MCM and we follow it. We think using different subsets of ImageNet as ID and OOD datasets (e.g., ImageNet10 v.s. ImageNet 100) is similar to the setting of OSR. But the setting of the spurious OOD detection task, i.e., WaterBirds v.s. several Spurious OOD, is quite different from OSR.