Paper: Out-of-Distribution Detection with Negative Prompts (ICLR-2024)

Reviewer comment (Presentation):
Meta-Net in Figure 6 is not introduced in the paper.

Author response:
**A1:** Thanks for pointing out the problem. Accoridngly, we have added the following explanations in our revision.

The Meta-Net in Figure 6 is a lightweight neural network built with a two-layer bottleneck structure (Linear-ReLU-Linear), with the hidden layer reducing the input dimension by 16×.

Reviewer comment (Novelty):
The motivation and model design of this paper are similar to DualCoOp. The authors claim that the proposed method could learn negative prompts to capture negative features compared to DualCoOp, but there is no visual or quantitative evidence to verify this statement.

Author response:
**A2:** Thank for your insightful comments and kind suggestion. Accordingly, we have added the following explanations to clarify the difference bewteen our method and DualCoOp.

- The  roles of negative prompts in DualCoOp and LSN are different. In DualCoOp, negative prompts act as a threshold for positive prompts of the corresponding category, whereas in LSN, negative prompts focus on learning generic negative features across all other categories except the corresponding category. 
- In DualCoOp, positive and negative prompts are optimized simultaneously with Asymmetric Loss, which make the learning of negative logits intractable, resulting DualCoOp tend to generate false negative responses, which is consistent with the observation in DualCoOp++[1]. In contrast, in LSN, the learning of positive and negative prompts is separated, and the learned negative prompts successfully focus on different region than the positive prompts, as demonstrated in Figure2 in our paper. 
- DualCoOp mainly focuses on the **multi-label recognition (MLR)**, which is a **closed-set** problem. This needs class names of the unseen classes for testing. In contrast, LSN focuses on the **OOD detection**, which is an **open-set** problem. This does not need unseen classes.

In response to your constructive comments, we conducted the following experiments on ImageNet-100 to hilight the difference from an empirical perspective. The corresponding results and discussions have been added to the revised paper. These results show that LSN can outperform DualCoOp.
|          | Average |       |
|----------|---------|-------|
| Method   |  PFR95   | AUROC |
| DualCoOp |  23.47   | 95.64 |
| LSN      |  19.70   | 96.22 |

Reviewer comment (Experiments):
Some existing OOD detection methods proposed in 2022 and 2023 are not compared or discussed[2][3][4].

Author response:
**A3:** Thanks for your valuable comments. Following your kind suggrestion, we conducted comparative experiments with KNN and LINe on raw CLIP model. The results on ImageNet-1k are as follows:

|                     |  Average |       |
|---------------------|---------|-------|
| Method              |  PFR95   | AUROC |
| KNN(k=1)            |  97.05   | 69.27 |
| KNN(k=10)           |  97.61   | 68.38 |
| KNN(k=100)          |  98.52   | 61.80 |
| LINe($\delta = 0$)    | 87.31   | 74.38 |
| LINe($\delta = 0.04$) | 70.46   | 85.63 |
| LINe($\delta = 0.07$) |  62.77   | 87.32 |
| LSN                 |  30.22   | 92.96 |

Our experimental results show that LSN outperforms these post-hoc methods on raw CLIP model. And the reason is the same as that in **A1** to **Reviewer Dyvx**, please see that.

As the mentioned DML requires re-training the model, it is challenging to report detailed results. Thus, we will add the results when finishing our experiments. We have added these discussions and outstanding works in our revision.

Reviewer comment (Theory):
In equation (9), the positive score and the negative score are obtained independently. If the obtained positive category is different from the negative category, this could lead to mistakenly recognizing a correct positive prediction as out-of-distribution.

Author response:
**A5:** We appreciate your careful review and indepth question. What you describe is exactly the reason we get positive score and negative score respectively. It can indeed happen that positive and negative categories are different, especially when the number of class is large. In this case, for ID samples:

$$\max _c Softmax( \cos \left(\boldsymbol{f}(\mathbf{x}), \boldsymbol{g}\circ \boldsymbol{V}(l_c) \right)  / \tau) - \min _c Softmax( \cos \left(\boldsymbol{f}(\mathbf{x}), \boldsymbol{g}\circ \bar{\boldsymbol{V}}(l_c) \right) / \tau) > \max _c (Softmax( \cos \left(\boldsymbol{f}(\mathbf{x}), \boldsymbol{g}\circ \boldsymbol{V}(l_c) \right)  / \tau) -  Softmax( \cos \left(\boldsymbol{f}(\mathbf{x}), \boldsymbol{g}\circ \bar{\boldsymbol{V}}(l_c) \right) / \tau))$$
Getting positive and negative scores independently can alleviate the problem you describe and this can be seen as an integration of positive classifier and negative classifier.

Reviewer comment (Novelty):
This paper is related to negative learning or learning with complementary labels. The authors may consider adding some related discussion in the related work section.

Author response:
**A1:**  Thanks for your constructive suggestion. We have added a section of **Complementary Label Learning** to the related work in our revised version

Reviewer comment (Evaluation):
The proposed method may double the training and inference time with the negative prompts.

Author response:
**A2:** We agree with your point. Due to the use of both positive and negative prompts, we indeed double our training and inference time. We will add the corresponding overhead analysis into our revision.

Reviewer comment (Novelty):
I found a related work that the authors may add discussion in the related work section: "How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?, IJCV 2023."

Author response:
**A3:** Thanks for mentioning the outstanding paper. This paper investigates the effect of fine-tuning on OOD detection in large vision-language models and the maximum concept matching (MCM) score is highlighted as effective. We have added the above discussion to the related work.

Reviewer comment (Experiments):
What's the ID dataset in Table 3?

Author response:
**A4:**  We apologize for the missing information. Accordingly, we have added the following explanations into our revised paper.

For the small-scale dataset, we follow the setting in ZOC[1]. For CIFAR10, 6 classes are selected as ID classes, and the 4 remaining classes are selected as OOD classes. Experiments are conducted on 5 different splits. For CIFAR+10, 4 non-animal classes from CIFAR10 are selected as ID classes, and 10 animal classes from CIFAR100 are selected as OOD classes. For CIFAR+50, 4 non-animal classes from CIFAR10 are selected as ID classes, and all 50 animal classes from CIFAR100 are selected as OOD classes. For CIFAR100, 20 consecutive classes are selected as ID classes. The remaining 80 classes are selected as OOD classes. For TinyImagenet, 20 classes are used as ID classes, and the remaining 180 classes are used as OOD classes. All the split is the same as ZOC. We give detailed split information in Table6.

Reviewer comment (Evaluation):
Why do CoOp/CoCoOp and CoOp/CoCoOp + LSN achieve the same ID results in Table 1 and Table 2?

Author response:
**A5:** Thanks for pointing out this potentially confusing description.

In our experiments we notice that the classification performance of negative prompts is slightly less than that of positive prompts. We are not committed to improving ID classification performance. To avoid further trouble, we only use positive prompts when determining the classes of ID samples. We have clarified this point in our revision.

Thanks again for your careful review.

Reviewer comment (Evaluation):
The ID results of CoOp/CoCoOp appear to be significantly lower than other baselines such as NPOS in Table 2. Can the authors explain some reasons?

Author response:
**A6:** We do not run NPOS on ImageNet-1k because it will consume a lot of resources. We get their experimental results directly from the original NPOS paper. We believe that the gap in ID classification performance comes from the difference in the amount of training data as well as the difference in training methods. NPOS uses the full ImgaNet-1k training set to fine-tune the CLIP model, which allows the fine-tuned model to fit better to the ImgaNet-1k test dataset. In contrast, LSN uses only 64 samples per class on ImageNet-1k and only learns prompts  without making changes to the model.

Reviewer comment (Presentation):
It is suggested to use `\citep` rather than `\cite` in the latex code.

Author response:
**A7:** Thank you very much for the kind reminder; in our resubmitted version, we have fixed these errors.

Reviewer comment (Writing):
Typo: 'we use we use' at the bottom of page 6.

Author response:
**A7:** Thank you very much for the kind reminder; in our resubmitted version, we have fixed these errors.

Reviewer comment (Novelty):
The overall prompt learning approach is still based on CoOp without a lot of innovation.

Author response:
**A1:** Thank you for your valuable comment. We agree with your point that LSN is built upon the success of CoOp, while we would like to provide more explanations to highlight the novel design for OOD detection. 

Although our approach to learn negative prompts looks basically the same as CoOp,  there is a fundamental difference between positive prompt learning and negative prompt learning. In positive prompt learning, using a shared prompt across classes is enough to achieve good performance. This is because in positive prompt learning, the positive features of each class are carried by the class names, and the positive prompt is only used to calibrate these features to the downstream dataset. In negative prompt learning, on the other hand, the situation is completely different. The negative features need to be contained in the learned negative prompts, and the class names appear to be less important. Diversity is the key to the success of negative prompts. We illustrate this in Table 5.

Reviewer comment (Theory):
The way to learn negative prompts is to mine the general negative features that each class of samples does not have but all other classes have, i.e., the negative classifier produces low activation values for that class and high activation values for other classes. Does this result in learning what are actually generic background features rather than general features that all other classes have?

Author response:
**A3:** We agree with your indepth point. It is indeed possible for the model to learn generic background features as negative features. However, it is noteworthy that even when background features are learned, they can at times serve as effective negative features. For example, consider the ID classes Horse, Wolf, and Pet Cat, and the OOD class Tiger. Due to similar positive features with pet cats, tigers can easily be mistaken for ID samples. But if we learn further about negative features the situation is different. Since horses and wolves are often found in grassland and pet cats are unlikely to be found in grassland, then the background grassland is likely to be learned as a negative feature for pet cats. The background of tigers is also usually grassland, and then the classifier can easily determine that tigers are not pet cats.

Reviewer comment (Experiments):
Comparison and related works to state-of-the-arts are missing (e.g., NNGuide [1], ASH [2], CLIPN [3], [4])

Author response:
**A1:** Thanks for your contructive comments. In accordance with your suggestion, we conduct comparison experiments with NNGuide and ASH to enrich our analysis. The results on ImageNet-1k are as follows:
|                                | Average |       |
|--------------------------------|---------|-------|
| Method                         | PFR95   | AUROC |
| NNGuide(k=1)                   | 93.02   | 80.18 |
| NNGuide(k=10)                  | 94.46   | 78.96 |
| NNGuide(k=100)                 | 96.38   | 75.52 |
| ASH-B(@90) |  91.76   | 66.52 |
| ASH-B(@65) | 85.05   | 68.17 |
| ASH-P(@90) | 83.10  | 69.78 |
| ASH-P(@65) | 65.11  | 79.85 |
| ASH-S(@90) | 78.10  | 73.20 |
| ASH-S(@65) | 65.24  | 79.25 |
| LSN        | 30.22   | 92.96 |

Our experimental results show that LSN outperforms these post-hoc methods on raw CLIP model. We believe the reason for the poor performance of these pos-hoc methods is that the **the difference in training data**. Many pos-hoc methods are designed on ImageNet pre-trained networks, where **only ID datas are used during training**. In contrast, when training CLIP, **both ID datas and OOD datas are used**. The difference in training data leads to different activation of OOD data.  Another reason is that the pos-hoc method relies heavily on the choice of hyperparameters. The hyperparameters need to be re-selected on different models.

CLIPN also employes CLIP to perform OOD detection, coming up with ideas similar to ours. We compare the best results of CLIPN and LSN on ImageNet-1k with CLIP ViT-B-16 as follows:

|        | iNaturalist |           | SUN       |           | Places    |           | Textures  |           | Average   |           |
|--------|-------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
| Method | PFR95       | AUROC     | PFR95     | AUROC     | PFR95     | AUROC     | PFR95     | AUROC     | PFR95     | AUROC     |
| CLIPN  | 23.94       | 95.27     | **26.17** | 93.93     | **33.45** | **92.28** | 40.83     | **90.93** | 31.10     | **93.10** |
| LSN    | **21.56**   | **95.83** | 26.32     | **94.35** | 34.48     | 91.25     | **38.54** | 90.42     | **30.22** | 92.96     |

- Experimental results show that LSN achieves comparable performance with CLIPN. 
- We have to point out that **this is not a fair comparison**. This is because, to endow OOD detector with the ability to say "no", CLIPN leverages a large-scale dataset [1] (containing 3.3 million image-text pairs) to finetune the text encoder.
- These results highlight the contribution of the proposed approach to learn negative prompts. Specifically, LSN only needs to use few shot samples to learn negative prompts, which can be done in a very short time.

Inspired by your constructive comments, we summarize the differences between LSN and CLIPN as follows.

- LSN only requires learning negative prompts at a small computational cost without training "no" text encoder and achieves performance comparable with CLIPN.
- LSN points out that the diversity of negative prompts is the key to success, and the use of class-specific negative prompts leads to a reduction in FPR95 from 81.27% to 21.94% compared to the use of class-shared negative prompts, please see Table 5.
- LSN focuses on prompt learning and points out how negative prompt learning differs from positive prompt learning, while CLIPN focuses more on learning "no" text encoder.

We will add the above results and discussion into our revision to highlight the difference bewteen our work and the mentioned work. Thanks again for mentioning these outstanding papers.

Reviewer comment (Novelty):
The main concept of the paper is too similar to CLIPN

Author response:
**A1:** Thanks for your contructive comments. In accordance with your suggestion, we conduct comparison experiments with NNGuide and ASH to enrich our analysis. The results on ImageNet-1k are as follows:
|                                | Average |       |
|--------------------------------|---------|-------|
| Method                         | PFR95   | AUROC |
| NNGuide(k=1)                   | 93.02   | 80.18 |
| NNGuide(k=10)                  | 94.46   | 78.96 |
| NNGuide(k=100)                 | 96.38   | 75.52 |
| ASH-B(@90) |  91.76   | 66.52 |
| ASH-B(@65) | 85.05   | 68.17 |
| ASH-P(@90) | 83.10  | 69.78 |
| ASH-P(@65) | 65.11  | 79.85 |
| ASH-S(@90) | 78.10  | 73.20 |
| ASH-S(@65) | 65.24  | 79.25 |
| LSN        | 30.22   | 92.96 |

Our experimental results show that LSN outperforms these post-hoc methods on raw CLIP model. We believe the reason for the poor performance of these pos-hoc methods is that the **the difference in training data**. Many pos-hoc methods are designed on ImageNet pre-trained networks, where **only ID datas are used during training**. In contrast, when training CLIP, **both ID datas and OOD datas are used**. The difference in training data leads to different activation of OOD data.  Another reason is that the pos-hoc method relies heavily on the choice of hyperparameters. The hyperparameters need to be re-selected on different models.

CLIPN also employes CLIP to perform OOD detection, coming up with ideas similar to ours. We compare the best results of CLIPN and LSN on ImageNet-1k with CLIP ViT-B-16 as follows:

|        | iNaturalist |           | SUN       |           | Places    |           | Textures  |           | Average   |           |
|--------|-------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
| Method | PFR95       | AUROC     | PFR95     | AUROC     | PFR95     | AUROC     | PFR95     | AUROC     | PFR95     | AUROC     |
| CLIPN  | 23.94       | 95.27     | **26.17** | 93.93     | **33.45** | **92.28** | 40.83     | **90.93** | 31.10     | **93.10** |
| LSN    | **21.56**   | **95.83** | 26.32     | **94.35** | 34.48     | 91.25     | **38.54** | 90.42     | **30.22** | 92.96     |

- Experimental results show that LSN achieves comparable performance with CLIPN. 
- We have to point out that **this is not a fair comparison**. This is because, to endow OOD detector with the ability to say "no", CLIPN leverages a large-scale dataset [1] (containing 3.3 million image-text pairs) to finetune the text encoder.
- These results highlight the contribution of the proposed approach to learn negative prompts. Specifically, LSN only needs to use few shot samples to learn negative prompts, which can be done in a very short time.

Inspired by your constructive comments, we summarize the differences between LSN and CLIPN as follows.

- LSN only requires learning negative prompts at a small computational cost without training "no" text encoder and achieves performance comparable with CLIPN.
- LSN points out that the diversity of negative prompts is the key to success, and the use of class-specific negative prompts leads to a reduction in FPR95 from 81.27% to 21.94% compared to the use of class-shared negative prompts, please see Table 5.
- LSN focuses on prompt learning and points out how negative prompt learning differs from positive prompt learning, while CLIPN focuses more on learning "no" text encoder.

We will add the above results and discussion into our revision to highlight the difference bewteen our work and the mentioned work. Thanks again for mentioning these outstanding papers.

Reviewer comment (Evaluation):
The performance improvement is very marginal

Author response:
**A3:** As mentioned above, we get the comparable performance at a much smaller cost compared to CLIPN. And compared to standard prompt learning CoOp and CoCoOp, LSN's FPR95 on ImageNet1k decreased by 6.86% and 8.41%, respectively.