Paper: LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors (ICLR-2024)

Reviewer comment (Novelty):
The use of region-level prompting for open-vocabulary detection is previously proposed in CVPR 2023 paper "CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching". The comparison (both at a idea level and a numeric comparison) with CORA work seems necessary regarding the technical similarity.

Author response:
In contrast to CORA's approach, our visual prompts integrate background information from images. Table 9 of our paper shows that incorporating background information leads to performance improvements, illustrating the effectiveness of our method. We believe that this design offers a more comprehensive and contextual understanding of the image, aligning more closely with the visual prompt concept. Here are the experimental results:

|  CCP  |   mAP_novel  |   mAP_base  |   
|:--------:|:---------:|:---------:|
|    w/o Background Information               	| 33.4      	| 51.8  |
| DVDet          	| **34.6**      	| **52.8**  |

This distinction in prompt design underlines the originality of our approach and its effectiveness in enhancing detection performance.

Reviewer comment (Experiments):
It is not clear what the optimal number of iterations is for the dictionary update. Is "the number of high-frequency descriptors" in Table 7 the number of iterations?

Author response:
We would clarify that the update frequency of the descriptors dictionary is not equal to "the number of high-frequency descriptors". We conduct new experiments to assess its impact on performance. As the table below shows, DVDet maintains stable performance when the dictionary update frequency remains within a certain range

|  Update Frequency of Dictionary |   0  |   100  |   200  |   400  |
|:--------:|:---------:|:---------:|:---------:|:---------:|
|    mAP_novel               	| 33.2      	| **34.8**  | 34.6      	| 34.6  |
| mAP_base          	| 51.9     	| **53.2**  | 52.8      	| 52.6  |

Reviewer comment (Experiments):
There are non-negligible number of hyper parameters (number of descriptors, number of update iterations, thresholds).

Author response:
Actually, our method focuses on three key parameters: the length 'm' of the enlarged proposals, the update frequency 'N' of descriptors, and the number 'TopN' of high-frequency descriptors. However, our experimental data indicate that the algorithm's performance is not highly sensitive to these parameters. This is supported by the results shown in Table 6 of our paper regarding the 'TopN' parameter. In response to Q2, we have analyzed the impact of the Descriptor Update Frequency 'N'. Additionally, to further demonstrate the importance of the parameter 'm', which determines the width and height of the proposal, we present the following experimental results:

|  the values m and n  |   0  |   20  |   30  |   40  | 
|:--------:|:---------:|:---------:|:---------:|:---------:|
|    mAP_novel     | 33.4    | 34.6  | 34.8  | 34.4  |
|  mAP_base  |  51.8      	|  52.8  |  52.9  | 52.4  |

Moving forward, we plan to implement learnable parameters and structures, aiming to reduce the necessity for manual hyperparameter tuning.

Reviewer comment (Evaluation):
The gain in the base vs novel categories are similar (e.g., + 2.7 novel AP and +2.2 base AP in Table 1). This leads to a question whether the DVDet method is beneficial in standard detection instead of open-vocabulary detection. That is, the proposed method benefits from augmenting the vocabulary description (text embeddings) in general, not being specifically helpful for the novel categories. What is the motivation of evaluating on open-vocabulary detection? How would DVDet perform in the standard detection setting?

Author response:
Thank you for highlighting this important aspect. While our method does not show more pronounced improvements in novel categories compared to base categories, it achieves consistent enhancements in both. As depicted in Figure 1 of our paper, unlike VLM-based classification models [1], current open-vocabulary detectors have difficulty aligning visual features with category descriptions. Our ablation studies reveal that descriptors do not boost the pretrained model during inference in detection tasks as effectively as in classification tasks, sometimes even leading to performance decline. Thus, we focus on aligning both base categories and descriptors during training. This approach yields improvements in novel categories, akin to the zero-shot performance seen in classification models, affirming our method's ability in enhancing open-set detection. As shown in our paper's Figure 5 and Figure 8, extensive visualizations further illustrate how descriptors progressively refine detection results in new categories.

[1] Menon S, Vondrick C. Visual classification via description from large language models[J]. ICLR2023.

Reviewer comment (Experiments):
Please compare with CORA (CVPR 2023) which also uses visual region prompting for open-vocabulary detection.

Author response:
In contrast to CORA's approach, our visual prompts integrate background information from images. Table 9 of our paper shows that incorporating background information leads to performance improvements, illustrating the effectiveness of our method. We believe that this design offers a more comprehensive and contextual understanding of the image, aligning more closely with the visual prompt concept. Here are the experimental results:

|  CCP  |   mAP_novel  |   mAP_base  |   
|:--------:|:---------:|:---------:|
|    w/o Background Information               	| 33.4      	| 51.8  |
| DVDet          	| **34.6**      	| **52.8**  |

This distinction in prompt design underlines the originality of our approach and its effectiveness in enhancing detection performance.

Reviewer comment (Evaluation):
A comparison of computation costs with the baseline has not been conducted.

Author response:
* We appreciate the reviewer pointing out the need for experimental analysis on efficiency. In our study, the primary factor impacting training efficiency is the update of category descriptors. Notably, the update time for the descriptor dictionary accounts for only **20.16%** of the total training time when the update frequency is set at every 100 iterations in our implementation. This ratio demonstrates that our approach achieves an efficient balance, being comprehensive while remaining resource-effective.


* We further conducted experiments to evaluate this, as shown in the table below. The results indicate that within a certain range, reducing the update frequency of the dictionary allows DVDet to maintain stable performance.

|  Update Frequency of Dictionary |   0  |   100  |   200  |   400  |
|:--------:|:---------:|:---------:|:---------:|:---------:|
|    mAP_novel               	| 33.2      	| **34.8**  | 34.6      	| 34.6  |
| mAP_base          	| 51.9     	| **53.2**  | 52.8      	| 52.6  |

Reviewer comment (Reproducibility):
The prompt generated by LLM may possess an element of randomness, potentially making it challenging to reproduce.

Author response:
We are pleased that the reviewer has pointed out the impact of descriptor randomness on experimental performance. In fact, DVDet's interactive mechanism was designed with the generative model's randomness in mind. We have implemented two relevant mechanisms:

* Descriptor Update Mechanism: By retaining high-frequency descriptors and discarding low-frequency ones, we ensure the visual relevance of the descriptors.
* Descriptor Selection Mechanism: During the inference phase, we discard irrelevant descriptors, thereby enhancing stability.
* We further present the performance variability across five experimental runs to demonstrate the robustness of our algorithm.

|  LVIS  |   mAP_novel  |   mAP_all  |   
|:--------:|:---------:|:---------:|
|    VLDet      | 27.5  $\pm$ 0.2      	| 41.8 $\pm$ 0.3 |

Reviewer comment (Experiments):
No ablation experiment about the hyper-parameter $m$ of Conditional Context Prompt.

Author response:
* Firstly, in Table 9 of our paper, we provided ablation studies regarding background information, showing the experimental outcomes when parameter m is set to 0.

* We conducted further experiments about the parameter m. As the table below shows, DVDet's performance remains stable when m lies within a certain range:

|  the values m and n  |   0  |   20  |   30  |   40  | 
|:--------:|:---------:|:---------:|:---------:|:---------:|
|    mAP_novel     | 33.4    | 34.6  | 34.8  | 34.4  |
|  mAP_base  |  51.8      	|  52.8  |  52.9  | 52.4  |

Reviewer comment (Evaluation):
How about the extra computation overhead?

Author response:
* We appreciate the reviewer pointing out the need for experimental analysis on efficiency. In our study, the primary factor impacting training efficiency is the update of category descriptors. Notably, the update time for the descriptor dictionary accounts for only **20.16%** of the total training time when the update frequency is set at every 100 iterations in our implementation. This ratio demonstrates that our approach achieves an efficient balance, being comprehensive while remaining resource-effective.


* We further conducted experiments to evaluate this, as shown in the table below. The results indicate that within a certain range, reducing the update frequency of the dictionary allows DVDet to maintain stable performance.

|  Update Frequency of Dictionary |   0  |   100  |   200  |   400  |
|:--------:|:---------:|:---------:|:---------:|:---------:|
|    mAP_novel               	| 33.2      	| **34.8**  | 34.6      	| 34.6  |
| mAP_base          	| 51.9     	| **53.2**  | 52.8      	| 52.6  |

Reviewer comment (Experiments):
LLM may generate different prompts. Does it significantly affect the performance?

Author response:
We are pleased that the reviewer has pointed out the impact of descriptor randomness on experimental performance. In fact, DVDet's interactive mechanism was designed with the generative model's randomness in mind. We have implemented two relevant mechanisms:

* Descriptor Update Mechanism: By retaining high-frequency descriptors and discarding low-frequency ones, we ensure the visual relevance of the descriptors.
* Descriptor Selection Mechanism: During the inference phase, we discard irrelevant descriptors, thereby enhancing stability.
* We further present the performance variability across five experimental runs to demonstrate the robustness of our algorithm.

|  LVIS  |   mAP_novel  |   mAP_all  |   
|:--------:|:---------:|:---------:|
|    VLDet      | 27.5  $\pm$ 0.2      	| 41.8 $\pm$ 0.3 |

Reviewer comment (Experiments):
How about different $m$ in Conditional Context Prompt?

Author response:
* Firstly, in Table 9 of our paper, we provided ablation studies regarding background information, showing the experimental outcomes when parameter m is set to 0.

* We conducted further experiments about the parameter m. As the table below shows, DVDet's performance remains stable when m lies within a certain range:

|  the values m and n  |   0  |   20  |   30  |   40  | 
|:--------:|:---------:|:---------:|:---------:|:---------:|
|    mAP_novel     | 33.4    | 34.6  | 34.8  | 34.4  |
|  mAP_base  |  51.8      	|  52.8  |  52.9  | 52.4  |

Reviewer comment (Novelty):
The idea of using LLM to generate more textual description has been explored in recent/concurrent work [2]. It'd be great to see a comparison with the existing ideas. Similarly, the idea of context regional prompt has been explored in recent/concurrent work [1], although the exact instantiation may differ. It'd be helpful to tease apart the contribution of context (which makes the crop more image-like) vs prompting (which adapts the features for detection use cases) through an ablation, and compare/discuss the similarity/differences with existing work. I understand the references listed are very recent/concurrent works, but I think it'd still be valuable to have some comparison with them for scientific understanding.

Author response:
> W1: Discussion with Concurrent Work [1] and [2].

As the reviewer noted, our work mainly involves the generation of category descriptors and the regional visual prompt. We appreciate the reviewer pointing out the need to compare our approach with the latest methods CORA [1] and MCVOD [2], and we provide a comparison with each below.

Firstly, regarding the generation of descriptors, our approach differs significantly from MCVOD [2] in the way of interacting with LLMs. Specially, we leverage the interactive capabilities of LLMs to dynamically generate visually relevant descriptors for each category, rather than treating the process as a one-time interaction. As shown in Table 5 of our paper, introducing this interaction mechanism generally leads to a performance gain of around 1.5%. The experimental results are as follows:

|  Interaction Strategy  |   mAP_novel  |   mAP_base  |   
|:--------:|:---------:|:---------:|
|    Static               	| 33.2      	| 51.9  |
| Interactive (DVDet)          	| **34.6**      	| **52.8**  |

Secondly, regarding visual prompts, our design differs from the prompts in CORA [1] by integrating background information from images. As Table 9 of our paper shows, we observe further performance improvements when the background information is incorporated. As the reviewer pointed out, prompts without background information can be considered tailored for detection tasks (**prompting**). Incorporating background information further merges contextual information (**context**), making it more consistent with the image. The experimental results are as follows:

|  CCP  |   mAP_novel  |   mAP_base  |   
|:--------:|:---------:|:---------:|
|    w/o Background Information               	| 33.4      	| 51.8  |
| DVDet          	| **34.6**      	| **52.8**  |

We will add the discussion with the concurrent methods[1, 2] in our updated manuscript.

[1] Wu X, Zhu F, Zhao R, et al. CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching[C]. CVPR 2023.

[2] Kaul P, Xie W, Zisserman A. Multi-Modal Classifiers for Open-Vocabulary Object Detection [C]. ICML 2023

Reviewer comment (Experiments):
Table 3 (row 2) shows that adding fine-grained descriptors by itself without prompting hurts the performance of the model. This seems counter-intuitive to me. It'd be helpful to see if the same observation holds for other OVD models e.g. ViLD or Detic.

Author response:
> W3: Decreased Performance with Descriptors without Prompt Learning

Thank you very much for the constructive suggestion. The reason for this phenomenon, where adding fine-grained descriptors by itself without prompting hurts the performance of the model, lies in the disparity between the capabilities of VLMs and current open vocabulary detectors in cross-modal alignment. As shown in Figure 1 of our paper, the visual features extracted by VLMs maintain high consistency with the textual embeddings of descriptors. Therefore, in classification tasks [3], they can directly enhance performance in the inference stage without additional training. However, this is challenging to achieve in detection tasks. To make this conclusion more convincing, we have also compiled results on Detic and provided the following ablation study:

|  Detic  |   mAP_novel  |   mAP_base  |   
|:--------:|:---------:|:---------:|
|    w/o Prompt Learning               	| 25.5      	| 47.9  |
| Prompt Learning          	| **27.8**      	| **51.1**  |

We will update the experiments and discussions.

Reviewer comment (Experiments):
Table 4 shows transfer detection from COCO to PASCAL and LVIS. It's nice to see a clear boost there. I'd recommend trying out the LVIS-trained model on Objects365 instead since it's a more commonly used transfer detection setting by e.g. ViLD, DetPro, F-VLM.

Author response:
> W4: Transfer LVIS Detection Model to Objects365.

Thank you for your suggestion. We conducted the suggested experiments and the experimental results are presented below, where we conduct the experiments on representative DetPro as suggested:

|  Objects365  |   AP_s  |   AP_m  |   AP_l  |   
|:--------:|:---------:|:---------:|:---------:|
|    DetPro               	| 4.5      	| 11.5  | 18.6  |
| +DVDet         	| **5.1**      	| **12.3**  | **19.8**  |

Reviewer comment (Evaluation):
What's the variance of the proposed method on LVIS open-vocabulary benchmark over e.g. 3 or 5 independent runs?

Author response:
> Q: In response to the query about variance on the LVIS Open-Vocabulary Benchmark, we conducted three independent runs to assess the consistency of our method. The experiments show that the variance is very small, indicating the stability of our approach:

|  LVIS  |   mAP_novel  |   mAP_all  |   
|:--------:|:---------:|:---------:|
|    VLDet      | 27.5  $\pm$ 0.2      	| 41.8 $\pm$ 0.3 |

Reviewer comment (Experiments):
when LLM interacts with VLM, Could you give more examples of prompts to use and how the prompt may influence the final results?

Author response:
In this paper, we utilize three prompts to interact with LLMs to gather descriptors, 'Template N' only includes the category name, ‘Template H’ [1] includes high-frequency descriptors and the category name, and ‘Template C’ includes confusing categories and the category name. The performance using different prompts is as follows:

|  Interaction Strategy  |   mAP_novel  |   mAP_base  |   
|:--------:|:---------:|:---------:|
|    VLDet               	| 32.0      	| 50.6  |
|    +template N               	| 33.1      	| 51.2  |
|    +template C               	| 33.8      	| **52.3**  |
|    +template H               	| **34.0**      	| 52.2  |

Our experimental findings show that our designed templates, namely templates C and H, are more effective than the template N used in [1] for prompting LLMs. This results in the generation of effective descriptors and an overall improvement in performance, highlighting the efficacy of our method in descriptor generation.

[1] Menon S, Vondrick C. Visual classification via description from large language models[J]. ICLR2023.

Reviewer comment (Experiments):
when iterative updating of LLM descriptor, It seems to just give a general description of objects. Then what will happen if we just let LLM do some general description of objects without seeing the objects?

Author response:
* Firstly, in our method, the Large Language Models (LLMs) understand objects through their interaction with Vision Language Models (VLMs). So, during training, the LLMs don't see novel classes. The improvement in detecting these novel classes, which are unknown to LLMs, shows how effective our generated descriptors are.

* Secondly, we present experimental results where all classes were unknown to the LLMs. The results from Table 5 of our paper are as follows:

|  Interaction Strategy  |   mAP_novel  |   mAP_base  |   
|:--------:|:---------:|:---------:|
|    VLDet               	| 32.0      	| 50.6  |
|    Static               	| **33.2**      	| **51.9**  |


These results indicate that even without interacting with detection models, the descriptors from LLMs still improve the performance of these models.

Reviewer comment (Experiments):
if the regional prompt is not accurate, how will the performance look like when interacting with LLM?

Author response:
As the reviewer correctly points out, the accuracy of prompts can indeed affect the algorithm's performance. As shown in Table 3 of our paper, we present the performance of DVDet with Prompt Learning. To further illustrate the impact of prompt accuracy, we added a new experiment using a Random Prompt. This additional experiment clearly demonstrates that inaccurate, randomly selected prompts lead to decreased performance. In contrast, our original approach with prompt learning significantly improves detection accuracy.
 
|  Regional Prompt  |   mAP_novel  |   mAP_base  |   
|:--------:|:---------:|:---------:|
|    VLDet               	| 32.0      	| 50.6  |
|    +random prompt | 27.3      	| 45.2  |
|    +prompt learning | **33.1**      	| **51.2**  |

Reviewer comment (Evaluation):
The paper does not sufficiently address the analysis of descriptors produced by Large Language Models, which is crucial for understanding the quality and relevance of the generated descriptors in enhancing object detection.

Author response:
Thank you for highlighting the importance of descriptor analysis, which is indeed a central aspect of our work. Effectively generating descriptors through LLMs forms the core of our approach.
*  Firstly, we designed an interactive mechanism for refining descriptors through ongoing interactions with LLMs. In contrast to static descriptors, which are produced through a one-time interaction, our interactive approach updates existing descriptors based on their contributions in enhancing detection performance. This point is well demonstrated in the experiment table below:

|  Interaction Strategy  |   mAP_novel  |   mAP_base  |   
|:--------:|:---------:|:---------:|
|    Static               	| 33.2      	| 51.9  |
| Interactive          	| **34.6**      	| **52.8**  |

* Secondly, our semantic selection mechanism chooses visual-related descriptors, thereby avoiding the influence of descriptors that are not visible in the current sample. This design highlights the importance of the relevance of the descriptors as validated in the table below. 

|  TopN  |   0  |   1  |    3  |   5  |  10  |   15  | 
|:--------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
|    mAP_base               	| 50.2      	| 51.2  | 52.0    | **52.8**  |51.9      	| 51.7  |
| mAP_novel          	| 32.0      	| 32.4  |34.4      	| **34.6**  |33.0      	| 32.2  |

* Lastly, we provide extensive visualization in Figures 7 and 8 of our paper, which illustrate how the descriptors improve detection performance. These visualizations clearly demonstrate the effectiveness of our descriptor generation in enhancing object detection performance.

Reviewer comment (Novelty):
The demonstrated enhancements (~2% on average) in object detection performance, while positive, do not represent a substantial leap forward when considering the benchmarks established by current leading methods.

Author response:
We would like to share that a 2% improvement in performance metrics is quite acceptable in the current setting. For instance, methods like VLDet [1] which incorporates caption data also achieve only around 2% improvement compared to those without caption data. However, we would like to reiterate two key points: first, our method employs lightweight prompt learning, and second, it achieves consistent improvements across multiple detectors.

[1] Lin C, Sun P, Jiang Y, et al. Learning object-language alignments for open-vocabulary object detection[J]. ICLR, 2023.

Reviewer comment (Evaluation):
How does the descriptors generation process affects the training efficiency?

Author response:
We appreciate the reviewer pointing out the need for experimental analysis on efficiency. In our study, the primary factor impacting training efficiency is the update of category descriptors. Notably, the update time for the descriptor dictionary accounts for only **20.16%** of the total training time when the update frequency is set at every 100 iterations in our implementation. This ratio demonstrates that our approach achieves an efficient balance, being comprehensive while remaining resource-effective.

Reviewer comment (Evaluation):
Can the authors provide insights into any observed trade-offs between the complexity of the Descriptor-Enhanced Open Vocabulary Detection (DVDet) and its performance gains?

Author response:
Building on our previous discussion, we identified that the frequency of dictionary updates (i.e., the frequency of interactions with LLMs per number of iterations) is a key factor influencing training efficiency. We conducted experiments to evaluate this, as shown in the table below. The results indicate that within a certain range, reducing the update frequency of the dictionary allows DVDet to maintain stable performance.

|  Update Frequency of Dictionary |   0  |   100  |   200  |   400  |
|:--------:|:---------:|:---------:|:---------:|:---------:|
|    mAP_novel               	| 33.2      	| **34.8**  | 34.6      	| 34.6  |
| mAP_base          	| 51.9     	| **53.2**  | 52.8      	| 52.6  |

Reviewer comment (Evaluation):
How does the system perform under varying conditions, such as different object scales, occlusions, and lighting? Are there specific scenarios where the performance improvement is more pronounced?

Author response:
In Figure 7 of our paper, we demonstrate how fine-grained descriptors correct the model's detection results. Further visualizations of this process are provided in Figure 8 of our paper. These visualizations clearly show that our approach significantly improves detection, particularly in challenging scenarios. These scenarios include (1) detecting distant or occluded objects, and (2) distinguishing objects with small inter-class variations.