Paper: Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (ICLR-2024)

Reviewer comment (Novelty):
The key components of proposed method (e.g., feature enhancer and cross-modality decoder) are not new. The authors should thoroughly discuss the relation and difference of between feature enhancer in GLIP [1] and the counterpart in this work, and conduct the same analysis for cross-modality (with X-Decoder [2]).

Author response:
Thank you for your insightful questions. In our revised paper, we have included a detailed discussion in Sec. M comparing our method with existing approaches, specifically GLIP and X-Decoder. 

In summary, our feature enhancer is similar with GLIP but more reasonable in our pure Transformer architectures. X-Decoder has no similar modules like our feature enhancer. Our cross-modality decoder, especially the text cross-attention, is unique from the designs in GLIP and X-Decoder.

Below are detailed comparisons:

- **Comparison with GLIP:**
    - **Feature Enhancer:** Our feature enhancer, though similar to GLIP's, is more aligned with our pure Transformer architecture. While GLIP uses DyHead for enhancing visual features, our method employs a Deformable Transformer for image encoder features, ensuring a more consistent architecture across different modalities.
    - **Cross-Modality Decoder:** Unlike GLIP, which uses the same head as ATSS/RetinaNet, our DETR-like model uniquely incorporates a cross-modality decoder. This decoder leverages the Transformer's capability to attend to features from both text and image modalities, a distinction we've validated through our ablation studies.

- **Comparison with X-Decoder:**
    - **Feature Fusion:** X-Decoder utilizes a standard image encoder and object decoder for different tasks, without integrating visual and text features during the encoding phase. In contrast, our model employs a feature enhancer for fusing features from both modalities.
    - **Queries in Decoder:** Our model's queries in the decoder aggregate features from both image and text, unlike X-Decoder's approach where the interactions are limited to queries and image features only.

These comparisons highlight the unique aspects of our method in the context of existing models. We have cited both GLIP and X-Decoder in our revised paper and dedicated a new paragraph to further elaborate on these comparisons.

Thank you again for your valuable questions, which have enriched our paper's discussion section.

Reviewer comment (Experiments):
Though utilizing better detector and better pretrained models, this work does not lead to better performance than previous work, e.g., DetCLIP-V2 [3] (CVPR 23), which obtains much better performance on LVIS benchmark and ODinW benchmark than Grounding DINO.

Author response:
Thanks for the valuable questions.

**ODinW Benchmark Performance**

It's important to note that **DetCLIP-v2 did not report zero-shot performance on the ODinW benchmark** in their paper, and we could not find their results on the ODinW leaderboard. Without access to their models, code, and training data, it's not feasible for us to independently evaluate their zero-shot performance. However, in terms of fine-tuning performance, **our Grounding DINO Tiny model outperforms DetCLIP-v2 Tiny (Grounding DINO Tiny: 70.0 vs. DetCLIP-v2 Tiny: 68.0) with much less data (Grounding DINO Tiny 5.2M vs. DetCLIP-v2 Tiny: 16.2M data)**. For this comparison, we used the ODinW-13 metric, a subset of the standard ODinW-35 metric, to align with DetCLIP-v2's benchmarking approach.

**LVIS Benchmark Performance**

While DetCLIP-v2 demonstrates better zero-shot performance on the LVIS benchmark, they utilize a substantially larger dataset (\~16.2M data) compared to Grounding DINO (\~5.2M data). This difference in data volume is a significant factor in performance. 

To illustrate the potential of our model under similar conditions, we conducted oracle experiments. We pseudo-labeled the ImageNet dataset using a pre-trained Detic model and filtered out LVIS-related images for training, creating a new pseudo-labeled dataset named IN22K-LVIS-1M. It contains about 1M pseudo-labeled images for training. Note that our model under the setting may be not a real zero-shot setting, as our pseudo labeler Detic is trained with LVIS data. The results from these experiments suggest that Grounding DINO can achieve promising LVIS performance, even without direct LVIS training data. A similarly distributed dataset can help the model to generalize well.

Thank you again for raising these points, which have significantly contributed to our paper's thoroughness. We have incorporated these new experiments and findings into our revised paper at Sec. N.

| Model            | Pre-train Data      | LVIS Zeroshot AP(r/c/f)      | LVIS Finetune AP(r/c/f)        |
|------------------|---------------------|-----------------------------|-------------------------------|
| DetCLIPv2-T      | OG + CC15M          | 40.4 (36.0 / 41.7 / 40.0)   | 50.7 (44.3 / 52.4 / 50.3)     |
| Grounding DINO T | OG+IN22K-LVIS-1M    | 40.6 (38.5 / 41.1 / 40.4)   | 54.5 (47.3 / 53.9 / 56.1)     |

Table R.3. Oracle experiments on LVIS.

Reviewer comment (Novelty):
Novelty issues. Despite the effectiveness of the proposed architectures, the composed modules are not innovative enough as their own. For example, the major component that makes a difference should be the DINO architecture that has already set a record on various detection tasks. The query selection is also similar to that of the top-K operators in Efficient DETR/DINO. Although Grounding DINO extends its tasks to referring expression comprehension, it is only comparable to its precedent studies like GLIP and does not introduce specific modules to address the shortcomings on REC. Nevertheless, I would also admit that these novelty issues are not critical as each of them is not trivial in the task.

Author response:
We appreciate your acknowledgment of the effectiveness of our proposed architecture. While it is true that the major components, such as the DINO architecture and query selection, resemble existing methods like those in Efficient DETR/DINO, our work uniquely integrates these elements. Specifically, we are the first to adapt DETR-like models to grounded training for open-set detection tasks, thereby achieving superior performance over traditional convolution-based open-set detectors like GLIP.

Our approach may not introduce specific modules for addressing shortcomings in Referring Expression Comprehension (REC), but its strength lies in the effective amalgamation of existing designs. This synthesis itself, we believe, is a meaningful contribution to the field, demonstrating that a well-orchestrated combination of established techniques can yield significant advancements in performance.

Reviewer comment (Evaluation):
I have some concerns regarding its significance and effectiveness in true open-set scenarios. Despite the effectiveness of the zero-shot performance on MS-COCO and ODinW datasets, the performance seems to rely on its pertaining data heavily. As also revealed by the author, Grounding DINO transfers better on common classes, classes that are more likely included in the pertaining data in some format, yet performs worse on rare classes in LVIS. The zero-shot performance on the RefCOCO dataset is significantly lower than that after including RefCOCO in its pertaining data.

Author response:
Thank you for your insightful and professional comments regarding our model's effectiveness in true open-set scenarios. We acknowledge that while our model demonstrates strong zero-shot performance on the MS-COCO and ODinW datasets, its reliance on common object classes is evident. These common objects, frequently used in various scenarios, underscore the practical utility of our model in many real-world applications.

Regarding the performance on ODinW, which encompasses 35 datasets across diverse domains, our model shows commendable versatility. Some of these datasets feature uncommon objects, further demonstrating the model's capability in more varied open-set environments. 

Moreover, we recognize the challenges in evaluating open-set models, a complexity mirrored in fields like image generation and Large Language Models (LLMs). A more comprehensive evaluation set might indeed be necessary for a thorough comparison of open-set models.


We do agree that Grounding DINO is less effective in Referring Expression Comprehension (REC) tasks, primarily because it wasn't specifically designed for REC.  However, it shows promise in REC tasks when fine-tuned or pre-trained with REC data, highlighting the critical role of data diversity in open-set models. To address this, we plan to continue enriching our model with a broader range of data to cover more diverse scenarios.

Additionally, our model demonstrates strong few-shot learning capabilities in the ODinW benchmark, significantly outperforming previous works. This efficiency in adapting to new scenarios with minimal data is a key advantage, offering lower costs and greater flexibility compared to earlier models.

In summary, while our model may not be perfect in every open-set scenario, it stands as a robust option for real-world applications, especially given its excellent few-shot performance and adaptability to diverse situations.

Reviewer comment (Evaluation):
A closer look into Table 4 would also reveal that the performance on ODinW is actually on par with GLIPv2 if given the same pertaining data. This would also raise a concern about its true ability in true open-set scenarios.

Author response:
Thank you for your valuable questions regarding the performance comparison in Table 4.

Regarding the comparison with GLIPv2, it's true that Grounding DINO and GLIPv2-T show similar $AP_{average}$. However, a key distinction lies in the $AP_{median}$, where Grounding DINO significantly outperforms GLIPv2-T (11.9 vs 8.9). This suggests that while GLIPv2 may exhibit larger performance variance across different datasets, Grounding DINO maintains a more consistent performance level. This consistency is particularly relevant in true open-set scenarios, where variance can greatly impact overall effectiveness.

Additionally, it's important to note the complexity and size differences between the models. GLIPv2 incorporates advanced techniques like masked text training and cross-instance contrastive learning, making it more complex than our Grounding DINO model. Moreover, our model is more compact (172M parameters) compared to GLIPv2 (232M parameters). These factors combined—performance consistency, model complexity, and size—should address concerns about our model's capability in true open-set scenarios.

We have included this analysis in the discussion paragraph of Sec. 4.3 of our revised paper. Thanks again for your questions, which help to make our paper clear.

Reviewer comment (Evaluation):
The ablations in table 6 and 7 do not show strong contribution of individual modeling choices except tight fusion. The authors dont explain this

Author response:
Thank you for raising these important points regarding the ablations in Tables 6 and 7.

Regarding Table 7, our primary aim was to demonstrate an efficient approach for Grounding DINO training. As detailed in Section 4.5, we illustrate that freezing a pre-trained DINO and tuning additional parameters introduced in Grounding DINO can achieve faster model convergence and comparable performance to full-parameter pretrained checkpoints.

Addressing your concerns about Table 6, our data shows that encoder fusion (tight fusion) significantly improves model performance on both COCO and LVIS datasets. The results from comparing model #1 with the baseline model #0 validate this observation. Other techniques, such as language-guided query selection, text cross-attention, and sub-sentence text prompt, also contribute positively to the LVIS performance, yielding significant gains of +3.0 AP, +1.8 AP, and +0.5 AP, respectively. Additionally, these methods enhance the COCO zero-shot performance, further underscoring their effectiveness.

However, we observed that language-guided query selection and sub-sentence text prompt had minimal impact on the COCO fine-tune performance. This outcome is reasonable, given that these methods do not alter model parameters or add computational burdens. Text cross-attention, while introducing fewer parameters than encoder fusion, showed less performance improvement compared to encoder fusion (+0.6 vs. +0.8). This finding suggests that fine-tuning performance is predominantly influenced by the model's parameters, indicating that scaling models is a promising direction for enhancing performance.

We have included a detailed discussion of these points in Section 4.4 of our revised paper. Thank you again for your valuable questions, which have helped to improve our paper.