Paper: RegionSpot: Unleashing the Power of Frozen Foundation Models for Open-World Region Understanding (ICLR-2024)

Reviewer comment (Experiments):
The claim of 6.5% and 14.8% improvements over GLIP makes no sense. The comparison is not fair. The paper only compares with GLIP-Tiny, which is way smaller than the proposed RegionSpot. GLIP-Tiny uses Swin-Tiny which has 29M parameters, while RegionSpot-BB has at least 160M parameters. According to the GLIP paper, GLIP-L achieves 26.9 AP on LVIS, which is better than the best RegionSpot-BL's 23.7 AP.

Author response:
Thanks for pointing out this. To address this question, we have now provided extra comparison under GLIP protocol. This table below shows that our method is superior over GLIP-L despite using less training data (3M vs 27M) and less learnable parameters (35M vs. 289M). Both conditions are related to the training cost directly, which means our training is significantly more efficient.

Table 1: Comparison under GLIP protocol.
| Method | Training Data | Data Size | Fully finetune | Training Time | MiniValAPr | MiniValAPall | Val APr | Val APall |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GLIP-L | FourODs,GoldG,Cap24M | 27M | Yes | 120K | 28.2 | 37.3 | 17.1 | 26.9 |
| RegionSpot-BL | O365,OI,V3DET | 3M | No | 0.2k | 33.2 | 36.9 | 30.2 | 29.8 |

Reviewer comment (Experiments):
Evaluation is limited. The paper only tests on open-world LVIS recognition. It would be more convincing to do a more comprehensive evaluation.

Author response:
Thanks. For more comprehensive evaluation, we also perform experiments under the VILD protocol. The results, presented in Table 2 (Response to all reviewers), show that our method achieves a 19.7% APr, surpassing F-VLM 18.6% APr by 1.1% APr. Notably, F-VLM is also based on a Frozen Foundation model. This comparison demonstrates our model capability to effectively leverage pre-trained knowledge.

Reviewer comment (Writing):
Please check the reference carefully. For example, " However, the use of ROIAlign (Ren et al., 2015) for region feature extraction...". The ROIAlign is not proposed in Ren et al., 2015.

Author response:
Thanks, we will fix all in the revision.

Reviewer comment (Experiments):
The paper only experiments with different CLIP models, i.e., CLIP-base and large. What about using larger SAM models, e.g., RegionSpot-LL?

Author response:
Thanks for this great suggestion. To address this, we have conducted additional experiments with varying SAM model backbones. Our findings are summarized as follows:
1. Role of SAM in Position-Aware Knowledge and Mask Generation: Our results indicate that the use of larger SAM models (e.g., SAM-L) improves mask AP due to the higher quality of mask generation. However, for box AP, there is even some slight drop in the improvement. This is because the SAM mask token primarily contributes position-aware knowledge, which is already sufficiently captured by either ViT-B or ViT-L.
2. Choice of SAM Model: Given our focus on region recognition, we opted for SAM-B, balancing performance and computational efficiency.

| SAM-Model | Box AP_rare | Mask AP_rare |
| --- | --- | --- |
| ViT-B | 24.9 | 22.8 |
| ViT-L | 24.7 | 23.6 |

Reviewer comment (Reproducibility):
More details can be reported, e.g., the number of new parameters.

Author response:
Great point! As suggested, we provide the number of learnable parameters as following:
|  | Leanble Parameter(M) |
| --- | --- |
| RegionSpot-BB | 22 |
| RegionSpot-BL | 35 |

Reviewer comment (Experiments):
There are also many alternatives to perform the so-called region understanding task with SAM and CLIP. For example, SAM can directly extract region proposals and CLIP (or RegionCLIP) can predict the category tags of them. We can also add some projection layers in this way, freeze most parameters and finetune a small part of parameters for efficient training. The author should perform experiments to compare with baselines like this. Otherwise, the effect of the position-aware localization cannot be seen.

Author response:
Thanks for such detailed suggestions. In our submission, we already provided two variants of this suggested baseline by feeding ground-truth proposals in the first two rows of Table 1. They can be considered as the upper bounds. Following the suggestion, we further tested with SAM region proposals along with CLIP. The table below shows simply streamlining the off-the-shelf foundation models is still largely inferior than our method. We have added this new baseline in the revised version.

|  | Proposals | AP_rare |
| --- | --- | --- |
| SAM + CLIP | SAM | 8.6 |
| SAM + CLIP w/ projector | SAM | 9.6 |
| RegionSpot-BB | SAM | 10.6 |
| RegionSpot-BB | GLIP | 20.0 |

Reviewer comment (Experiments):
The experiment in Table 1 is also unfair. RegionCLIP simply performs image-text pre-training, without object detection finetuning. However, RegionSpot performs something about detection training. Therefore, the author should finetune RegionCLIP on the same detection datasets for a fair comparison.

Author response:
Thank you for the excellent suggestion. Given that RegionCLIP requires extensive computational resources and time for re-training, we conducted experiments following the ViLD protocol for a fair comparison. The results, presented in Table 2 (Response to all reviewers), demonstrate that our method achieves a 19.7%, outperforming RegionCLIP 17.1% by 2.6% APr. This is noteworthy as we achieve these results without refining the backbone using large-scale image-text pairs.

Reviewer comment (Experiments):
The author should also compare with some more recent methods, like GLIP v2, Grounding DINO and so on.

Author response:
Thank you for the valuable suggestion. As GLIP v2 does not operate in a zero-shot manner on LVIS, we have included the more robust Grounding DINO large model in Table 1 (Response to all reviewers) for a direct comparison. Please refer to this for detailed insights.

Reviewer comment (Experiments):
Limited Baseline Comparisons. Despite significant prior work in OVOD [1,2,3], authors primarily only compare with RegionCLIP and GLIP. In reality, prior work [1] significantly outperforms RegionSpot.

Author response:
Thanks for pointing this out. We note that comparing OWL-V2 and RegionSpot directly is not fair in terms of the usage of data scale: WebBI dataset at a size of 2B images vs. 3M images. This big difference in data size makes us hard to draw meaningful insights.Following additional suggestions, we have included more extensive comparison. We add  GLIP-L and Grounding DINO-L in Table 1 ( Response to all reviewers) to verify the efficient training and good performance.

Reviewer comment (Experiments):
Unfair Comparisons. Since authors show that pre-training data scale significantly contributes to model performance, comparing RegionSpot, a technically very similar method trained on much less data is unfair. Instead, it would make more sense to evaluate RegionSpot trained on only CC3M.

Author response:
Our training objectives differ from those of RegionCLIP, which requires large-scale image-text pairs to retrain a region-level backbone. Our goal is to leverage publicly available object detection datasets to train new layers that enable existing foundational models to perform region-level understanding. For fair comparison, we have conducted experiments following the ViLD protocal Please refer to our comprehensive response to all reviewers for detailed results.

Reviewer comment (Experiments):
Limited by Quality of Boxes. RegionSpot is always limited by the bounding box proposals provided as input to the system. As authors show in Table 1, the type of proposals has a significant impact on model performance. It would be interesting to evaluate how the impact of using proposals from one of the more performant open-vocabulary models than GLIP (e.g. GroundingDINO).

Author response:
Thanks for this insight. We have now evaluated the effect of input proposals on performance. As GroundingDINO comes with no open-source for LVIS evaluation, we instead utilize GLIP-L as the proposal generator. Our results underscore two key findings:
1. There is a significant improvement when the quality of prompt boxes is improved.
2. Despite GLIP-L generating high-quality boxes, it faces challenges in region recognition. Notably, RegionSpot achieves superior performance, exceeding GLIP-L by 13.1 in AP_r metric.
|  | Proposal | AP_rare |
| --- | --- | --- |
| GLIP-L | GLIP-L | 17.1 |
| RegionSpot-BL | GLIP-T | 24.9 |
| RegionSpot-BL | GLIP-L | 30.2 |

Reviewer comment (Evaluation):
Can this method be most improved by better classification (e.g. CLIP) or localization (e.g. SAM)? What are the typical error modes?

Author response:
Thanks for the great question. We would do more evaluation on their respective effects once more options become available. From our (not exhaustive) observation, it is hard to draw the typical error modes though.

Reviewer comment (Novelty):
**Insufficient novelty and contribution**: The newly proposed **RegionSpot** framework lacks justification for its design. The pipeline of fine-tuning a lightweight module while frozen SAM and CLIP models seems natural and basic. Additionally, only conducting experiments on object detection tasks is not convincing.

Author response:
> The newly proposed RegionSpot framework lacks justification for its design.  Although the authors claim that "This implementation effectively facilitates the fusion of semantic and location information in a manner that is amenable to learning and yields substantive efficacy.", they provide no experimental results. Also, the motivation is not clear. For example, why do the authors serve the localization feature as the role of query? what if the ViL feature assumes as the query?

1. Our motivation is to integrate position-aware localization knowledge from a localization foundation model (e.g., SAM) with semantic information extracted from a ViL model in a frozen to achieve the region understanding.  As Reviewer Tjjg  said, introducing vision foundation models into new tasks is an important thing.
2. In our framework, the mask tokens from SAM, which act as queries, inherently localize objects but do not convey semantic information. Our method thus uses these tokens as query to find the corresponding semantic details from the ViL feature map, enhancing the semantic understanding at a regional level. Using the ViL feature as a query, on the other hand, would be illogical for our purpose, as our objective is to augment already localized specific regions with semantic information from the ViL features.

> Only conducting experiments on object detection tasks is not convening. 

As we focus on region recognition, object detection is the most important and most suitable problem for evaluation in computer vision, per our knowledge. Also, we have evaluated the Instance Segmentation task with positive comparisons. We are open to evaluate more suitable problems if suggested kindly.

Reviewer comment (Experiments):
**Insufficient results for experiments**: Although the authors claim that "This implementation effectively facilitates the fusion of semantic and location information in a manner that is amenable to learning and yields substantive efficacy.", they provide no experimental results. Also, the motivation is not clear. For example, why do the authors serve the localization feature as the role of `query`? What if the ViL feature assumes as the `query`?

Author response:
> The newly proposed RegionSpot framework lacks justification for its design.  Although the authors claim that "This implementation effectively facilitates the fusion of semantic and location information in a manner that is amenable to learning and yields substantive efficacy.", they provide no experimental results. Also, the motivation is not clear. For example, why do the authors serve the localization feature as the role of query? what if the ViL feature assumes as the query?

1. Our motivation is to integrate position-aware localization knowledge from a localization foundation model (e.g., SAM) with semantic information extracted from a ViL model in a frozen to achieve the region understanding.  As Reviewer Tjjg  said, introducing vision foundation models into new tasks is an important thing.
2. In our framework, the mask tokens from SAM, which act as queries, inherently localize objects but do not convey semantic information. Our method thus uses these tokens as query to find the corresponding semantic details from the ViL feature map, enhancing the semantic understanding at a regional level. Using the ViL feature as a query, on the other hand, would be illogical for our purpose, as our objective is to augment already localized specific regions with semantic information from the ViL features.

Reviewer comment (Novelty):
**Excessive statement**: The authors claim that "Our model's flexible architecture allowed us to seamlessly replace one-hot labels with class name strings.". This may be overemphasizing their contribution.

Author response:
> Overemphasizing the contribution about "Our model's flexible architecture allowed us to seamlessly replace one-hot labels with class name strings."

We would tune done and refine this description. To further clarify: Using text embedding of class name strings instead of one-hot labels is the enabler for good zero-shot performance as exemplified with recent foundation models like CLIP and ALIGN (both function at the image level). In the domain of open-world object recognition, this approach is similarly employed to attain zero-shot capabilities, such as ViLD and GLIP. We have simply adhered to this established method.

Reviewer comment (Writing):
**Grammar and minor errors**: In section **Position-aware tokens selection in SAM**, a grammatical error is present in the sentence: "Surprisingly, although it can outperform GLIP (i.e., 17.2 vs. 18.6)."

Author response:
> Grammar mistakes

Thanks, we will fix all in the revision.

Reviewer comment (Writing):
**Grammar and minor errors**: In section **Prompt enginerring**, there is a discrepancy in the reported increase: "an increase of 1.4 AP" — should this be 1.6 AP? Additionally, clarification is needed regarding whether the method in the last line of Table 5 indeed represents the baseline with both prompts.

Author response:
> Ablation Study of Prompt engineering - "clarification is needed regarding whether the method in the last line of Table 5 indeed represents the baseline with both prompts."

Yes, we will clarify in the revision.

Reviewer comment (Novelty):
My major concern is the contributions of combining SAM and CLIP for object detection tasks.

Author response:
1. Our motivation is to integrate position-aware localization knowledge from a localization foundation model (e.g., SAM) with semantic information extracted from a ViL model in a frozen to achieve the region understanding.  As Reviewer Tjjg  said, introducing vision foundation models into new tasks is an important thing.
2. In our framework, the mask tokens from SAM, which act as queries, inherently localize objects but do not convey semantic information. Our method thus uses these tokens as query to find the corresponding semantic details from the ViL feature map, enhancing the semantic understanding at a regional level. Using the ViL feature as a query, on the other hand, would be illogical for our purpose, as our objective is to augment already localized specific regions with semantic information from the ViL features.