Paper: Detect Every Thing with Few Examples (ICLR-2024)

Reviewer comment (Novelty):
The idea is similar to OV-DETR since both uses the bypassing per-class inference procedure.

Author response:
**(W2, comparision with OV-DETR) "The idea is similar to OV-DETR since both uses the bypassing per-class inference procedure."**

Our idea is not just bypassing per-class inference but also training a binary classifier that can be used for all classes without any finetuning, and a novel propagation-based localization mechanism. On the contrary, OV-DETR [1] does not bypass the per-class inference procedure [2, 3].


The inefficiency of OV-DETR is also pointed out by CORA [2] as *“Conditional Matching in OV-DETR  also proposes to condition the queries on the text embedding for class-aware regression. But it suffers from repetitive per-class inference.”*  and Prompt-OVD [3] as *“However, the main limitation of OV-DETR is the need for multiple forward passes in the decoder due to its large number of object queries, which results in slow inference speeds that may not be suitable for real-world use cases."*, and *"OV-DETR needs linearly increasing number of object queries with respect to the number of classes.”*.


We updated Section 3.1 at Page 3 of the paper to clarify the procedure of bypassing per-class inference.

Reference: 

[1]: Open-vocabulary detr with conditional matching, ECCV 2022

[2]: CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching, CVPR 2023

[3]: Prompt-Guided Transformers for End-to-End Open-Vocabulary Object Detection, Arxiv 2023

Reviewer comment (Experiments):
I double that the proposed approach may not be a real open vocabulary setting, since the authors assume the novel classes of COCO can be accessed during the inference. This means the vocabulary is still limited and pre-defined. For example, the proposed model is hard to extend into zero-shot setting where the novel prototypes are not available.

Author response:
**(W3, open vocabulary setting) "I double that the proposed approach may not be a real open vocabulary setting, since the authors assume the novel classes of COCO can be accessed during the inference. This means the vocabulary is still limited and pre-defined. For example, the proposed model is hard to extend into zero-shot setting where the novel prototypes are not available."**

We agree with the reviewer that our proposed method is not an open-vocabulary detector. We revised the entire introduction and experiment section to clarify the  setting and comparison towards open-vocabulary detectors.

Our original intent is to use this comparison to indicate the significant improvement we achieved on few-shot object detection. Because the detection accuracy of existing few-shot methods fall behind that of open-vocabulary detectors, especially on challenging datasets such as LVIS (the open-vocabulary SoTA on LVIS has 31.2 box APr, while the few-shot one only has 16.6 box APr). Our proposed method DE-ViT outperforms the SoTAs on both few-shot and open-vocabulary (33.6 box APr), which has not been achieved before by existing few-shot work.

Reviewer comment (Experiments):
The visual novel objects are sampled from COCO or LVIS, which also may lead to data leakage. What about the classes sampled from ImageNet-21k to build prototypes?

Author response:
**(W4, data leakage) "The visual novel objects are sampled from COCO or LVIS, which also may lead to data leakage. What about the classes sampled from ImageNet-21k to build prototypes?"**

Sampling base and novel objects from the same datasets is the standard evaluation procedure for few-shot object detection [1].  Regarding the concern for data leakage, all annotations for novel categories are removed during training. So even if the network has seen the images with novel objects inside, the novel objects will be treated as background. There is no data leakage.  In fact, this makes it even more difficult for detecting novel objects that are present in the training set.

Reference: 

[1]: Few-shot object detection: a comprehensive survey, IEEE Transaction on Neural Networks and Learning Systems, 2023

Reviewer comment (Experiments):
The model needs extra Region Proposal Network along with backbone. What about the shared frozen DINOv2-ViT backbone design such as works [1][2]?

Author response:
**(W5, detector over frozen backbone) "The model needs extra Region Proposal Network along with backbone. What about the shared frozen DINOv2-ViT backbone design such as works [1][2]?"**


Though both using a frozen backbone, our proposed method DE-ViT delivers more accurate detection results than F-VLM [2] on COCO 2017 (39.5 AP50, 24.1 AP75 vs. F-VLM 28.0 AP50, AP75 not reported), with our propagation-based localization mechanism. Our region proposal network involves small computation cost.



| Conventional  REG Head (baseline) | Expanded Proposal | Region Propagation | COCO nAP50 | COCO nAP75 |
| --- | --- | --- | --- | --- |
| ✓ |  |  | 37.7 | 14.6 |
| ✓ | ✓ |  | 35.6 | 12.5 |
|  | ✓ | ✓ | **39.5** | **24.1**  |


In Table 8, our proposed method DE-ViT achieves 39.5 AP50 and 24.1 AP75 on COCO with our propagation-based localization. However, when replacing propagation architecture with a conventional regression head, DE-ViT (w contentional regression) only achieves 37.7(-1.8) AP50 and 14.6(-9.5) AP75. The massive drop of AP75 shows the effectiveness of our propagation localization on the frozen DINOv2 backbone. F-VLM [2] trains detectors on frozen backbone using conventional FPN detector head, and has 28.0 AP50. F-VLM does not report AP75.



Our region proposal network involves a small computation cost. In our model for LVIS, RPN has only **27M** parameters, and our total parameters for ViT-L backbone is 350M. While the F-VLM contains a total parameters of 450M without additional RPN, where the backbone (RN50x64) has 420M parameters. In our model for COCO, RPN has only **8M** parameters.


|  | Backbone | Total Params | Trained Params |  APr |
| --- | --- | --- | --- | --- | 
| F-VLM [2] | RN50x64 | 445M  (backbone 420M) | 25M |  32.8 |
| DE-ViT (Ours) | ViT-L | **350M** (backbone 320M, RPN 27M) | **23M** | **34.3** |



We added the mentioned papers [1, 2] into the reference of our paper and incorporated this answer into the efficiency analysis at Section A.1.3 .

Reference: 

[1]. Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP, NeurIPS 2023

[2]. F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models, ICLR 2023

Reviewer comment (Evaluation):
Missing parameters and GFLops analysis.

Author response:
**(W6, parameter analysis) "Missing parameters and GFLops analysis."**

We added the parameters and training epochs comparison for recent detectors on LVIS in Table 6.


Compared to the recent detectors trained on LVIS, our method only has 23M trainable parameters, and is trained orders of magnitude faster than F-VLM and OWL-ViT. We also provide inference time comparison in Tables 5, A2, and A3.


|  | Backbone | Total Params | Trained Params | Training Epochs | APr |
| --- | --- | --- | --- | --- | --- |
| OWL-ViT [3] | ViT-L | 433M | 433M | 1800 | 31.2 |
| F-VLM [2] | RN50x64 | 445M | 25M | 118 | 32.8 |
| DE-ViT (Ours) | ViT-L | **350M** | **23M** | **14.4** | **34.3** |


| Few-shot Methods | Backbone | Total Params | Trained Params | Inference Time (Sec / Img) | nAP50 |
| --- | --- | --- | --- | --- | --- |
| Meta Faster RCNN [4]  | RN101 | 82M | 82M | 0.61 | 31.8 |
| Cross Transformer [5] | Custom | 24M | 24M | 3 | 35.8 |
| DE-ViT (Ours) | ViT-S | 70M | **23M** | **0.25** | **42.7** |

| Method | COCO nAP50 | Secs/Img |
| --- | --- | --- |
| CORA [6] | 41.7 | 0.5 |
| DE-ViT (Ours)  | **50** | **0.33** |

| Method | LVIS box APr | Secs/Img |
| --- | --- | --- |
| OWL-ViT [3] | 31.2 | **0.42** |
| DE-ViT (Ours)  | **32.6** | 0.5 |



For few-shot detection, our proposed method DE-ViT has the smallest inference time of 0.25s/img, compared to 0.61s/img of Meta Faster RCNN and 3s/img of CrossTransformer, while having better performance. DE-ViT also has the smallest trainable parameters. For open-vocabulary detection, DE-ViT is faster (0.33s/img vs 0.5s/img) and more accurate (50 AP50 vs 41.7 AP50) than CORA , the SoTA on COCO. DE-ViT is more accurate (32.6 box APr vs 31.2 box APr) and slightly slower (0.5 s/img vs 0.42 s/img) than OWL-ViT. While our method DE-ViT uses PyTorch, OWL-ViT uses JAX [1]. JAX is commonly regarded as faster in general than PyTorch. Inference time of all methods is measured using  the same machine.

Reference:

[1]: "Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more" https://github.com/google/jax

[2]: F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models, ICLR 2023

[3]: OWL-ViT: Simple open-vocabulary object detection, ECCV 2022

[4]: Meta faster r-cnn: Towards accurate few-shot object detection with attentive feature alignment, AAAI 2022

[5]: Few-shot object detection with fully cross-transformer, CVPR 2022

[6]: CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching, CVPR 2023

Reviewer comment (Writing):
Should the notation be "[C]\\c_k" rather than "[C]/c_k"?

Author response:
Eq.3 sorts and resamples the input similarity map from shape $H\times W\times (C-1)$ to $H\times W\times T$, where $C$ is the number of classes and $T$ is a fixed hyper-parameter.


In details, since the number of classes $C$ is different per dataset, the shape of the similarity map is also different, which is problematic because the similarity map is used as the input to the binary classification network, which is the same for all datasets. Therefore, in order to standardize the input size, we resample them to $H\times W\times T$. The intent is not to balance the number of prototypes and base categories. 

We updated the paper to incorporate this answer in Section 3.1.

Reviewer comment (Theory):
The upsampling process described in Eq.3 is somewhat unclear. Could you first elucidate the underlying objective or insight you aim to convey? The rationale behind upsampling on the channel dimension is not immediately evident. If the intent is to balance the number of prototypes and base categories, might a re-weighting approach similar to the focal loss be more appropriate?

Author response:
We added a visual analysis of the propagation localization from expanded proposals covering same-class objects in the Section A.1.1 at Page 14. We will use the term over-expanded proposals to denote the scenario mentioned by the reviewer.


| Score Threshold | Average IoU of refined boxes with groundtruth from over-expanded Proposals | Average IoU of refined boxes with groundtruth for other proposals |
| --- | --- | --- |
| 0 | 0.49 | 0.59 |
| 0.1 | 0.56 | 0.69 |
| 0.2 | 0.58 | 0.72 |
| 0.3 | 0.60 | 0.74 |
| 0.4 | 0.62 | 0.75 |
| 0.5 | 0.63 | 0.77 |
| 0.6 | 0.65 | 0.78 |
| 0.7 | 0.68 | 0.80 |
| 0.8 | 0.72 | 0.83 |
| 0.9 | 0.76 | 0.86 |
| 0.95 | 0.82 | 0.87 |




As shown in Figure A1, over-expanded proposals generally degrade localization accuracy, as their final predicted boxes have smaller IOU towards the ground truth. But the degradation is far from total, e.g., they still produce boxes whose IOU > 0.7 on average, under the score threshold of 0.85,  which we use to generate qualitative visualizations. We also observe that the over-expanded proposals occupy around 7% of all proposals in our model for COCO, and only half of them appear in the final prediction (after NMS and score filtering). This means even if an over-expanded proposal predicts an inaccurate box, its impact is softened by filtering of NMS and score thresholding.

In terms of actual behavior, we listed success and failure cases of over-expanded proposals in Figure A7 and A8. In failure cases, propagation either encompasses both objects, or produces erratic boxes. But all failure cases happen under inferior proposals, where the proposals are already poorly located and do not cover any object, or cover multiple objects before expansion. In success cases, propagation generally prefers central objects, but can locate an object accurately regardless of the proposal quality. This means our propagation localization does not fully rely on the proposal quality.

Reviewer comment (Reproducibility):
During the propagation procedure, what transpires if the expanded box encompasses another object of the same category?

Author response:
The first line Eq.4 computes the expected position of the heatmap as the bounding box center, the second line computes a weighted average of row sum of the heatmap as the bounding box width, and column sum as the height. 


In more details, consider a toy example of converting a binary mask into a bounding box, a reasonable approach is to compute the mask center as the bounding box center, and pick the maximum row and column sum as width and height, as illustrated in Figure **[X]**. Motivated by this, our method estimates the center by computing the expected position under the spatial distribution $softmax(\textbf{g})$, where $\textbf{g}$ denotes the heatmap values. To emulate the row / column sum, we compute  $\sum_{j=1}^W  \sigma(\textbf{g})_{ij}$, which sums up the sigmoid activation along the i-th row. Instead of picking the maximum, we aggregate all row sums in terms of magnitude. The rationale of this aggregation is that a larger estimation is more likely to over-cover the entire object and a small estimation may produce a box that is too tight. The aggregation is done by sorting the estimation and then weighted averaging, which explains the use of order statistics notation $(i)$ and $(j)$. 

We updated the paper to incorporate this answer in Section 3.2 and add an illustrative example in Figure 5.

Reviewer comment (Writing):
Could you provide a more detailed explanation of Eq. 4, particularly concerning the `w` and `h` components? An illustrative figure might enhance comprehension.

Author response:
"bAP" denotes mAP on base classes. For "Split-n", one-shot methods divide 80 classes of COCO2017 into four even partitions, and alternatively takes three partitions as base classes and one partition as novel classes. There are 4 base/novel splits in total, named as Split-1/2/3/4. 



We clarify the evaluation metrics used in the papers. We updated Paragraph 2 in Section 3 for a more comprehensive description of evaluation metrics.

- Few-shot 
    - bAP: mAP on base classes
    - nAP, nAP50, nAP75: mAP, AP50, AP75 on novel classes
- One-shot
    - Seen -> Split-1/2/3/4: AP50 on base classes, for each split
    - Unseen -> Split-1/2/3/4: AP50 on novel classes, for each split
    - Avg: AP50 averaged among all splits, for base or novel classes
- Open-vocabulary
    - COCO
        - Novel: AP50 on novel classes
        - Base: AP50 on base classes
        - All: AP50 on all classes
    - LVIS
        - APr: AP on rare categories, which are novel classes
        - AP: AP on all classes
        - APc: AP on common categories, which are base classes
        - APf: AP on frequent categories, which are base classes

Reviewer comment (Writing):
For the definition of h, it might be better to use "h=\\bar{f}\\dot p" directly to create better reading experience.

Author response:
We thank the reviewer for pointing out the notation issue. We updated Figure 3 to include f and h notations as requested. We fixed all equations and figures to change $[C]/c_k$ to $[C]\backslash c_k$. We also rewrote the definition of h to use dot product.

Reviewer comment (Evaluation):
What are the limitations and failure cases of this work?

Author response:
One of the limitations is that our current architecture is a mix of ViT and RCNN, while a full transformer network can clearly be more scalable and unlock more possible abilities and integrations to other modalities. Another limitation is that our current model relies on an external region proposal network (RPN). It is possible to train an RPN on top of frozen DINOv2 with some engineering efforts. 


We added some failure examples on COCO in Figure A4 in Appendix. Image (a) in Figure A4 detects the floor as a dining table, possibly due to visual similarity. Image (b) shows a single box over two cuddling cats, likely caused by very little visual separation between the two.  We discussed limitations of our work in Section 5. We updated the paper to include more qualitative results and failure case visualization in Figure 8, A4, A9, A10, and A11.

Reviewer comment (Novelty):
The method lacks some novelty and doesn't look attractive to me. This is the main problem of the paper. So many parts ensembles together only cost more computational resources.

Author response:
We are greatly thankful to the reviewer for the comments. We hope the following reply resolve the concerns raised in the review.

**(W1, part1, novelty)  "The method lacks some novelty and doesn't look attractive to me. This is the main problem of the paper. "**
 

Existing few-shot detection approaches can be broadly classified into finetuning-based [3-6, 9-14] and meta-learning-based strategies [1-2]. Finetuning-based methods demand redundant multi-stage training procedures [7, 8]. Meta-learning methods avoid finetuning by online adaptation, but exhibit inferior accuracy [15].

Contrary to existing work, we are the first to build a few-shot detector on top of DINOv2. Our method transforms the multi-class classification into multiple binary classifications, so a binary classifier can be trained and used for all classes without any finetuning. Our method localizes objects with a novel propagation-based mechanism that we developed. The new mechanism is based on computing similarity maps between features and prototypes.



Our method outperforms existing few-shot detection models by a significant margin on COCO (+15 nAP on 10shot, +7.2 nAP on 30shot) and LVIS (+20 box APr) while eliminating the finetuning requirement. We conducted an ablation study (See Section 4.3) to demonstrate that every proposed component is important for the performance of our proposed method.

| Conventional Prototypes Head (baseline) | Prototype Projection | Background Tokens | Class Reorder and Resample | COCO nAP50 | COCO nAP75 |
| --- | --- | --- | --- | --- | --- |
| ✓ |  |  |  | 4.5 | 2.2 |
|  | ✓ |  |  | 26.2 | 9.7 |
|  | ✓ | ✓ |  | 38.4 | 23 |
|  | ✓ | ✓ | ✓ | **39.5**| **24.1** |


| Conventional  REG Head (baseline) | Expanded Proposal | Region Propagation | COCO nAP50 | COCO nAP75 |
| --- | --- | --- | --- | --- | --- |
| ✓ |  |  | 37.7 | 14.6 |
| ✓ | ✓ |  | 35.6 | 12.5 |
|  | ✓ | ✓ | **39.5** | **24.1**  |



We revised the introduction section to highlight our technical contributions.

Reference: 

[1]: Few-shot object detection via feature reweighting, ICCV 2019

[2]: Meta r-cnn: Towards general solver for instance-level low-shot learning, ICCV 2019

[3]: Frustratingly Simple Few-Shot Object Detection, ICML 2020

[4]: Query adaptive few-shot object detection with heterogeneous graph convolutional networks, ICCV 2021

[5]: Generalized few-shot object detection without forgetting, CVPR 2021

[6]: Fsce: Few-shot object detection via contrastive proposal encoding, CVPR 2021

[7]: Semantic-aligned fusion transformer for one-shot object detection, CVPR 2022

[8]: Balanced and hierarchical relation learning for one-shot object detection, CVPR 2022

[9]: Label, verify, correct: A simple few shot object detection method, CVPR 2022

[10]: Few-shot object detection and viewpoint estimation for objects in the wild, TPAMI 2022

[11]: Meta faster r-cnn: Towards accurate few-shot object detection with attentive feature alignment, AAAI 2022

[12]: Few-shot object detection with fully cross-transformer, CVPR 2022

[13]: NIFF: Alleviating Forgetting in Generalized Few-Shot Object Detection via Neural Instance Feature Forging, CVPR 2023

[14]: DiGeo: Discriminative Geometry-Aware Learning for Generalized Few-Shot Object Detection, CVPR 2023

[15]: Few-shot object detection: a comprehensive survey, IEEE Transaction on Neural Networks and Learning Systems, 2023

Reviewer comment (Evaluation):
On the few-shot benchmark, the good experiments are more shown on 10 or more shot rather than fewer-shot. There are so many different measures shown in the experimental results, which are not convincing to me about its real performance.

Author response:
**(W2, part1, performance on fewer shots) "On the few-shot benchmark, the good experiments are more shown on 10 or more shot rather than fewer-shot. "**


Regarding the results on fewer shots, it is common practice for few-shot detection work to only report 10, 30 shots results on COCO, such as [1-4].  In the revised paper, we included plots of the model performance for fewer shots in Figure 7 and A2 for COCO 2014 and COCO 2017, correspondingly. In Section 4.2, we identified the inflection points for each dataset, after which adding more shots does not help. Our model outperforms existing few-shot SOTA results on all shots. 

We updated the experiment section to include the performance analysis with more shots in Section 4.2 and Section A.1.2.

Reference:

[1]: Frustratingly Simple Few-Shot Object Detection, ICML 2020

[2]: Label, verify, correct: A simple few shot object detection method, CVPR 2022

[3]: Few-shot object detection and viewpoint estimation for objects in the wild, TPAMI 2022

[4]: DiGeo: Discriminative Geometry-Aware Learning for Generalized Few-Shot Object Detection, CVPR 2023

Reviewer comment (Presentation):
There is no visualization of the experimental results shown in the paper.

Author response:
**(W3, visualizations) "There is no visualization of the experimental results shown in the paper."**

In the original submission, we have the visualization on detecting YCB objects with our proposed method DE-ViT in Figure 1. In the revised version, we added Figures 8, A9, A10, and A11 to include qualitative results and comparisons of our method and Meta Faster RCNN [1], CORA [2] on COCO and LVIS datasets.  DE-ViT detects more novel objects while having much fewer false positives, as shown in Figures A10 and A11. Base and novel objects are colored in "green" and "yellow", respectively.

Reference: 

[1]: Meta faster r-cnn: Towards accurate few-shot object detection with attentive feature alignment, AAAI 2022

[2]: CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching, CVPR 2023

Reviewer comment (Presentation):
In figure 1, which objects come from based classes and which are the few-shot/novel categories? There are different colours used in the image but no legend. This makes it difficult to ascertain the performance of the proposed approach based on this image.

Author response:
We added legends and colors to differentiate the base and novel categories in Figure 1. Specifically, "green" denotes base classes, and “yellow” denotes novel classes. Our proposed method DE-ViT's performance is demonstrated by detecting novel classes such as: mustard, clamp, cheez-it, coffee jar, peg-hole, toy airplane box, comet pine, wood blocks jar, chips, spray bottle, cleaser bottle.

Reviewer comment (Experiments):
If you are selecting the top-K classes in one step, why is there a need to up-sample to T? Can't we just keep using "K" because that is deterministic - just up/down-sample to K?

Author response:
$K$ and $T$ serve different purposes. $K$ is used to select classes, reducing unnecessary computation. $T$ is used to standardize the shape of the similarity map, while preserving information. So they should not be set identical.


In details, we select top $K$ classes to estimate class score, so we want $K$ to be as small as possible. In practice, we use $K=3$ or $K=10$ in our experiments. To estimate probability for each selected class, the input is a similarity map $H\times W\times C$ between features and prototypes, where $C$ denotes number of classes. As $C$ is different per dataset, we standardize the input shape to $H\times W\times T$, where $T$ is a hyperparameter and set to 128 in our experiments. 


We updated the paper to incorporate this answer in Section 3.1 .

Reviewer comment (Novelty):
Do prior one-shot object detection methods also follow the few-shot paradigm of fine-tuning the model? If not, what exactly is the difference between the proposed method and one-shot methods?

Author response:
One-shot object detection does not follow the finetuning paradigm of few-shot. However, one-shot detection restricts the setting to single-class detection [1], and focuses on designing interaction mechanisms of dense spatial features between the support image and target images. One-shot detection requires separate inference for each class and does not leverage additional support images even if available, as mentioned in the related work [2]. 

Compared to one-shot detection work, our method focuses on the interaction between class-level prototypes with dense features of target images, instead of interaction of dense spatial features between support and target images. Our method does not require separate inference for each class and can leverage multiple support images. 

[1]: One-Shot Instance Segmentation, NeurIPS 2018

[2]: One-Shot Object Detection with Co-Attention and Co-Excitation, NeurIPS 2019