Paper: InstructDET: Diversifying Referring Object Detection with Generalized Instructions (ICLR-2024)

Reviewer comment (Experiments):
The experiments are insufficient. The author only presents the result in their own proposed dataset and the original ref coco benchmark. I am not certain how much the dataset actually helps the performance of the model, or helps equip the model with new capabilities. Specifically, the performance on refcoco is not extraordinarily high and is not compared with SOTA methods such as poly former.

Author response:
Thanks for your valuable comments. From Table 2, we observe that ROD models trained on the existing datasets are not effective to ground objects in our InDET test set. This is because our InDET dataset contains much more diversity of human-like expressions than existing datasets. To this end, the evaluation on the RefCOCO test set, whose diversity is limited, is not effective to demonstrate the superior model generalizations brought by our InDET dataset. 

In order to further evaluate how our dataset helps model to improve performance or equip new capabilities, we have performed a new evaluation below. We train existing models (i.e., MDETR and UNINEXT)  using different training sets (i.e., RefCOCO/+/g, Flicker, and our InDET). Then, we select 2000 test samples from our InDET test set where these samples focus on logical reasoning in ROD. The evaluation results are shown in the following table:
| Method | Datasets | AP |
| :-----| :----: |:----: |
| MDETR | Refcoco/+/g | 18.91 |
| MDETR | Flicker | 10.14 |
| MDETR | InDET | 49.37 |
| UNINEXT | Refcoco/+/g| 22.88 |
| UNINEXT | Flicker | 16.07 |
| UNINEXT | InDET | 56.90 |

This table indicats that exisiting models achieve significant performance gain by using our InDET dataset. As our test set focuses on logic reasoning, we can conclude that our InDET improves the logic reasoning of representative existing models. We have added this comparison in Fig.7 together with visual examples.

Revised Sections:

In Sec. 6, we incorporate further discussions on multimodal logical reasoning capabilities. Additionally,  Fig. 7 provides a quantitative and qualitative analysis of this ability.

Reviewer comment (Reproducibility):
What does the author mean by “initializing LLAVA with minigpt4 weights”? I think LLAVA finetunes the entire LLM, while minigpt4 only finetunes a linear layer, and the architectures are different.

Author response:
We apologize for the confusion and would like to make a clarification here. The LLaVA we mention here is indeed a multimodal paradigm that maps visual features into token embeddings with further alignment. This paradigm is similar to minigpt4 and differs from the structure from the original paper. We have changed this expression into "initializing a derived model from LLaVA with minigpt4 weights''.

Revised Content: We have added the footnote on page 4 for clarification.

Reviewer comment (Experiments):
It's not clear from the paper that the method comparison in table 2 is fair because it's not clear if the models were trained on the same images and bounding boxes.

Author response:
We would like to clarify that in Table 2 where we evaluate on the InDET test set, existing methods (i.e., MDETR, G-DINO, UNINEXT) are trained on RefCOCO/+/g datasets where the images and bounding boxes are the same. For a fair comparison, our DROD model in Table 2 only utilizes RefCOCO/+/g datasets but with diversified instructions. Our superior results indicate that our diversified instructions enrich the alignment of visual objects and language expressions in DROD model.

On the other hand, we perform another evaluation below where all the methods only use RefCOCO dataset for training and testing. The training data of existing methods is the RefCOCO training set, while the training data of our DROD is the same training set but with diversified instructions. For the testing data, all the methods use RefCOCO testing set but with diversified instructions. Through this configuration, we would like to show that existing models trained on RefCOCO dataset are not effective to generalize, while our diversified instructions improve our model generalizations to achieve favorable performance.

| Method | Images (train and test) | AP |
| :-----| :----: | :----: |
| MDETR | Refcoco | 36.83 |
| G-DINO | Refcoco | 37.90 |
| UNINEXT | Refcoco | 44.70 |
| DROD (Ours) | Refcoco | 64.38 |

The results on the table show that the diversified instructions improve our model performance under the same image and bbx dataset.  
&nbsp;

Reviewer comment (Experiments):
One additional interesting experiment that would make this a stronger paper - run SAM on web scale images to get bboxes for images and then scale up this approach further to understand if the benefits saturate.

Author response:
Thanks for your comments. Our data-centric method is capable of generating image-bbx-instruction triplets (i.e., expanding dataset) automatically given bbxs predicted by SAM on web scale images. And our ultimate goal is to train a ROD model with this dataset to generalize ROD on a wide range of images and user expressions. As the size of dataset increases, we need to redesign this ROD model from the perspectives of model size and training configurations accordingly. Such effort needs a significant amount of time and we are unable to accomplish it at present. Nevertheless, we will definitely set this direction as our future work and introduce possible solutions (e.g., advanced learning strategy such as curriculum learning) to overcome the saturation if it really happens.

Reviewer comment (Writing):
There are many language errors throughout the submission, including strange use of words, grammatical mistakes etc.. It is still possible to comprehend the ideas despite those writing issues, but this submission can use additional proofreading. Part of the issue is with the naming of concepts. The "pathways" are easily confused with the terminology used to describe parallel branches in a deep network. The "dropout" can be confused with the regularization method. These names are unfortunate in the sense that they are in fact significant distractors for readers in this field, and not really descriptive of the concepts.

Author response:
We apologize for making you confused while presenting our method and would make a clarification here. The "pathway'' we use is to describe that in our framework, there are two pipelines to produce expressions (as shown in Fig. 2).  We have replaced "pathway'' with "pipeline'' in our revised manuscript. For "dropout'', we mean to wipe out expressions that are inaccurately generated via both LLaVA and LLaMA. We have replaced "dropout'' with "filter'' in our revised manuscript. Moreover, we have thoroughly proofread our paper to improve our presentation for reducing distractions of readers. For example, in Sec 3.1, we revised "The text prompt we use for LLaVA contains our language guidance **that the specific object category** shall be mentioned" into "The text prompt we use for LLaVA contains our language guidance **emphasizing that several specific interested object categories** shall be mentioned". And in Sec 3.3, we revised "The global image description is mitigated via our model finetuning to focus on local object, but not completely disappeared" into "The **tendency to generate** global image description is mitigated via our model finetuning to focus on local object, but not completely disappeared".

Reviewer comment (Writing):
I find the definition of "single" versus "multi" modality pathways confusing. To my best understanding, the "single modality pathways" refers to a sub-process in the data generation that uses LLaVA to create output texts descriptions for an input image. As such, it needs both an image and some text prompts to obtain text result. The "multi modality pathways" refers to a procedure that similarly use LLavA to create results from image & text prompts. The two pathways have the same form of inputs and outputs. Why one is "single modality", while the other is "multi modality"? It is also hard to see, at least from the discussions provided in the submission, why the "single modality" case can use an out-of-box LLaVA model, whereas the "multi-modality pathways" must use a finetuned version.

Author response:
We apologize for the confusion and make the clarification here. We agree that "single modality pipeline'' refers to the global image description that involves LLaVA for prediction. Specifically as illustrated in Sec. 3.1, we first obtain global image description and then sent this description, object bbxs axis, and in-context examples to LLaMA. Originally, we mean "single modality'' because we feel all the elements sent to LLaMA are text tokens. And we mean "multi-modality'' because we send visual images with marked bbxs, together with text tokens, to LLaVA as illustrated in Sec. 3.2. 

For your comments, we agree that it is inappropriate to name same form of inputs and outputs as "single modality'' and "multi modality''. To this end, we have modified our illustration that we replace "single modality pathway'' with "global prompt pipeline'', and replace "multi-modality pathway'' with "local prompt pipeline''. This replacement is based on the functions of LLaVA. In "global prompt pipeline'', we use LLaVA to obtain the global text description (i.e., Sec. 3.1). In "local prompt pipeline'', we use LLaVA directly to perceive the local visual objects (i.e., Sec. 3.2). We have revised these illustrations in our revised manuscript according to your advice. 

The reason why we use an out-of-box LLaVA model in "global prompt pipeline'' (i.e., "single modality'') is that LLaVA only provides global image description for an input image. In this pipeline, expressions are generated via LLaMA that takes only text tokens as inputs. As LLaVA is initially trained via image and text pairs, there is no need to finetune LLaVA to obtain global image descriptions. However, in the "local prompt pipeline'' (i.e., "multi modality''), we directly leverage LLaVA to generate expressions by taking images with marked bbxs and text tokens as ipnuts. As we expect LLaVA to effectively perceive local regions around each object, a finetuning is required to transfer the global image description of LLaVA into local perception.  In Figure 14 of Appendix H, we show that without fine-tuning, LLaVA achieves only 11.4% in generating local target descriptions. However, when fine-tuned on the REC dataset, LLaVA is improved up to 80.2% .

Reviewer comment (Reproducibility):
Could you include more details in terms of the prompts that are use, the exact procedure to generate them (number of queries, the services used, costs etc.)? I think these are interesting information for the community that should be included to the paper wherever appropriate.

Author response:
Sure, we'd love to share more details of the generation procedure in InDET. 

To be clear, the diversified instructions in InDET are generated by the procedure we illustrate in Sec. 3, which leverages LLAMA and LLAVA. The prompts for LLAMA and LLAVA are empirically designed.

We propose both local and global pipelines to generate diversified instructions in InDET. These steps are illustrated in Sec. B and Sec. C. The objective and prompt in each pipeline can be summarized as follow: 

- **local prompt pipeline** (i.e., multi-modality pipeline)  
In this pipeline, we obtain diversified instructions of one single bbx in an image by querying LLAVA for multiple times. The prompts for LLAVA consist of visual and textual content. The visual content is the image with a red object bbx marked on it, and the textual content is empirically designed and provided in Table 4.

- **global prompt pipeline** (i.e., single modality pipeline)  
In this pipeline, we obtain instructions related to multiple interested objects in a given image by LLAMA. The prompts for LLAMA consist of task description, in-context examples and current image information. The details of these elements are explained in **Response for Weakness 3**. 

To design appropriate prompts for LLAMA and LLAVA, we start from prompts commonly used in previous works or in the LLM community. We have tried some methods such as editing language expression, adding in-context examples, etc, to optimize the current prompts until achieving expected output. 

For services and cost of the procedure, thanks to the contributors of LLAMA and LLAVA, we can run those methods locally on our machines based on available implementations. And we add the details of hardware resource and time consumption cost of running the two methods in Appendix K.

Reviewer comment (Reproducibility):
Do you plan to release the dataset curated for this paper?

Author response:
Yes. We will release our InDET dataset to benefit future ROD research.

Reviewer comment (Experiments):
More experiments are expected. For instance, previous REC models should be trained on InDET and evaluated on RefCOCO/+/g.

Author response:
We have conducted evaluations on previous REC models under the suggested configurations. The results are shown on the below table. 
| Method | backbone | Data | Refcoco val | Refcoco testA | Refcoco testB| Refcoco+ Val | Refcoco+ testA | Refcoco+ testB | Refcocog val | Refcocog test |
| :-----| :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |
| MDETR | Res101 | RefC | 82.98 | 83.84 | 77.83 | 77.94 | 80.28 | 68.33 | 78.30 | 77.37 |
| MDETR | Res101 | InDET | 84.67 | 85.34 | 80.51 | 79.17 | 83.39| 71.49 | 78.95 | 77.84 |
| G-DINO | SwinB | RefC | 88.76 | 91.07 | 85.23 | 79.41 | 86.31 | 70.64 | 82.79 | 81.35 |
| G-DINO | SwinB | InDET | 89.04 | 91.22 | 87.42 | 79.97 | 86.72 | 73.85 | 86.26 | 83.12 |

For each REC method, we perform two experiments. One is to finetune the model on RefCOCO/+/g datasets directly. The other one is to pretrain on InDET and then finetune on RefCOCO/+/g datasets. For a fair comparison, here our InDET dataset only utilizes RefCOCO/+/g datasets but with diversified instructions. 

For MDETR, the original MDETR performs pre-processing on the expression to extract root of the expression using a dependency parser, and train the model to predict bbox match the root token only. This training framework is unsuitable for ROD task where the explicit name of the referring target might not show up in the expression. Inspired by UNINEXT, we replace the predefined linear class embedding layer with a visual-language alignment module to compute the matching extent between boxes and the whole expression. For G-DINO, the results are produced by the model retraining. The above table shows that the performance of existing REC models (i.e., MDETR and G-DINO) is improved by training with our InDET dataset, compared to the results by training with RefCOCO/+/g datasets.