Paper: GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest (ICLR-2024)

Reviewer comment (Presentation):
On presentation, a flowchart of the two-stage training process might be useful to readers. Indicating which datasets are used in which fashion at which stage, with the purpose of bringing about which kind of capabilities.

Author response:
In the current presentation format, it does appear that the two stages with multiple datasets are somewhat confusing. Introducing a flowchart could indeed be a viable improvement, and we will strive to enhance the presentation format in future versions accordingly. Thank you for your valuable suggestion.

Reviewer comment (Experiments):
How about visual question-answering datasets?

Author response:
When aiming to enhance performance in image-level VQA tasks, a practical approach for GPT4RoI could involve integrating an off-shelter detector and combining region of interest features with image features to represent an image. Previous research [1] has demonstrated the effectiveness of this technique in image caption task. Given the specific focus of our work on region-understanding tasks, we will further investigate this approach in our new research.

Reviewer comment (Experiments):
Is it possible to design some synthetic tasks to train the model, perhaps inserted between the two stages you have right now, to smooth up the training process?

Author response:
Utilizing region-text pair data and ChatGPT, building region-level multi-round understanding and reasoning tasks is a practical way to improve the dialogue ability of GPT4RoI. A related study [2] has conducted similar work following our research.

Reviewer comment (Reproducibility):
Is it possible to let the LM output coordinates somehow so to interact with users by referring to certain regions in the picture?

Author response:
This suggestion is highly valuable. Currently, GPT4RoI can only mention the input RoI and requires users or detectors to provide all the region of interest. Two remarkable follow-up works [3][4]  of GPT4RoI utilize a significant amount of visual grounding data, allowing LLM to output new regions. This enables the model to have an enhanced interactive experience. We will continuously improve the GPT4RoI to consolidate these two functions in the future.

Reviewer comment (Experiments):
For region caption, the paper only compares the proposal to one model GRiT, which is limited.

Author response:
To assess the region's comprehension capabilities, we rely on ground truth bounding boxes or detection model outputs as input. Consequently, access to the method source code is necessary for conducting the evaluation. **GRiT is the top-1 performance model among all open-sourced projects in the region-caption field**, and serves as a representative baseline in recent studies when specifically evaluating region-caption abilities. We will continuously track the latest advancements in the region caption field and update our results accordingly. We appreciate your valuable feedback and suggestions.

Reviewer comment (Evaluation):
Can you interpret the performance of different tasks (ie Q->A, QA-> R, Q->AR) in Table 6?

Author response:
The VCR dataset is a particularly difficult dataset for comprehension and reasoning. It has gained attention from several well-known companies, who have submitted their solutions on the leaderboard. The dataset's distinctive challenge is that a model not only needs to answer complex visual questions but also provide a rationale for why its answer is correct.

The VCR task consists of two subtasks: Question Answering (Q->A) and Answer Justification (QA->R). In the Q->A setup, a model is given a question and must select the correct answer from four choices. In the QA->R setup, a model is provided with a question and the correct answer, and it needs to justify the answer by selecting the most appropriate rationale from four choices. The performance of models is evaluated using the Q->AR metric, where accuracy is measured as the percentage of correctly answered questions along with the correct rationale.

Reviewer comment (Novelty):
Expanding from image-level to region-level instruction tuning seems like a natural progression, and the approach is straightforward without providing a fresh perspective. Some other papers also explore the region-level large language models [1] but lack the performance comparison.

Author response:
(1). Regarding question 1, about the novelty and comparison with ChatSpot:

- **Expanding from image-level to region-level instruction tuning is a fundamental progression in multimodal large language models**
    
    The emergence of ChatGPT(LLM) is well-recognized as a new era of deep learning, just like AlexNet in 2012. At that time, how to expand AlexNet used for image classification task to object-level recognition tasks is a big challenge for the whole community, until the proposal of R-CNN in 2014. Although R-CNN is a quite simple design, I believe no one would say that it is just a natural progression from AlexNet to R-CNN. Similarly,  our proposed GPT4RoI is motivated to expand the LLM’s visual instruction tuning from image-level to region-level and serves a fundamental role in the related area.
    
    A large portion of academic researchers pursue sophisticated designs and even agree that complexity is equal to innovation, this is definitely wrong! No matter Occam’s razor or practical applications is telling ***a basic rule: the simpler, the better***. GPT4RoI incorporates region references in instruction and developing the pioneering model for general-purpose region understanding, we definitely believe it is novel and provides a fresh perspective for the region understanding field.
    
- **Concurrent Independent Work: impossible to compare when submitting the manuscript**
    
    It is important to note that GPT4RoI **predates** ChatSpot by **about two weeks** and serves as a concurrent research project. Furthermore, the unavailability of ChatSpot's code poses challenges in conducting a comprehensive comparative analysis. Based on their report, ChatSpot addresses the reference problem by incorporating position tokens, which may require more region data to realign the position token with the image position compared to GPT4RoI. Additionally, it is worth mentioning that ChatSpot concatenates region position at the end of the instruction, while GPT4RoI formulates interleaved instructions. This difference makes it challenging for ChatSpot to handle multiple-region understanding tasks like VCR. As a verification, ChatSpot has not presented any demo for multi-region understanding.

Reviewer comment (Evaluation):
It appears that while this paper utilized more datasets for training, the improvement in results is relatively marginal, as shown in Table 5.

Author response:
(2). Regarding question 2, utilizing more datasets for training, the improvement in results is relatively marginal, as shown in Table 5.

- The baseline **12-in-1[1]** in Table.5 utilizes 12 pre-training datasets, clearly indicating the use of more data compared to our method.
- Table.5 (visual 7W) is just one of the benchmarks for reasoning ability. On the other hand, the VCR dataset (Table.6), which has gained considerable attention in recent years and attracted submissions from many companies, is more popular and challenging. We have demonstrated our strong performance on this dataset with significantly less training data, achieving an improvement of **6 points** compared to the previous state-of-the-art. Therefore, we believe that labeling our method's improvement in results as relatively marginal is one-sided.

Reviewer comment (Experiments):
This work lacks a comparison of parameters. The current models seem to be quite large, especially large language models. A comparison should be conducted at the same parameter level, e.g., Table 6.

Author response:
(3). Lacks a comparison of parameters in Table.6.

The table below presents the rank, and accuracy of Q-AR, along with the parameters and datasets of various solutions, from the most recent **[VCR leaderboard](https://visualcommonsense.com/leaderboard/)** update on November 11, 2023. We retained the top 6 methods and 2 methods that have reports.

| model | rank | Q-AR(%) | Parameters | Datasets | Open Source |
| --- | --- | --- | --- | --- | --- |
| GPT4RoI | 1 | 81.6 | 13B+ | 7 open-source region data | Y |
| HunYuan | 2 | 75.6 | confidential | confidential | N |
| Shopee | 3 | 74.4 | confidential(ensemble of 4 models) | confidential | N |
| KDDI Research and SNAP | 4 | 74.3 | confidential | confidential | N |
| Kuaishou | 5 | 74.0 | confidential | confidential | N |
| VQA-GNN + MerlotReserve-Large | 6 | 74.0 | 1B+ (ensemble of 2 models) | YT-Temporal-1B | N |
| MerlotReserve-Large | 11 | 71.5 | 644M | YT-Temporal-1B | Y |
| UNIMO+ERNIE | 12 | 71.4 | confidential（ensemble of 7 models），each maybe 355 M | BookWiki, OpenWebText, COCO, Visual Genome, Conceptual Captions, SBU Captions | - |

Finding fair comparisons among solutions on the VCR leaderboard can be challenging due to the confidentiality of most submissions. Even in open-source projects, fair comparisons are often overlooked due to the use of different technical approaches, data, training methods, and even model ensembles. In this diverse landscape, our method, GPT4RoI, stands out by utilizing fully open-source region-level data and surpassing all previous approaches by a significant margin (over 6 points) using just a single model. We believe that our method, as the pioneering usage of LLM for the VCR task with a completely open-source pipeline, establishes a robust new baseline.

Reviewer comment (Experiments):
Comparisons to other works on the same parameter level.

Author response:
(3). Lacks a comparison of parameters in Table.6.

The table below presents the rank, and accuracy of Q-AR, along with the parameters and datasets of various solutions, from the most recent **[VCR leaderboard](https://visualcommonsense.com/leaderboard/)** update on November 11, 2023. We retained the top 6 methods and 2 methods that have reports.

| model | rank | Q-AR(%) | Parameters | Datasets | Open Source |
| --- | --- | --- | --- | --- | --- |
| GPT4RoI | 1 | 81.6 | 13B+ | 7 open-source region data | Y |
| HunYuan | 2 | 75.6 | confidential | confidential | N |
| Shopee | 3 | 74.4 | confidential(ensemble of 4 models) | confidential | N |
| KDDI Research and SNAP | 4 | 74.3 | confidential | confidential | N |
| Kuaishou | 5 | 74.0 | confidential | confidential | N |
| VQA-GNN + MerlotReserve-Large | 6 | 74.0 | 1B+ (ensemble of 2 models) | YT-Temporal-1B | N |
| MerlotReserve-Large | 11 | 71.5 | 644M | YT-Temporal-1B | Y |
| UNIMO+ERNIE | 12 | 71.4 | confidential（ensemble of 7 models），each maybe 355 M | BookWiki, OpenWebText, COCO, Visual Genome, Conceptual Captions, SBU Captions | - |

Finding fair comparisons among solutions on the VCR leaderboard can be challenging due to the confidentiality of most submissions. Even in open-source projects, fair comparisons are often overlooked due to the use of different technical approaches, data, training methods, and even model ensembles. In this diverse landscape, our method, GPT4RoI, stands out by utilizing fully open-source region-level data and surpassing all previous approaches by a significant margin (over 6 points) using just a single model. We believe that our method, as the pioneering usage of LLM for the VCR task with a completely open-source pipeline, establishes a robust new baseline.

Reviewer comment (Experiments):
Finetuning on a specific dataset can lead to the case that the model forgets all other knowledge. For example, fine-tuning on the multichoice dataset will lead to the case that model can not speak out natural languages whatever you ask.

Author response:
(2). For Question 2, fine-tuning on a specific benchmark to do the evaluation

As pioneers in the field of region-understanding general-purpose models, there is currently no specific benchmark available for evaluating region-level dialogue in this context. Creating such an evaluation benchmark falls outside the scope of our research. Therefore, we have chosen to conduct our testing on traditional region understanding tasks, specifically multiple-choice questions. It is common practice in this field to fine-tune models on multiple-choice question tasks to evaluate performance. For instance, LLaVA [9] is fine-tuned for image-level understanding tasks, such as science QA [10].

Reviewer comment (Experiments):
From the qualitative examples, seems like the model can only produce short descriptions, which may not be suitable when answering with long context. I think the author's model may overfit to such datasets with short captions.

Author response:
(3). For Question 3, answering with long context

Not answering meaningful long context is a common issue for current large vision language models, since all open-source data only includes brief text descriptions. To mitigate this problem, LLaVa[1] and MiniGPT4[2] utilize ChatGPT/GPT4 to refine these short captions. However, we believe that this approach has limited significance. This is because the additional description is inferred by the LLM based on suspicion and often lacks accuracy or informative value. To achieve accurate and meaningful long context, the only way is to involve human participation in a data annotation pipeline to establish large datasets. We consider this to be beyond the scope of this paper.

Reviewer comment (Experiments):
When converting LLaVa instruct dataset into the format with bounding box, I cast doubt on how accurate it is. If many instances appear in the same image, is it ok to attach so many detection results beforehand? Besides, the LLaVa instruct dataset is known to include hallucination.

Author response:
(4). For Question 4, usage of LLaVA instruct dataset

As mentioned in the introduction, our approach involves detection results to leverage LLaVA data in order to enhance the multi-round conversation ability. This is because our other constructed datasets consist of only single or two-round conversations. To this end,  we do not need to specifically consider whether the detection results strictly correspond to the text in the LLaVA data. From the demonstration results, it can be observed that GPT4ROI has acquired the ability to engage in multi-round conversations.

Reviewer comment (Novelty):
How do authors claim on other methods which use textual coordinate as the grounding token? In this way, they can not only use bbox as input, but also use them in output.

Author response:
(5). For Question 5, About using textual coordinates as the grounding token.

This is an extremely valuable question that we have dedicated a lot of thought to. We are delighted to share these perspectives with you.

The key distinction lies in whether to incorporate the detection function into the LLM. For the method that uses textual coordinates as the grounding token, They have to solve the following challenge

- Aligning a large number of position tokens with their corresponding positions in the image by training on a large set of datasets. But this is actually a simple rule that can be naturally implemented with the operation in detection architectures.

- Modeling geometric properties can be challenging. For example,  if the ground truth box is  $<x_1=0, y_1=0, x_2=5, y_2=5>$, a predicted box of $<x_1=1, y_1=1, x_2=4, y_2=4>$ would be considered a better result than $<x_1=1, y_1=1, x_2=8, y_2=8>$. because it has a higher overlap with the ground truth. However, incorporating this geometric property into the next token prediction task using cross-entropy loss can be challenging. On the other hand, utilizing traditional loss functions such as L1 or IoU loss can naturally handle this geometric constraint.
- Dense to Sparse [1][4] is a crucial design for detection performance, but embedding such an idea into the sequential form of LLM is challenging.

We provide two pieces of evidence to support our argument

- The performance of pix2seq[2][3], which utilizes object365 pretrain, falls significantly behind the corresponding specialist [4][5].
- Even with scaled-up data and parameters, GPT4V still faces challenges in object counting[6]. However, this is a trivial task for detection methods.

Another approach is to use an external detector to find the potential region of interest, whereas LLM only focuses on analyzing the corresponding region of interest. This is the motivation of GPT4RoI. It requires much less data and allows for quick adaptation to specific domain problems with the corresponding detector. However, the drawback is that the framework may appear less elegant and it assumes input contains all region of interest that need to be analyzed.

Both approaches have their advantages and disadvantages, and academic research in both directions is thriving (including concurrent works or follow-ups on GPT4RoI). For the first approach, relevant references include [7, 8], while for the second approach, there are [9, 10] besides GPT4RoI. Additionally, there has been research that explores a fusion of the two approaches, as shown in references [11, 12, 13]. 

***It is still too early to make a definitive judgment on which approach has absolute superiority at this time.***

Reviewer comment (Theory):
For each region, it is compressed into one token? It is enough?

Author response:
(7). For Question 7, about number of tokens for each region

Our experiments found that increasing the number of tokens does not make a difference in these open-source datasets and corresponding benchmarks.