Paper: MiniGPT-v2: Large Language Model as a Unified Interface for Vision-Language Multi-task Learning (ICLR-2024)

Reviewer comment (Experiments):
For Table 5, better to evaluate performance for other task identifiers to show the effectiveness of this.

Author response:
**Q1 More performance evaluation for other task identifiers**

We also provide the ablation results on [refer] identifiers. The results are demonstrated in the following table, and it can be observed that with the [refer] task identifier can overall improve the model performance on RefCOCO/+/g by 1.3 acc, indicating the efficiency of using [refer] task identifier. 

| Model                              	| RefCOCO (test-B) | RefCOCO+ (test-B) | RefCOCOg (test) | average |
|----------------------------------------|:----------------:|:-----------------:|:---------------:|:-------:|
| Ours w/o [refer] task identifier   	|   	73.2   	|    	59.7   	|   	72.8  	|  68.6   |
| Ours w/ [refer]                    	|   	74.7   	|    	60.9   	|   	74.2  	|  69.9   |

Reviewer comment (Evaluation):
Discussion is a bit weak, for example, the paper should have some discussion on the error analysis and some detail on the strengths and weaknesses.

Author response:
**Q2 Discussion is a bit weak, for example, the paper should have some discussion on the error analysis and some detail on the strengths and weaknesses.**


Thank you for your constructive feedback. In light of your comments, we have revised our paper to provide a more robust discussion. It includes the discussion of strengths and weaknesses including the discussion regarding the training efficiency, robustness, and hallucination.

Reviewer comment (Evaluation):
Should add more explanation of the evaluation metrics.

Author response:
**Q3 Explanation of the evaluation metrics and dataset.**
We add a more detailed explanation of the evaluation metrics and the dataset in the appendix. This consists of the evaluation metrics for visual question answering, RefCOCO, and hallucination. It also provides a rich explanation of the dataset.

Reviewer comment (Experiments):
Should add more explanation of the dataset, like what is the difference between GQA and VQA-v2, what are the sizes of them?

Author response:
**Q3 Explanation of the evaluation metrics and dataset.**
We add a more detailed explanation of the evaluation metrics and the dataset in the appendix. This consists of the evaluation metrics for visual question answering, RefCOCO, and hallucination. It also provides a rich explanation of the dataset.

Reviewer comment (Experiments):
The model is trained with various tasks, but only evaluation on a subset of the tasks.

Author response:
**Q4 Evaluation on a subset of the tasks**

In our study, we mainly focused our evaluation on the tasks of visual question answering and referring expression comprehension. This decision was driven by two key considerations:

1. Representative Tasks: These two tasks are quite essential in evaluating the foundational capabilities of vision-language models, specifically in visual knowledge querying and visual grounding. 

2. Benchmark Availability: The tasks such as detailed image captioning, detailed grounded image captioning, and object parsing and grounding, etc, lack well-established benchmarks.  It's important to note that the capabilities demonstrated in those tasks, being only emerged in the large language models. Hence it lacks standardized evaluation metrics to evaluate their performance. Therefore, we focus on the well-established benchmarks such as various VQA and RefCOCO

Reviewer comment (Evaluation):
Can you explain why your model does not perform well on RefCOCO+ in Table 4?

Author response:
**Q6 Explain that your model does not perform well on RefCOCO+ in Table 4.**

Here are the dataset features of RefCOCO, RefCOCO+, and RefCOCO(g):

1. RefCOCO has expressions that are generally simpler and may include location-based descriptions.
2. RefCOCO+ is designed to exclude location-based expressions. The referring expressions here are more complex, focusing on describing the appearance of objects rather than their location.
3. RefCOCO(g) has the referring expressions that focus on more general and longer descriptions.


As we can see from Table 4, the performance of MiniGPT-v2 on RefCOCO+ also performs well and can outperform Shikra on Test-B split. The results in Table 4 indicate that the general performance of MiniGPT-v2 on RefCOCO+ does not perform as well as RefCOCO and RefCOCO(g). From the difference among RefCOCO, RefCOCO+, and RefCOCO(g), we hypothesize that our model performs better on RefCOCO and RefCOCO(g) because our model is stronger in processing location-based referring expressions and longer descriptions compared to other baseline models, which can contribute to our model design and training strategy.

Reviewer comment (Evaluation):
Better to give more illustration on the evaluation metrics on hallucination-CHAIR. How does this calculate? What do CHAIR_I, CHAIR_S and Len mean? How to interpret the result for three different prompts in MiniGPT-v2?

Author response:
**Q7 CHAIR hallucination metrics and three different prompts evaluation in MiniGPT-v2.**


In our study, we employ the evaluation metrics for hallucination as defined in [1] to assess the performance of MiniGPT-v2. Specifically, we use CHAIR_i and CHAIR_s metrics to quantify hallucination at different levels:

CHAIR_i (Instance-level Hallucination): This metric evaluates the degree of hallucination at the object instance level. It is calculated as the ratio of hallucinated objects to the total number of objects mentioned in the caption. Mathematically, it is expressed as:

$$ CHIAR_i = \frac{| \{\text{hallucinated objects}\} |}{| \{\text{all mentioned objects}\} |} $$



CHAIR_s (Sentence-level Hallucination): This metric assesses hallucination at the sentence level. It is the ratio of captions containing hallucinated objects to the total number of captions generated. It is given by:
$$ CHAIR_s = \frac{| \{\text{captions with hallucinated objects}\} |}{| \{\text{all captions}\} |} $$



​​Len (Length of Caption): This refers to the number of words in the generated caption, providing a measure of caption verbosity or conciseness.

By adding the different task identifiers, we observe that MiniGPT-v2 can generate the caption with different granularity, which is also the unique advantage of MiniGPT-v2 compared with other models.

1. Without Task Identifier: MiniGPT-v2 tends to generate captions with rich details
2. With [grounding] Identifier: The model produces captions more grounded to the image objects. However, this mode exhibits a reduction in the description of visual attributes and relationships compared to the mode without any task identifier.
3. With [caption] Identifier: This tends to generate very short image caption.

[1] Li, etc. Evaluating Object Hallucination in Large Vision-Language Models.

Reviewer comment (Evaluation):
What are the evaluation metrics for Table 3, Table 4, and Table 5?

Author response:
**Q3 Explanation of the evaluation metrics and dataset.**
We add a more detailed explanation of the evaluation metrics and the dataset in the appendix. This consists of the evaluation metrics for visual question answering, RefCOCO, and hallucination. It also provides a rich explanation of the dataset.

Reviewer comment (Presentation):
“More qualitative results can be found in the Appendix“ - but I did not see the Appendix.

Author response:
**Q8 Qualitative results in the appendix**
Thank you for pointing out the discrepancy regarding the location of our qualitative results. In our initial submission, these results were indeed included in as a separate file in the supplementary materials. However, for easier accessibility and reference, we have relocated them to the appendix in our revised manuscript.

Reviewer comment (Experiments):
How was the “concatenate four adjacent visual output tokens” determined? Why four? Have the authors tried making the visual features even smaller (concatenate adjacent 8/16/32 etc)? Any discussions or ablations on this?

Author response:
**Q1 Making visual tokens even smaller (8 and 16 tokens):**

We perform the ablation with concatenating more adjacent visual output tokens, e.g. 8 and 16. We provide the results in the following table. Our findings reveal that with merging more tokens, visual question answering and visual grounding performance can also drop. It's particularly noteworthy that the impact on visual grounding is more pronounced, which might because merging more tokens make it lose spatial understanding. 

| Model               	| OKVQA |  GQA  | vizwiz |  VSR  | IconVQA |  HM   | average | 
|-------------------------|:-----:|:-----:|:------:|:-----:|:-------:|:-----:|:-------:|
| Concatenate 4 tokens	| 52.1  | 54.6  |  29.4  | 59.9  |  45.6   | 57.4  |  49.8   | 
| Concatenate 8 tokens	| 50.3  | 53.9  |  29.1  | 55.6  |  45.9   | 57.1  |  48.6   |  	
| Concatenate 16 tokens   | 50.5  | 52.5  |  26.7  | 55.4  |  45.8   | 57.2  |  48.0   | 


| Model               	| RefCOCO (testB) | RefCOCO+ (testB) | RefCOCOg (test) | average |
|-------------------------|:---------------:|:----------------:|:---------------:|:-------:|
| Concatenate 4 tokens	|  	74.7   	|   	60.9   	|   	74.2  	|  69.9   |
| Concatenate 8 tokens	|  	70.3   	|   	57.4   	|   	71.1  	|  66.3   |
| Concatenate 16 tokens   |  	66.3   	|   	52.4   	|   	64.4  	|  61.0   |

Reviewer comment (Experiments):
The authors claim to use image with higher resolution (448x448). While this sounds like a intuitive motivation, it lacks solid discussion to support such a claim. How does it compare to the previous traditional setups of 224x224 or 336x336? Does using 448x448 images and then concatenating 4 adjacent visual features outperform directly using 224x224 images?

Author response:
**Q2 Comparing with training on the 224x224 image resolution**

We evaluate the model with training 224x224 images (without any token merging), and we demonstrate the results in the following table, 

| Image Size  | OKVQA |  GQA  | vizwiz |  VSR  | IconVQA |  HM   | average |
|--------|:-----:|:-----:|:------:|:-----:|:-------:|:-----:|:-------:|
| 224x224| 52.0  | 53.9  |  30.2  | 58.6  |  46.5   | 56.2  |  49.6   |
| 448x448| 52.1  | 54.6  |  29.4  | 59.9  |  45.6   | 57.4  |  49.8   |

| Image Size | RefCOCO (testB) | RefCOCO+ (testB) | RefCOCOg (test) | average |
|------------|:---------------:|:----------------:|:---------------:|:-------:|
| 224x224	|  	63.3   	|   	52.6   	|   	63.1  	|  59.7   |
| 448x448	|  	74.7   	|   	60.9   	|   	74.2  	|  69.9   |

From the results shown in the table, we can see that training on 448x448 images has similar performance on visual question answering, but it has much higher (+10.2 acc) visual grounding performance than training on 224x224 images.

Reviewer comment (Experiments):
How did the authors select the baseline models for each task? Why weren’t the vision-and-language foundation models listed in Table 6 being compared in Table 3 and Table 4? For instance, why Minigpt4 appeared in Table3, but mplug-owl didn’t? The selection of baseline models for each task seems pretty random, and the authors lack an explanation for such an experimental design.

Author response:
**Q3 Random Baseline selection**

Thank you for your question regarding the selection of baseline models. Here are the key points addressing your concerns:

Table 4: Our choice of baseline models for Table 4 aligns with the evaluation benchmarks established  in the work [1]. This decision was made to ensure a direct and consistent comparison with the existing works

Table 6: We incorporated evaluation results and baselines from the work [2]. This is to make it consistent with previous established  evaluation benchmarks.

missing mPLUG-Owl baseline: ​​We realize it important to compare with mPLUG-Owl baseline and have included it in our updated paper (Table 3).

[1] Chen, etc.. Shikra: Unleashing multi-modal llm’s referential dialogue magic.
[2] Li, etc. Evaluating Object Hallucination in Large Vision-Language Models.

Reviewer comment (Experiments):
Another missing baseline on vision-and-language instruction following is “MIMIC-IT: Multi-Modal In-Context Instruction Tuning”.

Author response:
**Q4 Missing MIMIC-IT baselines.**

Thank you for pointing out this work. MIMIC-IT is a dataset with 2.8 million multi-modal instruction-response pairs, predominantly fine-tuned on the Otter model. Acknowledging its importance, we have revised our paper (in the table 3) to include a comparison with the Otter model. And the results demonstrate that our model consistently outperforms the Otter model in the VQA performance.

Reviewer comment (Writing):
In Section 3.2, what does it mean for “
 is considered as the user role, and 
 is considered as the assistant role”? What is the user role and what is the assistant role? Does it mean that the “
” token will be followed by user input, and the “
” will be followed by the assistant’s output?

Author response:
**Q5 Explain 
 vs 
**

In Section 3.2, “
” and “
” are critical markers within the LLaMA-2 language model. “
” is used to indicate the start of user input, defining the user role in the interaction. This token signals that the following text is a user-generated query or statement.

On the other hand, “
” marks the beginning of the assistant's response, representing the assistant role. This token signals that the following text is a user-generated query or statement.

Reviewer comment (Reproducibility):
Were the task identifier tokens and the spatial location tokens (<0>, <1>, …<100>) extended to the vocabulary as special tokens? If yes, how were the newly added tokens’ embeddings initialized?

Author response:
**Q6 Task identifier and spatial location tokens representation**

In our research, we did not introduce any new tokens to the vocabulary. We exclusively utilized the original token vocabulary provided by LLaMA-2.