Paper: Visual Evidence Prompting Mitigates Hallucinations in Multimodal Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
The authors are recommended to provide the performance comparisons on object and relation hallucinations where only object labels in the input image are provided. If the performance is also good, please make explanations on the necessity of the proposed method.

Author response:
Thanks for the inspiring and interesting comments!

We conduct validation experiments using the detr-resnet-101 model to provide object labels as evidence for relation hallucination.

| Relation Hallucination | baseline | + object label |+ relation label |
| --- | ---| --- | --- |
| Qwen-VL-Chat | 63.62 | 71.41 | 75.68 |
| mPLUG-Owl| 62.58 | 66.88 | 68.46 |

The results show that providing object labels as evidence also has some improvement although not as effective as relation label. We suppose it is because object labels themselves contain crucial object information from the image, which leads to mitigating relation hallucination. This result not only validates the necessity of relation labels but also further verifies that our approach is orthogonal to the specific task. We have added the above discussions in the revision (Appendix A.4).

Reviewer comment (Experiments):
The authors are recommended to provide the hallucinations mitigations on open vocabulary objects and few-shot relations.

Author response:
Thanks for your insightful question. We conduct experiments in response to your valuable advice. The following discussions are added in the revision (Appendix A.5).

a) open vocabulary objects: We construct a new out-of-domain object hallucination dataset using the test sets from Object365 [A]. This dataset is divided into two parts. One part includes 80 objects that appear in COCO, while the other portion consists of objects that do not appear in COCO. The performance of these two parts are shown in the table below. It can be observed that there is a consistent improvement in performance for both in-domain and out-of-domain object categories.
| Model |  | In-domain objects | Out-of-domain objects |
| --- | --- | --- | --- |
| mPLUG-owl | baseline | 58.68 | 48.45 |
|  | **+ visual evidence** | **65.38** | **60.87** |
| Qwen-VL-Chat | baseline | 74.64 | 67.87 |
|  | **+ visual evidence** | **79.77** | **75.10** |

b) few-shot relation: We chose the bottom-10 tail relations as defined in [C] of VG to construct a medium-sized relation hallucination dataset with 1006 samples. We used OpenPSG as the SGG model and conducted experiments on Qwen-VL-Chat and mPLUG-Owl. The experiment results are shown in the table below, and it can be seen that our framework still achieves significant improvements in few-shot relations.

| Model | | Accuracy (%) |
| --- | --- | --- |
| mPLUG-Owl | baseline | 61.23 |
|  | **+ visual evidence** | **67.89** |
| Qwen-VL-Chat | baseline | 55.67 |
|  | **+ visual evidence** | **68.63** |

> [A] Objects365: A Large-scale, High-quality Dataset for Object Detection  
> [B] Simple Open-Vocabulary Object Detection with Vision Transformers  
> [C] Learning from the Scene and Borrowing from the Rich: Tackling the Long Tail in Scene Graph Generation

Reviewer comment (Evaluation):
Make explanations on the decline of 'at' or slight improvement on 'under', displayed in Figure 4.

Author response:
Thanks for your careful review. We conducted statistical analysis on the samples of these two relations. It is observed that "at" is naturally ambiguous with "on", "next to", and "in", and "under" is ambiguous with "below" and "around". We will carefully review the collected data to minize the confusions.

Reviewer comment (Evaluation):
Make necessary analysis for situations where the performance of RelTR on SGG is significantly higher than motifs, while the performance of hallucination mitigations on LVLMs is equal or even opposite, displayed as in Table 9.

Author response:
Thank you for your meticulous review. In the Table 9, different scene graph generation models (RelTR, MOTIFS and OpenPSG) have comparable improvements on mPLUG-Owl and Qwen-VL-Chat. For example, RelTR achieves 5.92% and MOTIFS achieves 5.8% improvement on mPLUG-Owl. RelTR achieves 11.35% and MOTIFS achieves 12.55% improvement on Qwen-VL-Chat. The gains brought by different scene graph generation models to LVLM are within a stable range (saturated).

| LVLM | Small model | mAP | Accuracy (%) |
| --- | --- | --- | --- |
| mPLUG-owl | baseline | - | 62.58 |
|  | RelTR | 18.9 |  68.50 (+5.92) |
|  | MOTIFS | 20.0 | 68.38 (+5.8) |
|  | OpenPSG | 28.4 | 68.25 (5.67)|
| Qwen-VL-Chat | baseline | - | 63.62 |
|  | RelTR | 18.9 | 74.97 (+11.35) |
|  | MOTIFS | 20.0 | 75.80 (+12.18) |
|  | OpenPSG | 28.4 | 76.17 (+12.55) |

Reviewer comment (Evaluation):
Have the authors explored questions regarding how overlapped between objects in questions and objects in evaluation datasets' questions?

Author response:
Thanks for the inspiring and interesting comments! 

a) We calculate the current stats of the overlap using detr-resnet-101. In the 3,000 visual evidence prompts, there are 298 prompts that contains object that are not in the question (Type A), and 1,202 prompts that contain objects that are not in the questions exclusively (Type B).


|  | LVLM right->right | LVLM right->wrong | LVLM wrong->right | LVLM wrong->wrong |
| -------- | -------- | -------- | -------- | -------- |
| Type A | 139 (46.7%)     | 8 (2.7%)    | 110 (36.9%)     | 41  (13.8%)   |
| Type B | 415 (34.5%)     | 22 (1.8%)    | 563 (46.8%)     | 202  (16.8%)   |

b) Following Figure 3 in the draft, we calculate the stats of samples which were initally answered corrently/wrongly and answer correctly/wrongly after provided with Type A/B prompts. In the 298 Type A prompts, 110 of which (36.9%) allievates the hallucination of LVLM with detr-resnet-101 on Qwen-VL-Chat. In the 1,202 Type B prompts, 563 of which (46.8%) allievates the hallucination of LVLM.

c) With the help of ChatGPT, we manually change the object appear in question to its synonyms respectively. The evaluation of object hallucination slightly decreases from 87.70% to 86.53% on Qwen-VL-Chat and from 78.38% to 71.54% on mPLUG-Owl, but there is still a non-trival improvement over the baseline especially on mPLUG-Owl, the results are shown in the table below.
| Model | Setting | Accuracy (%) |
| ---| --- | --- |
| mPLUG-Owl | baseline | 57.29 |
|  | before switching to synonyms | 78.38 |
|  | after switching to synonyms | **71.54** |
| Qwen-VL-Chat | baseline | 81.23 |
|  | before switching to synonyms | 87.70 |
|  | after switching to synonyms | **86.53** |


In summary, with the LVLM's intrinsic robust ability to understand the output visual evidence of small models, visual evidence prompting is also robust in terms of the specific words used for objects in the given problem. We have add the above discussions in the revision (Appendix A.3).

Reviewer comment (Evaluation):
What is the current stats on how overlapping between objects in questions and objects in evaluation datasets' questions?

Author response:
Thanks for the inspiring and interesting comments! 

a) We calculate the current stats of the overlap using detr-resnet-101. In the 3,000 visual evidence prompts, there are 298 prompts that contains object that are not in the question (Type A), and 1,202 prompts that contain objects that are not in the questions exclusively (Type B).


|  | LVLM right->right | LVLM right->wrong | LVLM wrong->right | LVLM wrong->wrong |
| -------- | -------- | -------- | -------- | -------- |
| Type A | 139 (46.7%)     | 8 (2.7%)    | 110 (36.9%)     | 41  (13.8%)   |
| Type B | 415 (34.5%)     | 22 (1.8%)    | 563 (46.8%)     | 202  (16.8%)   |

b) Following Figure 3 in the draft, we calculate the stats of samples which were initally answered corrently/wrongly and answer correctly/wrongly after provided with Type A/B prompts. In the 298 Type A prompts, 110 of which (36.9%) allievates the hallucination of LVLM with detr-resnet-101 on Qwen-VL-Chat. In the 1,202 Type B prompts, 563 of which (46.8%) allievates the hallucination of LVLM.

c) With the help of ChatGPT, we manually change the object appear in question to its synonyms respectively. The evaluation of object hallucination slightly decreases from 87.70% to 86.53% on Qwen-VL-Chat and from 78.38% to 71.54% on mPLUG-Owl, but there is still a non-trival improvement over the baseline especially on mPLUG-Owl, the results are shown in the table below.
| Model | Setting | Accuracy (%) |
| ---| --- | --- |
| mPLUG-Owl | baseline | 57.29 |
|  | before switching to synonyms | 78.38 |
|  | after switching to synonyms | **71.54** |
| Qwen-VL-Chat | baseline | 81.23 |
|  | before switching to synonyms | 87.70 |
|  | after switching to synonyms | **86.53** |


In summary, with the LVLM's intrinsic robust ability to understand the output visual evidence of small models, visual evidence prompting is also robust in terms of the specific words used for objects in the given problem. We have add the above discussions in the revision (Appendix A.3).

Reviewer comment (Experiments):
What if visual evidence prompt contains objects that are not in the question? What if the prompt contains objects that are not in the questions exclusively?

Author response:
Thanks for the inspiring and interesting comments! 

a) We calculate the current stats of the overlap using detr-resnet-101. In the 3,000 visual evidence prompts, there are 298 prompts that contains object that are not in the question (Type A), and 1,202 prompts that contain objects that are not in the questions exclusively (Type B).


|  | LVLM right->right | LVLM right->wrong | LVLM wrong->right | LVLM wrong->wrong |
| -------- | -------- | -------- | -------- | -------- |
| Type A | 139 (46.7%)     | 8 (2.7%)    | 110 (36.9%)     | 41  (13.8%)   |
| Type B | 415 (34.5%)     | 22 (1.8%)    | 563 (46.8%)     | 202  (16.8%)   |

b) Following Figure 3 in the draft, we calculate the stats of samples which were initally answered corrently/wrongly and answer correctly/wrongly after provided with Type A/B prompts. In the 298 Type A prompts, 110 of which (36.9%) allievates the hallucination of LVLM with detr-resnet-101 on Qwen-VL-Chat. In the 1,202 Type B prompts, 563 of which (46.8%) allievates the hallucination of LVLM.

c) With the help of ChatGPT, we manually change the object appear in question to its synonyms respectively. The evaluation of object hallucination slightly decreases from 87.70% to 86.53% on Qwen-VL-Chat and from 78.38% to 71.54% on mPLUG-Owl, but there is still a non-trival improvement over the baseline especially on mPLUG-Owl, the results are shown in the table below.
| Model | Setting | Accuracy (%) |
| ---| --- | --- |
| mPLUG-Owl | baseline | 57.29 |
|  | before switching to synonyms | 78.38 |
|  | after switching to synonyms | **71.54** |
| Qwen-VL-Chat | baseline | 81.23 |
|  | before switching to synonyms | 87.70 |
|  | after switching to synonyms | **86.53** |


In summary, with the LVLM's intrinsic robust ability to understand the output visual evidence of small models, visual evidence prompting is also robust in terms of the specific words used for objects in the given problem. We have add the above discussions in the revision (Appendix A.3).

Reviewer comment (Reproducibility):
Another question is regarding the pixel locations of detected objects — does it matter to change the scale of the image? For example, `cup <0, 284, 133, 424>` to `<0 /width, 284 / height, 133 / width, 424 / height>`, or to `<0 /width * 2, 284 / height * 2, 133 / width * 2, 424 / height * 2>`?

Author response:
Thanks for your careful review. In Qwen-VL-Chat, for any given bounding box, a normalization process is applied (within the range [0, 1000)). 
For example, x_input = x_pixel / width * 1000, y_input = y_pixel / height * 1000. So the scale of the image does not differentiate any object relations. We clarify this setting in Sec. 4.1.3 in the revision.

Reviewer comment (Evaluation):
Efficiency: What degree of efficiency degradation can be expected with the introduction of new mini models for object detection and scene graph generation into the original method?

Author response:
Thanks for your constructive comments. We compute the average inference time of several small models on one A100 GPU.

| yolos-small | detr-resnet-101 | RelTR | OpenPSG |
| -------- | -------- | -------- | -------- |
|0.193s     | 0.163s     | 0.394s     |0.218s|

Meanwhile, the average time for one inference of Qwen-VL-Chat and mPLUG-Owl is:

| Qwen-VL-Chat | mPLUG-Owl |
| -------- | -------- |
| 1.077s    | 1.157s     |

After incorporating the small models, the relative increase in inference time is an average of 21.78%. Moreover, in multi-turn dialogue scenarios, the image is only uploaded once, regardless of how many questions are asked. The small model only needs to perform inference once, making the cost of inference relatively smaller.

Reviewer comment (Experiments):
Domain Compatibility: Is there any overlap between the datasets used for training the small target detection and scene graph generation models and the LVLM hallucination test set?

Author response:
a) Thanks for pointing out this potentially confusing point. There is no overlap between the datasets used for training small models and the LVLM hallucination test sets. We are using the open-source small models trained on the COCO  or Visual Genome training set. The test set we have constructed is derived from the COCO validation set or Visual Genome test set. We have clarified this in the revision in Sec. 4.1.1.

Reviewer comment (Evaluation):
In the context of an open-world scenario where the small model might struggle to efficiently capture target and relation information, to what extent will the acquisition of false evidence affect the LVLM's otherwise accurate responses? It's worth noting that despite the findings presented in Figure 3, my skepticism regarding this issue persists.

Author response:
b) Thanks for your insightful question. We conduct more evaluations and incorporate the discussions into our revision in Appendix A.1. 
* Firstly, we collect 2,540 additional samples from open-world datasets and scenarios to further quantitatively evaluate the genaliation ability of our method. More details of these two out-of-domain datasets Object365 [A] and OpenImage [B]are added in the revision (Appendix A.1). In 20.6% of the images, small model captures incorrect or partial correct object or relation information. With these visual evidences, **only 8%** of the false evidence confuse the LVLM and change the response from collect to wrong. 
* Secondly, besides Figure 3, we follow CLIP and select 2 samples from 10 open-world datasets (without groundtruth) for qualititive analysis. These 20 cases are in the revised Appendix F.1. From these results and cases, it can be seen that in **open-world scenarios, incorrect evidence still has limited impact on cases that are already correct**. 
* Thirdly, The fifth prompt template in Table 3 in the draft shows that we can tell the LVLM that *the evidence might be wrong and keep your answer if you think the evidence is wrong or missing.*

In summary, quantitative and qualititive analysis show that false evidence has limited affect on LVLM's accurate responses with the help of carefully designed prompt.

> [A] Objects365: A Large-scale, High-quality Dataset for Object Detection  
> [B] The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale