Paper: Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning (ICLR-2024)

Reviewer comment (Evaluation):
Another useful experiment would be to report the correlation of GPT-4 ratings with human ratings. The authors appear to already have a dataset with expert raters, albeit on a different scale, so such a correlation should be easy to report.

Author response:
**GAVIE Aligns well with Human Experts**

To assess the correlation of GPT-4 rating with the human rating, we hired three experts specializing in NLP to perform the human assessment. From Tab. 4 in the paper, all experts agree that the output from our model is the best, followed by InstructBLIP in the second place, and MMGPT performs the worst. This ranking aligns with our GAVIE. More details are shown in the A.1.1 (appendix).

Reviewer comment (Evaluation):
For GAVIE, the scores are on a 0–10 scale. The intuition for using such a large scale is unclear, and the reviewer feels the task could be summarised to a 0/1 score. Given the analysis in “Is GPT4-Assisted Evaluation Stable?”, changing the benchmark to a 0–4 scale could be more appropriate.

Author response:
**Why GAVIE Uses Scores 0-10 instead of 0/1.**

In an open-ended setting, the model’s answer to a question may be completely wrong, partly wrong, or entirely correct. With a score of 0-10, GAVIE can evaluate the model’s answer and produce a detailed level of correctness instead of only 0/1(correct/wrong), which does not give differentiable scores to answers that are partly wrong or completely wrong (i.e., both will receive a score of 0). However, our GAVIE can assign a score of 0 for the completely wrong answer and a partial score (e.g., 5) for the partly wrong answer. Therefore,  GAVIE can not only give a more accurate ranking among compared models but also help people understand the level of correctness for answers from different models. The relationship between correctness and scores in GAVIE is summarized as follows: 

RELEVANCY has four grade levels: (1) The response is entirely relevant (9-10), (2) The response is mostly relevant (6-8), (3) The response is partly relevant (3-5), (4) The response is seldom relevant (0-2). ACCURACY has four grade levels: (1) The response is completely accurate (9-10), (2) The response has minor errors (6-8), (3) The response is partly accurate (3-5), (4) The response is mostly or completely wrong (0-2).

Reviewer comment (Evaluation):
The reviewer does not understand the last sentence in the cited part:  
"Alternatively, (Li et al., 2023c; Fu et al., 2023) formulate the evaluation of hallucination as a binary classification task that prompts LMM to output 'Yes' or 'No'. However, it is hard to evaluate the LMM output in an open-ended manner."  
They question why the evaluation would still be challenging with a yes/no question, as this is a straightforward classification task that does not need human or parametric intervention.

Author response:
**GAVIE Can Evaluate Open-ended Questions:**

POPE [8] formulates the evaluation of hallucination as a binary classification task that prompts LMM to answer with "Yes" or "No".  However, if the question is a multiple choice question (e.g., “Choose the correct statement about the weather conditions in the image: (a) Cloudy and rainy, (b) Clear blue sky, (c) Foggy and misty, (d) Snowy and cold”) or  an open-ended question (e.g., “What objects are on the toddler’s feet?”), POPE can not evaluate model hallucinations on these questions as the answer space is much more complicated.  In contrast, our proposed GAVIE can evaluate model hallucinations even when the answers are open-ended manners.  (Fig 21-Fig 26 in the appendix)

Reviewer comment (Presentation):
There are three incorrect examples in Figure 2 (neg). For example, the question is about "Merkel" but the answer is about "Hilary Clinton."

Author response:
**Incorrect examples in Figure 2 (neg):**

The question mentioned "Merkel" is not incorrect but a negative instruction/question. That is, "Merkel" doesn’t appear in the image. Therefore, the answer should be, "No, Hillary Clinton arrived at the Los Angeles Get Out The Vote Rally in the image.” We use this kind of negative samples to finetune Large Multimodal Models (LMMs), so that these models can learn to say “no” if the objects mentioned in the questions do not appear in the image, hence reducing hallucinations.

Reviewer comment (Novelty):
As cited in the manuscript, (Liu 2023d) proposed one of the first GPT-based automatic evaluations without human intervention. The manuscript does not explain the main differences from (Liu 2023d), except the application to vision - language domain.

Author response:
**Difference with G-Eval [5]**

The difference between our evaluation method (GAVIE) and G-eval is as follows:

(1) Unlike G-Eval [5], which mainly focuses on text summarization and dialogue tasks, our GAVIE method can adapt to diverse instruction formats and tasks. To demonstrate the effectiveness of GAVIE, we compare GAVIE and human expert evaluation on our evaluation set, which includes 16 vision and language (VL) tasks. From Tab 4, we found that GAVIE aligns well with human evaluation. 

(2)Unlike G-Eval [5], which only generates one score for each sample according to its evaluation form, our GAVIE generates different scores for different criteria:  (a) Relevancy: whether the response directly follows the instruction; (b) Accuracy: whether the response is accurate concerning the image content. Additionally, GAVIE also provides explanations to support the scores. In this way, people can clearly understand how the models perform in terms of different criteria and why GAVIE gives this score.

Reviewer comment (Reproducibility):
GAVIE relies on the black-box GPT-4 engine. It means the evaluation is affected by the system update of the GPT-4, implying we cannot assure the loyal reproducing of GAVIE results in months/years after. Are there any remedies for this problem?

Author response:
**Blackbox GPT-4:**

To reproduce the results of GAVIE, we indicate the GPT4 version ( GPT4-32k-0314) we used in Section 5. We also show some examples in Fig 34 and Fig 35  by replacing GPT4 with current popular open-sourced Large Multimodal Models (LMMs), including LLaVA1.5-13B [1] and MiniGPT4-v2 [2]. From Fig 34, we can see that LLaVA1.5-13B can follow the instructions to evaluate the answers according to two criteria, but it fails to detect that the answers are incorrect, given the image content. From Fig 35, we can see that MiniGPT4-v2-13B can not follow the instructions to generate a score for each criterion.  What’s more, the score MiniGPT4-v2-13B  generates is not reasonable. Therefore, current open-sourced LMMs like LaVA1.5-13B and MiniGPT4-v2 may struggle when evaluating hallucination in vision-and-language tasks.

We come up with two possible remedies, which are promising directions to explore in future works:

(1) One can finetune LLaVA1.5 on the vision and language evaluation tasks to enhance its instruction following abilities. Additionally, to further enhance LLaVA1.5, one can also replace the vision encoders with [6] to improve its visual perception ability to make it a better evaluator.

(2) One can explore LLM-free methods for VL hallucination evaluation. For example,  we can design questions with answers from a closed set (e.g., binary classification) based on different types of hallucinations. For object hallucination, we can construct the question as “Is there a {hallucination object} in this image?”.  For attribute hallucination, we can design questions like “​​Is the {object} {state} in this image?”, “Does the {object} {action} in this image?”, “Is the {object} {color} in this image?”. For relation hallucination, we can construct questions like “Is there a direct contact between the {object 1} and {object 2} in this image?”.  For all the cases described above, the groundtruth answers to these questions will be “yes” or “no”. In this way, we can evaluate the output of LMMs without GPT4.

Reviewer comment (Experiments):
Concerning the 2nd point of the "weakness", I wonder what happens to GAVIE if we replace the engine from GPT-4 to (possibly weaker) multi-modal LLMs.

Author response:
**Blackbox GPT-4:**

To reproduce the results of GAVIE, we indicate the GPT4 version ( GPT4-32k-0314) we used in Section 5. We also show some examples in Fig 34 and Fig 35  by replacing GPT4 with current popular open-sourced Large Multimodal Models (LMMs), including LLaVA1.5-13B [1] and MiniGPT4-v2 [2]. From Fig 34, we can see that LLaVA1.5-13B can follow the instructions to evaluate the answers according to two criteria, but it fails to detect that the answers are incorrect, given the image content. From Fig 35, we can see that MiniGPT4-v2-13B can not follow the instructions to generate a score for each criterion.  What’s more, the score MiniGPT4-v2-13B  generates is not reasonable. Therefore, current open-sourced LMMs like LaVA1.5-13B and MiniGPT4-v2 may struggle when evaluating hallucination in vision-and-language tasks.

We come up with two possible remedies, which are promising directions to explore in future works:

(1) One can finetune LLaVA1.5 on the vision and language evaluation tasks to enhance its instruction following abilities. Additionally, to further enhance LLaVA1.5, one can also replace the vision encoders with [6] to improve its visual perception ability to make it a better evaluator.

(2) One can explore LLM-free methods for VL hallucination evaluation. For example,  we can design questions with answers from a closed set (e.g., binary classification) based on different types of hallucinations. For object hallucination, we can construct the question as “Is there a {hallucination object} in this image?”.  For attribute hallucination, we can design questions like “​​Is the {object} {state} in this image?”, “Does the {object} {action} in this image?”, “Is the {object} {color} in this image?”. For relation hallucination, we can construct questions like “Is there a direct contact between the {object 1} and {object 2} in this image?”.  For all the cases described above, the groundtruth answers to these questions will be “yes” or “no”. In this way, we can evaluate the output of LMMs without GPT4.

Reviewer comment (Evaluation):
One of the evaluations is "Relevancy: whether the response directly follows the instruction." This actually evaluates the instruction following ability, which is kind of tricky, cause it's hard to say if the training data difference brings instruction domain difference, which makes the evaluation not fair.

Author response:
**Relevancy Score:**

To make a fair comparison, we compare our model with LLaVA1.5 and MiniGPT4-v2, which both use the Visual Genome as their training data to address the domain difference.  From our evaluation set (GAVIE),  our model outperforms both LLaVA1.5-7B and MiniGPT4-v2-7B regarding the instruction-following ability.
|  Method   | GAVIE-Relevancy  |
|  ----  | ----  |
| LLaVA 1.5-7B [1] | 8.20 |
| MiniGPT4-v2-7B [2] | 8.10 |
| Ours-7B | 8.46 |

Reviewer comment (Evaluation):
Human label with 4 scores. It would be great to give some explanation on what kind of aspect humans evaluate and preferred means it will be better if the author could provide a more specific evaluation policy and rules.

Author response:
**Human Evaluation Policy:**

The details of human evaluation guidelines are shown below:

As for each question, there are an instruction, an image, and several answers. Suppose you are a smart teacher, please score the answers according to the two criteria. (1) Accuracy: whether the response is accurate concerning the image content. (2) Relevancy: whether the response directly follows the instruction without unrelated answers. There are four options for the scores:
A score of 1 ( Very Poor): The response is seldom relevant to the instruction and is mostly wrong according to the image.
A score of 2 ( Poor): The response is partly relevant to the instruction and partly accurate according to the image.
A score of 3 (Good): The response is mostly relevant to the instruction and has minor errors according to the image.
A score of 4 (Excellent): The response is completely relevant to the instruction and completely accurate according to the image.