Paper: Aligning Large Multimodal Models with Factually Augmented RLHF (ICLR-2024)

Reviewer comment (Evaluation):
The authors proposed that the evaluation indicates that this benchmark dataset aligns well with human evaluations, especially when scores are adjusted for anti-hallucinations. I didn’t find how to get this conclusion in the experimental section or other sections.

Author response:
After designing the questions, we have tested LLaVA-13B and IDEFICS-80B on MMHal-Bench. We manually inspected 1) the answers produced by these two LMMs, and 2) their corresponding reviews given by GPT-4. When judging whether hallucinations exist in the LMM responses, we found that the agreement between human inspection and GPT-4 evaluation was 94% (Appendix C). It is worth noting that IDEFICS-80B was not involved when we designed the question-answer pairs, but the hallucination evaluation is still highly consistent between humans and GPT-4. These observations demonstrate that the automated GPT-4 evaluation in MMHal-Bench is well-aligned with humans.

Reviewer comment (Presentation):
The method section requires an overall overview of the relationships between the various parts of the method.

Author response:
We thank the reviewer for the insightful suggestion. We have added an overview paragraph in Section 2.

Reviewer comment (Experiments):
Are answers and questions in the MMHAL-BENCH dataset written by humans or constructed by automatic methods?

Author response:
The questions and answers are created by humans. As introduced in Appendix C, we have selected 8 question categories $\times$ 12 object topics, and manually designed questions where prior LMMs usually make false claims about image contents (hallucinations).

Reviewer comment (Evaluation):
I could NOT find direct evidence that the proposed method reduces hallucinations and mitigates reward hacking. As shown in Table 6, LLaVA-RLHF did not significantly differ from LLaVA-SFT in the hallucination rate. LLaVA-RLHF even falls behind LLaVA-SFT in the 13B model. More convincing evidence or a detailed analysis is required that Fact-RLHF truly improves hallucinations. I am not expecting the response that LLaVA-RLHF just outperforms LLaVA-SFT on human alignment benchmarks (i.e., LLaVA-Bench and MMHAL-BENCH).

Author response:
Our proposed Fact-RLHF indeed has helped the model to reduce hallucination. After investigating the evaluation by GPT-4 on MMHal-Bench, we would like to clarify that our model trained with **Fact-RLHF was actually at a disadvantage** when being evaluated with open-ended questions. The reason is that RLHF encourages the model to make detailed, informative responses, which are often significantly longer than responses given by a model only trained with SFT, as well as human annotators. Therefore, our RLHF model may include some information that are valid statements about the image, but just not included in the human-generated answer or image annotations. Consequently, GPT-4 would mistakenly believe that our RLHF model had hallucinations. We observe that such biased evaluation exists in our RLHF-based models (both 7B and 13B), but not in our model only trained with SFT or other LMMs, since they are not further tuned by RLHF.

Let’s see some examples:

Example 1
- Image: https://farm1.staticflickr.com/3815/9340455006_25a885f7ef_o.jpg
- Question: Who in the picture is wearing trousers?
- Image content annotations: "Clothing", "Human body", "Person", "Man"
- Human answer: The woman in the image wears the pants.
- LLaVA-SFT+-13B answer: The woman in the picture is wearing trousers.
- GPT-4 rating for LLaVA-SFT+13B: 5, very informative, no hallucination
- LLaVA-RLHF-13B answer: In the picture, the woman is wearing trousers while sitting next to the man. The man is wearing shorts, and both of them are sitting in front of a campfire. This indicates that they are likely on a camping trip or enjoying an outdoor activity together. The presence of a campfire suggests that they are in a natural setting, possibly in a park or a campsite. The fact that they are both sitting in chairs around the fire implies that they are taking a break from their activities, perhaps to relax, socialize, or cook food. Their choice of clothing, with the woman wearing trousers and the man wearing shorts, suggests that they are dressed for a warm and comfortable outdoor experience. The woman's choice of trousers may provide her with more protection from the elements, such as cold temperatures or insects, while the man's shorts allow for better air circulation and freedom of movement.\n\nOverall, the image of a woman wearing trousers and a man wearing shorts sitting around a campfire represents a moment of leisure and enjoyment in the outdoors, where they can appreciate the natural beauty and serenity of their surroundings.
- GPT-4 rating for LLaVA-RLHF-13B: 1, somewhat informative, with hallucination
- Explanation: LLaVA-RLHF-13B correctly identifies the person “wearing trousers,” but also includes additional information that can be observed in the image. However, the additional information (such as the campfire and chairs) are not included in the human-generated answer or the annotations. As a result, GPT-4 incorrectly assigns a low score to the answer.

Example 2
- Image: https://c2.staticflickr.com/4/3501/4068897399_a729022693_o.jpg
- Question: Describe the image in detail.
- Image content annotations: "Window", "Clothing", "Person", "Building"
- Human answer: The image captures a bustling street lined with various shops and an adjoining sidewalk. Several pedestrians are present, with two individuals notably passing a store with a green sign saying "Hawkin's Bazaar". At this store's entrance, a man stands, overseeing a spacious window display that houses diverse items, highlighted by a red and white sign. Interestingly, no trees or parked cars are present in the street. The mood is energetic, showcasing individuals engaged in their daily routines and shopping pursuits.
- LLaVA-RLHF-13B answer: The image depicts a lively street scene with a group of people walking past a store called "Hawkins Bazaar." The storefront is adorned with Christmas decorations, and the store appears to be selling a variety of items, including toys and gifts. There are several people walking along the sidewalk, with some of them carrying handbags. The pedestrians seem to be enjoying their time, browsing the storefronts and engaging in conversation. The street is bustling with activity, and the store is likely attracting customers with its festive decorations and diverse selection of products.
- GPT-4 rating for LLaVA-RLHF-13B: 1, somewhat informative, with hallucination
- Explanation: Similarly in this example, the model mentions additional information about “Christmas decorations,” which is valid but not included in the annotations or human-generated answer. Therefore, GPT-4 fails to correctly assess the response.

Due to limited space, we cannot include too many examples in this response, but we will include more detailed analysis regarding the under-estimation of RLHF models’ performance in the manuscript revision.

To summarize, RLHF would enrich our LMM’s responses. However, informative and detailed responses by RLHF models sometimes cannot be matched with elements in the annotations and human-generated answers. Although we have tried our best to make human answers as comprehensive as possible, there could always be more to include. As a result, the hallucination rate of our RLHF model is overly estimated by GPT-4. If we take this inherent bias into account, RLHF is indeed reducing the hallucinations.

Reviewer comment (Experiments):
Fact-RLHF works when additional human-annotation information (e.g., image captions and OK-VQA rationale) is available. It implies that we can’t use Fact-RLHF when such additional information does not exist. The paper did not study more realistic scenarios where additional information is unavailable or noisy.

Author response:
Fact-RLHF only needs question or image related factual information to augment the reward model. It is relatively cheap to annotate such factual information [1][2]. We also expect automatic caption generation techniques such as CapFilt [3] or DSC [4] (i.e., from a different model) can help automate this process.

Reviewer comment (Experiments):
Descriptions of each benchmark and task (e.g., basic statistics, data format, what should the model predict?) are missing.

Author response:
Thanks for pointing this out. We have previously included the implementation details as well as the hyperparameters in Appendix I. 

Regarding the description of each dataset, task and metrics, we have added more information in Appendix. Specifically, the three employed supervised fine-tuning datasets are VQA-v2, AK-VQA and Flickr30k as listed in Section 2.2.  We use “Yes” or “No” queries from VQA-v2 (83k), multiple-choice questions from A-OKVQA (16k), and grounded captions from Flickr30k (23k). The 10k human preference data are paired outputs from the base 7B LLaVA model and we ask the Amazon Turker annotators to label which one contains less hallucinations. The details about the collection process is in Appendix G. 


For each evaluation task, we report the accuracy for MMBench, which is a multiple-choice question benchmark consisting of 1031 questions. We report the F1 score for the POPE, which is a “Yes/No” question benchmark and consists of 3k questions in three categories (random, adversarial and popular). 
The LLaVA bench consists of 100 questions and will be evaluated by GPT4 against the outputs from text-only GPT4. Finally, we report the GPT4 score on MMHalBench, which has 96 questions as well targeting the hallucination level of each model.

Reviewer comment (Evaluation):
Evaluation protocol for each benchmark (e.g., which evaluation metric is used and how to compute the results) is missing.

Author response:
Thanks for pointing this out. We have previously included the implementation details as well as the hyperparameters in Appendix I. 

Regarding the description of each dataset, task and metrics, we have added more information in Appendix. Specifically, the three employed supervised fine-tuning datasets are VQA-v2, AK-VQA and Flickr30k as listed in Section 2.2.  We use “Yes” or “No” queries from VQA-v2 (83k), multiple-choice questions from A-OKVQA (16k), and grounded captions from Flickr30k (23k). The 10k human preference data are paired outputs from the base 7B LLaVA model and we ask the Amazon Turker annotators to label which one contains less hallucinations. The details about the collection process is in Appendix G. 


For each evaluation task, we report the accuracy for MMBench, which is a multiple-choice question benchmark consisting of 1031 questions. We report the F1 score for the POPE, which is a “Yes/No” question benchmark and consists of 3k questions in three categories (random, adversarial and popular). 
The LLaVA bench consists of 100 questions and will be evaluated by GPT4 against the outputs from text-only GPT4. Finally, we report the GPT4 score on MMHalBench, which has 96 questions as well targeting the hallucination level of each model.

Reviewer comment (Reproducibility):
The key issue of this work may be lack of details, as the method section only covers high-level procedures.

Author response:
We have updated the supplementary material to share all the details of our implementation. We will also open-source the code, model, and the collected datasets.

Reviewer comment (Experiments):
No information is given about labelers, such as how many annotators participate in, who they are, and how they are recruited, etc.

Author response:
We have added more information about the anonymized labelers in the Appendix G of the paper. We spend around $5000 to collect the annotations. It took around 1 a week on the MTurk platform, and 28 labelers were involved.

Reviewer comment (Reproducibility):
The supplementary file is useless for reviewing. Only a GitHub link is provided, but it is not accessible.

Author response:
We have updated the supplementary material to share all the details of our implementation. We will also open-source the code, model, and the collected datasets.

Reviewer comment (Experiments):
FACT-RLHF is sourced from QA pairs in the LLaVA, A-OKVQA, and VQA-v2. Are there any other datasets to get more samples?

Author response:
LLaVA-1.5 [1] is trained on LLaVA, A-OKVQA, VQA-v2, and many more datasets like TextCaps, RefCOCO, and VG. LLaVA-RLHF can also potentially benefit from those additional vision-language alignment data.

Reviewer comment (Reproducibility):
How much does it cost to collect FACT-RLHF annotations? How long does it take? How many labelers are involved?

Author response:
We have added more information about the anonymized labelers in the Appendix G of the paper. We spend around $5000 to collect the annotations. It took around 1 a week on the MTurk platform, and 28 labelers were involved.

Reviewer comment (Novelty):
This is the first LMM trained with RLHF.

Author response:
Thank you for your comments! We appreciate your recognition of the significance of our work as the first LMM trained with RLHF and its state-of-the-art results on LLaVA-Bench and MMHAL-BENCH. We are glad that these key aspects of our research were well received and appreciated. We address your questions below.

Reviewer comment (Evaluation):
The RLHF model degrades slightly on the capability benchmarks. Can you please cite the reference to the scaling law of LLaVA-RLHF mentioned in the paper?

Author response:
> Concern 1: The RLHF model degrades slightly on the capability benchmarks.

While there is a slight degradation at the 7B scale, the RLHF significantly improves performance on the MMBench at the 13B scale.

> Concern 2: Can you please cite the reference to the scaling law of LLaVA-RLHF mentioned in the paper?

We have added citations to the alignment scaling laws of [1,2].

[1] Askell, Amanda, et al. "A general language assistant as a laboratory for alignment." arXiv preprint arXiv:2112.00861 (2021).

[2] Kaplan, Jared, et al. "Scaling laws for neural language models." arXiv preprint arXiv:2001.08361 (2020).

Reviewer comment (Experiments):
Slightly confused by the data used for RLHF-RL Tuning. AFAIU the authors collect human preferences on 10k hold out LLaVA dataset. But how do you get the factual information for these examples (captions or rationals) for these required by the reward model?

Author response:
> Question 1: how do you get the factual information for these examples

Since the images are from the MS COCO dataset, they come with 5 captions for each image.

Reviewer comment (Experiments):
Can you please clarify how is the image captioning data from COCO converted to instruction tuning data and is it also used for RL Tuning?

Author response:
> Question 2: clarify how is the image captioning data from COCO converted to instruction tuning

We did not use captioning data for instruction tuning. It is only used as factual augmentation for the reward model during the RL phase.