Visual Hallucinations of Multi-modal Large Language Models (2024)

Abstract: Visual hallucination (VH) means that a multi-modal LLM (MLLM) imagines incorrect details about an image in visual question answering. Existing studies find VH instances only in existing image datasets, which results in biased understanding of MLLMs’ performance under VH due to limited diversity of such VH instances. In this work, we propose a tool called VHTest to generate a diverse set of VH instances. Specifically, VHTest finds some initial VH instances in existing image datasets (e.g., COCO), generates a text description for each VH mode, and uses a text-to-image generative model (e.g., DALL-E-3) to generate VH images based on the text descriptions. We collect a benchmark dataset with 1,200 VH instances in 8 VH modes using VHTest. We find that existing MLLMs such as GPT-4, LLaVA-1.5, and MiniGPT-v2 hallucinate for a large fraction of the instances in our benchmark. Moreover, we find that fine-tuning an MLLM using our benchmark dataset reduces its likelihood to hallucinate without sacrificing its performance on other benchmarks. Our benchmarks are publicly available: https://github.com/wenhuang2000/VHTest.

## introduction a multi-modal llm (mllm) @xcite @xcite @xcite generates a text response for a given image and question. an mllm typically comprises three components: a vision encoder, a vision-language connector, and an llm. the vision encoder (e.g., clip @xcite ) converts an image into an embedding vector. the vision-language connector projects an image embedding vector into the llm's word embedding space. the projected vector is concatenated with the token embeddings of the question to form an input to the llm, which generates a text response. there are two lamps in the picture: one on the bedside table to the right, and another on the desk to the left in the background. text response figure 1 : an example of mllm's visual hallucination. the text in red highlights the hallucinated detail in the image, where there are three lamps. however, mllms often generate text responses containing factually incorrect details about an image, known as visual hallucination (vh) @xcite . figure 1 shows an example where the mllm hallucinates two lamps, contradicting the three lamps in the image. vhs in mllms pose obstacles to developing safe and trustworthy ai, which is emphasized in a recent u.s. executive order calling for rigorous testing to address potential harms from advanced ai systems (the white @xcite . prior works have tried to benchmark mllms' vhs related to object existence @xcite , optical character recognition (ocr), object counting, object positions comparing @xcite , orientation, and viewpoint @xcite (concurrent to ours). however, they collect vh images only from existing image datasets like coco @xcite . this limits the diversity of vh images since they can only find a limited number of them. moreover, existing image datasets may have been used to pre-train an mllm, leading to data contamination @xcite . as a result, such vh images lead to a biased understanding of an mllm's performance, e.g., an mllm is incorrectly concluded to perform well under vh. ## related work hallucinations hallucinations are well-known issues for generative ai, including llms @xcite , mllms @xcite @xcite , and text-to-image generative model @xcite . in general, hallucination refers to a generative model imagines factually incorrect details in its response for a given input. vh occurs when an mllm imagines incorrect details about an image in visual question answering. ## discussion we attribute such high challenge level to the adversarial construction idea during vh instance generation, which is similar in spirit to adversarial examples @xcite commonly used to test and improve model robustness. unlike most existing mllm benchmarks that are limited to the image space of existing image datasets like coco, vhtest derives text descriptions from initial vh instances, which play a role of "adversarial direction". then vhtest exploits text-to-image generative models to generate challenging vh instances following such adversarial direction. this moves beyond the image space of limited existing image datasets and into the manifold of text-to-image generative models, thereby making it possible to generate more challenging vh instances. thus, our vhtest offers a new perspective for constructing mllm benchmarks for future research. ## conclusion we propose vhtest to generate vh instances to test mllms. we collect vh benchmarks using vhtest and we find that state-of-the-art mllms exhibit high hallucination rates on our benchmarks. moreover, fine-tuning mllms on our benchmark reduces hallucination without sacrificing their other performance/capability.