V is D ia H al B ench: A Visual Dialogue Benchmark For Diagnosing Hallucination in Large Vision-Language Models (2024)

Abstract: Despite the significant success of large vision-language models (LVLMs), some studies have revealed that LVLMs suffer from the hallucination problem, where the LVLMs’ response contains descriptions of non-existent objects. Although various benchmarks have been proposed to investigate this problem, they mostly focus on single-turn evaluation and overlook the hallucination raised by textual inputs. To investigate the hallucination problem of LVLMs when given long-term misleading textual history, we propose a novel visual dialogue hallucination evaluation benchmark VisDiaHalBench. The benchmark consists of samples with five-turn questions about an edited image and its original version. VisDiaHalBench differs from previous hallucination benchmarks in the following three points: 1) The questions and answers are unambiguously grounded by annotated scene graphs. 2) The images are uncommonly edited to inspect the visual model and common-object hallucination in LLMs. 3) The carefully designed dialogue refers a same object in different turns to assess the image consistency and influence of history for LVLMs. The detailed analysis of several state-of-the-art LVLMs across image consistency, visual understanding, history influence, and other dimensions reveals their substantial performance gap with single-turn VQA tasks. The benchmark is released in: https://github.com/qingxingcao/VisDiaHalBench

## introduction large language models (llms) @xcite @xcite have shown exceptional capabilities in semantic understanding, reasoning, and commonsense utilization. to expand their capabilities into the vision domain, recent research integrated vision-pretrained models with llms, resulting in large vision-language models (lvlms) @xcite @xcite @xcite . through instruction finetuning on textimage paired data, lvlms achieved great performance on a variety of multimodal tasks that require a vast range of skills. despite the recent success, lvlms also suffer from the hallucination problem, where the responses are nonfactual or contradictory to the given inputs. to study this problem, recent works (yifan @xcite @xcite have proposed different benchmarks and methods to diagnose hallucination within lvlms. however, while previous works have shown that llm will over-commit to early mistakes and leading to snowballing hallucination @xcite , these studies in multi-modal domains mostly focus on the visual side, neglecting to fully investigate the textual inputs, such as misleading dialogue history or prior responses. given that lvlms are built upon llms, there is a strong likelihood that lvlms inherit the similar hallucination problem from llms. as shown in the second example of figure 1 , gpt-4v correctly answers the relation of the frisbee but subsequently hallucinates about a non-existent "white frisbee" given a follow-up similar question. to analyze hallucinations in lvlms when faced with multi-turn visual and textual inputs, we introduce visdiahalbench, a novel visual dialogue benchmark grounded by image scene graphs. each sample in visdiahalbench contains an edited image and its original version to evaluate lvlms' performance with unseen images; a scene graph to ground questions-answers for unambiguous evaluation; and a five-turn coherent dialogue with last turn references to investigate image consistency and history influence of lvlms. for example, a correct answer about the edited brown frisbee will be an incorrect description of the original white frisbee, which may affect lvlms when answering the last question. to construct the benchmark, we leverage the normalized scene graphs and corresponding images from the gqa (hudson and manning, 2019) dataset to construct visdiahalbench. for each image, we first edit the gqa image as well as the scene graph by removing or changing an object. then in each turn, we sample a question type and scene graph paths based on the last turn object, attribute, or relation. with the grounded path and predetermined answers, we employ gpt-4 @xcite to generate a natural language question. we further benchmark several sota lvlms and conduct comprehensive studies of their hallucination results. our analysis reveals that current lvlms still suffer from insufficient capabilities in vision understanding and handling complex dialogue history. our contribution can be summarized as follows: 1) we introduce the first visual dialogue benchmark focusing on diagnosing hallucinations raised by visual and long-term textual inputs. 2) we conduct comprehensive studies on sota lvlms to investigate their image consistency, history handling, and other relevant aspects. we hope our benchmark and analysis can shed light on further research in addressing the hallucination problem in both language and vision domains. ## visdiahalbench we introduce visdiahalbench, a new visual dialogue benchmark to analyze hallucination in lvlms under different visual and textual inputs. based on the normalized scene graph and image in the gqa dataset, each sample contains an edited image, a five-turn dialogue, and grounding scene graph paths. the questions are associated with the edited object in different aspects, including queries about the uncommon edited results, pronoun-based reference, and misleading history. to construct the benchmark, we first edit the image by randomly changing an object's attribute, swapping it with another object, or removing it from the image. then at each turn, we sample a question type and scene graph path based on the edited object or previous questions. lastly, we prompt gpt-4 @xcite to generate a natural language question strictly grounded to the supporting paths and ground truth answer. the overall generation pipeline is shown in figure 2 . to ensure the unambiguity of generated questions and answers, we rely on sampling scene graph path that can uniquely identify an object. also, by designating the start or end object of a path, we can generate coherent multi-turn questions concerning the last-turn context. to better illustrate the construction process, we first present our path sampling method and question type details, then describe each construction step. ## conclusion in this work, we propose visdiahalbench, a novel visual dialogue benchmark specifically designed to investigate the hallucination phenomenon in large vision-language models (lvlms). our comprehensive analysis shows the weaknesses of current lvlms in terms of image consistency and dialogue history, highlight the challenge of hallucination in vision-language understanding.