A uto H allusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Models (2024)

Abstract: Large vision-language models (LVLMs) are prone to hallucinations, where certain contextual cues in an image can trigger the language module to produce overconfident and incorrect reasoning about abnormal or hypothetical objects. While some benchmarks have been developed to investigate LVLM hallucinations, they often rely on hand-crafted corner cases whose failure patterns may not generalize well. Additionally, fine-tuning on these examples could undermine their validity. To address this, we aim to scale up the number of cases through an automated approach, reducing human bias in crafting such corner cases. This motivates the development of AutoHallusion, the first automated benchmark generation approach that employs several key strategies to create a diverse range of hallucination examples. Our generated visual-question pairs pose significant challenges to LVLMs, requiring them to overcome contextual biases and distractions to arrive at correct answers. AutoHallusion enables us to create new benchmarks at the minimum cost and thus overcomes the fragility of hand-crafted benchmarks. It also reveals common failure patterns and reasons, providing key insights to detect, avoid, or control hallucinations. Comprehensive evaluations of top-tier LVLMs, e.g., GPT-4V(ision), Gemini Pro Vision, Claude 3, and LLaVA-1.5, show a 97.7% and 98.7% success rate of hallucination induction on synthetic and real-world datasets of AutoHallusion, paving the way for a long battle against hallucinations. The codebase and data can be accessed at https://github.com/wuxiyang1996/AutoHallusion

## introduction large vision-language models (lvlms) @xcite bring powerful tools for content generation @xcite , autonomous * equal contribution. figure 1 : autohallusion: we propose three image manipulation strategies to induce hallucinations: abnormal object insertion, paired object insertion, and correlated object removal, which trigger the conflicts between the images and lvlm priors. given generated images, we ask lvlms questions on object existence and their spatial relations for visual question answering. it is critical for the research community to collect these cases and investigate the reasons behind them. with sufficient hallucination examples, finetuning lvlms on them with the original training data may reduce hallucinations and alleviate those biases. however, crafting those cases in previous work requires expensive human labor and is highly time-consuming @xcite @xcite @xcite . moreover, it is unclear whether those handcrafted examples are rare corner cases or indicate general fail patterns. without an in-depth understanding of the common mechanism generating them, it is hard to extract useful insights to improve lvlms. on the other hand, finetuning on those small benchmarks without sufficient representative examples may lead to overfitting. to address those challenges, we develop an automated pipeline, autohallusion, to generate diverse hallucination cases and mass-produce them at the minimum cost of human efforts. to generate (image, question) pairs that can trigger the hallucinations of lvlms, we take a reverse-engineering path: it starts from exploring output answers due to hallucinations, by probing lvlms' language modules to allocate the strong language priors on certain objects or their contextual relations. it then creates (1) an image containing objects that contradict the probed priors (the presumed answers), and (2) questions on two types of conflicts, the existence of contextual-related objects and their spatial relationships. if the lvlm reasoning is biased or dominated by the language prior, it tends to generate incorrect or inconsistent responses conflicting with the ground truth in the images, hence the hallucinations. we provide an optimization formulation and develop three principal strategies, abnormal object insertion, paired object insertion, and correlated object removal, to manipulate the objects in a scene and thus create images conflicting with the language prior, as illustrated in figure 1 . the detailed designs of these hallucination strategies are inspired by schema @xcite @xcite from cognitive science. schema refers to the tendency of humans to organize information and interpret the world based on patterns of past experiences foot_0 . following its concept, irregular schema with cognitive dissonance @xcite , e.g., an octopus in front of a monitor, and breaking a schema with expectancy violation @xcite , e.g., the absence of a keyboard and a mouse in front of a monitor, can both induce contradictions and discomforts in the memory. the three strategies reveal common patterns and mechanisms of how hallucinations are generated, hence providing critical insights to detect, combat, avoid, or control hallucinations of lvlms. main contributions: inspired by an analogy to human cognition in terms of schema, we investigate the mechanism of hallucinations in lvlms by reverse-engineering (image, question) pairs with probed language priors and biases. we develop autohallusion that synthesizes images by manipulating the objects in the scenes to conflict with lvlms' memory (i.e., its language priors), and generates questions about the conflicts. the novelties of our work can be summarized as: ## related work vision-language models (vlms). the recent increase in large language models (llms) @xcite , including gpt-3 @xcite , palm @xcite , and bloom @xcite , has significantly improved natural language processing. llama @xcite further advanced this field, and models like alpaca @xcite , inspired by instructgpt @xcite and chatgpt, utilized human-annotated data to refine llama, enhancing its interaction abilities. additionally, large visual language models (lvlms) such as gpt-4 @xcite , flamingo @xcite , bard (ai, 2023), minigpt-4 @xcite , instructblip @xcite , and llava @xcite have developed. these models combine visual and language processing to manage both textual and visual inputs and produce textual outputs. their architecture generally includes a visual encoder (often based on clip @xcite ), a modality connection module, and an llm. lvlms excel in generating text descriptions from images and multimodal learning through pre-training on image-text pairs @xcite and instruction-tuning with various tasks @xcite @xcite . however, addressing hallucinations in their textual outputs remains a challenge, emphasizing the need for reliability and accuracy in real-world applications. benchmarks. several benchmarks have been developed to assess hallucination in vlms in various aspects. chair @xcite evaluates object hallucination by measuring word accuracy against ground-truth sentences and segmentation for 80 mscoco objects. pope @xcite improves upon chair for better stability and flexibility while openchair @xcite extends chair to open-vocabulary settings. hallu-sionbench (guan et al., 2024a) targets visual commonsense and reasoning with 455 visual-question control pairs. hal-eval @xcite introduces and focuses on event hallucination while correlationqa @xcite examines the impact of spurious visual inputs. our work differs from previous benchmarks @xcite using hand-crafted examples from datasets like coco @xcite or visual genome @xcite by employing an auto-generated approach to synthesize hallucination cases through contextual influences, making our method more effective and scalable. object hallucination. large vision language models (lvlms) hold great potential but struggle with object hallucination, generating incorrect descriptions that include nonexistent objects or omit key details. this problem can adversely affect applications in robotics @xcite , medical imaging @xcite , and human-computer interaction @xcite . object hallucination in lvlms manifests as fictional objects, false attributes, or inaccurate relationships between objects @xcite . previous methods, like fine-tuning smaller multimodal models @xcite , are less effective for lvlms due to their distinct ar-chitectures. recent efforts focus on improving dataset quality for fine-tuning @xcite , but acquiring such data remains labor-intensive. metrics like chair @xcite and pope @xcite , which assess caption relevance and hallucination levels, are crucial for evaluation. standard text quality metrics can be misleading, as high scores may still correlate with significant hallucination. in this paper, we investigates contextual biases in language priors causing hallucinations and introduces two new metrics for more effective detection. ## problem formulation pronounced bias in llms hinders the reasoning capability of lvlms, resulting in hallucinations from the images @xcite . inspired by this, we target the biases in llms to induce hallucinations in lvlms. definitions and objective. our objective is to find things that are correlated in the llm but not present in the picture to induce hallucinations in lvlms. let f lvlm , f llm denote the lvlm and its llm component, respectively. f lvlm (image, query) can take an image-query pair as inputs, and f llm (context, query) can take a textonly context-query pair as inputs. we use sets as universal representations for the images and texts and detailed as below. we denote v as the set of all contextual elements in an image i, where each element can be an object, an attribute associated with an object, or the relation between/among multiple objects, etc. foot_1 these elements in the set can be considered as a statement, which could be either affirmative or negative. similarly, for text modality, we denote q as the set containing objects of interest for questions and c as the set of objects in this scene for context. we use a mapping function t (•) to transform a set of contextual elements into a text, which can be either a description from c or a query question from q. finally, we introduce the contextual distance d[•, •] between two descriptions or texts. when two pieces of text convey similar information or affirm each other, the contextual distance d is considered small; otherwise, the contextual distance is large. let y q be the ground truth answer set with respect to the query set q given the image i. the objective function can be formulated as follows: abnormal object insertion 2. image manipulation paired objects insertion correlated object removal 3. questions construction is in the image ? we have & in this image. is in the image ? is on the left of in the image ? existence spatial relation or victim lvlm edited image language prior probing we have in the image. what is the object to insert / remove under the strategy x where we wish to … 1. scene generation scenes set a objects set existing dataset generative model (dall•e ) ground truth 4. hallucination detection correctness consistency based on the ground truth , is the visual question pair created in correct? are there any conflicts between the responses from ? object status: existing / removed / objects in the image object of interests context in image after that, we use text to probe the language prior of the victim lvlm and then propose three manipulation strategies to induce hallucination in scene images(yellow). we finally use two metrics to detect hallucinations (blue). (2) the objective function (1) maximizes the distance between the generated text f lvlm and y q to produce hallucination. to leverage and probe the bias in the language components f llm of the victim lvlm, we use constrain (2) to control the discrepancies between responses, with and without visual input, within a tolerance ϵ. this ensures that the answer is dominated by the prior language component rather than the visual component. the visual information v from the image i provided to f lvlm is partially converted to a text, t (c), as the input to f llm , therefore c ⊆ v. remark. it is important for the language component f llm to have the constraint q ∩ @xmath0. if the interested element from q is directly given in the context c, it would be difficult to exploit the bias of f llm since it is mostly likely to answer based on the provided context c. for example, if a fact is directly given in the prompt, it is hard for the model to make a contradictory claim. in addition, q is not required to be the subset of v since we can ask questions on objects that are not included in the image i. approach. it is hard to optimize i, q, and c by directly optimizing eq. ( 1 ). instead, we probe the lvlm and the language prior from its llm component to determine (q, c) such that the elements in q are highly likely (or unlikely, depending on the attack strategy) to be present with c in the same scene. such bias in the language prior helps us achieve the constraint (2). this ensures that the language prior is strong and highly confident on the co-occurrence of (q, c), i.e., pr(q | c) ≤ δ (q is abnormal given c) or pr(q | c) ≥ 1δ (q is hypothetical given c), where pr(q | c) is the probability of the existence of elements in q given the presence of c and δ is the confidence level. if the assumption on (q, c) pairs that the lvlm reasoning is dominated by its language prior, i.e. eq. ( 1 ) holds true, we can create i from such (q, c) pairs to maximize the discrepancy in eq. ( 1 ). ## methodology the overall pipeline of our methodology is presented in fig. 2 . we break down the automated procedure of creating hallucination cases into 4 stages: scene generation, image manipulation, question construction, and hallucination detection. questions constructed to induce potential hallucination cases vary depending on these strategies, mainly focusing on object existence and spatial relations between the target object and others. we detect hallucinations through correctness and consistency among answers generated by the victim lvlm. ## conclusion in this paper, we introduce autohallusion, the first automatic benchmark generation approach to create diverse hallucination examples. inspired by schema in cognitive science, we analyze the mechanism of how and when lvlm hallucinations are triggered. we then reverse-engineer the hallucinating images based on probed lvlms' language priors by three principal strategies, abnormal object insertion, paired object insertion, and correlated object removal, that manipulate scene images using object insertion or removal to create conflicts with the priors. we construct textual probing methods to construct and detect hallucinations created. au-tohallusion achieves a significant success rate of inducing lvlm hallucinations on manipulating both synthetic and real-world data. we will keep improving the quality of the synthesized images by inpainting techniques based on more recent text-toimage models. meanwhile, we will explore better textual probing methods extracting more diverse contextual information within the image. we will also further investigate the causes of multi-modal hallucinations and build a more rigorous mathematical model for them.