HALLUCINOGEN : Benchmarking Hallucination in Implicit Reasoning within Large Vision Language Models (2025)

Abstract: Large Vision-Language Models (LVLMs) have demonstrated remarkable performance in complex multimodal tasks. However, these models still suffer from hallucinations, particularly when required to implicitly recognize or infer diverse visual entities from images for complex vision-language tasks. To address this challenge, we propose HALLUCINOGEN, a novel visual question answering (VQA) benchmark that employs contextual reasoning prompts as hallucination attacks to evaluate the extent of hallucination in state-of-the-art LVLMs. Our benchmark provides a comprehensive study of the implicit reasoning capabilities of these models by first categorizing visual entities based on the ease of recognition in an image as either salient (prominent, visibly recognizable objects such as a car) or latent entities (such as identifying a disease from a chest X-ray), which are not readily visible and require domain knowledge or contextual reasoning for accurate inference. Next, we design hallucination attacks for both types of entities to assess hallucinations in LVLMs while performing various vision-language tasks, such as locating or reasoning about specific entities within an image, where models must perform implicit reasoning by verifying the existence of the queried entity within the image before generating responses. Finally, our extensive evaluations of eleven LVLMs, including powerful open-source models (like LLaMA-3.2 and DeepSeek-V2), commercial models like Gemini, and two hallucination mitigation strategies across multiple datasets, demonstrate that current LVLMs remain susceptible to hallucination attacks.

## introduction in recent years, large language models (llms) have made significant advancements in natural language understanding and natural language generation, significantly advancing the field of artificial 1 please find the benchmark here in the image, there is a person standing near the yellow car parked in the parking. ## hallucinogen: a benchmark for evaluating hallucinations in lvlms in this section, we present the details of our proposed benchmark, hallucinogen, as illustrated in fig 2 . we first outline the construction of hal-lucinogen in section 3.1. next, in section 3.2, we provide the details on categorising various hallucination attacks introduced in hallucinogen. ## experimental results in this section, we demonstrate the utility of hallucinogen in studying the hallucination of lvlms and evaluating their effectiveness against mitigation and reasoning techniques. we first describe our experimental setup and then discuss the key findings of our benchmarking analysis. ## conclusion in this work, we introduce hallucinogen, a novel benchmark for evaluating hallucination in large vision-language models. it incorporates a diverse collection of visual entities and complex contextual reasoning prompts, referred to as hallucination attacks. these attacks are specifically designed to assess models' ability to perform implicit reasoning, such as inferring the presence or absence of a visual entity while executing complex visual-language tasks. through comprehensive qualitative and quantitative evaluations across a variety of lvlms, as well as testing various defense strategies on hallucinogen, we demonstrate that most lvlms perform near the level of random guessing when subjected to our attacks.