FIHA : Automated Fine-grained Hallucinations Evaluations in Large Vision Language Models with D avidson Scene Graphs (2025)

Abstract: The rapid development of Large Vision-Language Models (LVLMs) often comes with widespread hallucination issues, making cost-effective and comprehensive assessments increasingly vital. Current approaches mainly rely on costly annotations and are not comprehensive – in terms of evaluating all aspects, such as relations, attributes, and dependencies between aspects. Therefore, we introduce the FIHA (automated Fine-graIned Hallucination evAluation in LVLMs), which could access LVLMs hallucination in an LLM-free and annotation-free way and model the dependency between different types of hallucinations. FIHA can generate Q&A pairs on any image dataset at minimal cost, enabling hallucination assessment from both image and caption. Based on this approach, we introduce a benchmark called FIHA-v1, which consists of diverse questions on various images from three datasets. Furthermore, we use the Davidson Scene Graph (DSG) to organize the structure among Q&A pairs, in which we can increase the reliability of the evaluation. We evaluate representative models using FIHA-v1, highlighting their limitations and challenges. We released our code and data at https://github.com/confidentzzzs/FIHA.

## introduction large vision-language models (lvlms) such as minigpt-4 @xcite and llava @xcite , which extend large language models (llms) by incorporating visual encoders, such as clip @xcite , have shown prominent capabilities in visual understanding and generation @xcite . however, lvlms suffer from the issue of hallucination, which can lead to misinterpretation or erroneous assertions of the visual inputs, thus hindering the performance of models in multi-modal tasks @xcite @xcite . specifically, the models may describe objects that do not exist in the image or incorrect object attributes and relations between objects. generating such unreliable content will greatly reduce the model's credibility. therefore, it is crucial to establish a benchmark for evaluating the hallucination level of lvlms. previous studies @xcite , as shown in to mitigate these limitations, we propose finegrained hallucination evaluation (fiha), an automatic evaluation framework for assessing finegrained and diverse hallucinations in large-scale vision-language models. the framework accepts either images or captions as input and generates q&a pairs by extracting objects, attributes, and entity relations. it then formulates diverse questions (e.g., "what", "who", "which", etc.) that allow for free-form responses. by integrating blip-2 @xcite for caption generation, fast r-cnn @xcite for feature extraction, and a question-generation template, our pipeline enables fully automatic q&a generation without relying on llms or manual annotations. we organize q&a pairs using the davidson scene graph (dsg) @xcite . the dsg ensures leaf node responses depend on root node answers, reducing errors and improving reliability. our q&a pairs include negative, narrative, and interrogative questions, allowing a progressive, comprehensive evaluation of image understanding. we make the following key contributions through this work: ## methodology in this section, we introduce the overall pipeline of fiha as illustrated in figure 1 . in summary, our pipeline offers two approaches for q&a generation. the first is based on images: given an image i, we extract the necessary entities, including features such as objects, object attributes, and relations. using a rule-based method, we then generate the q&a pairs. the second approach is based on captions. if an image does not already have a caption, we can use blip-2 @xcite to generate captions. alternatively, if the dataset includes original captions, we can use them directly as input, pass them through the feature extraction process, and generate the corresponding q&a pairs. 2.1 fine-grained information extraction 2.1.1 information acquisition from the caption q&a generation based on caption includes caption generation (optional if original datasets include captions) and extract information (object existence, object attributes and object relations) and using these information to generate q&a pairs. ## analysis in this section, we further evaluate the effectiveness of our benchmark fiha-v1 by four research questions. ## related work in this section, we mainly discuss existing large vision-language models (lvlms) and the hallucination problems that exist in lvlms. ## conclusion in recent years, large vision-language models have developed quickly, but hallucinations remain a serious concern. current hallucination evaluation methods face problems like high costs, limited scope, and lack of generalization. thus, we introduce fiha, a multi-dimensional detection method that requires no llms and no annotations. fiha can automatically create high-quality q&a pairs for any image dataset. we conducted a thorough analysis of the performance of mainstream lvlms, identified the issues, and proposed potential methods for improvement. in the future, we will delve deeper into methods for alleviating hallucinations.