Diving into Mitigating Hallucinations from a Vision Perspective for Large Vision-Language Models (2025)

Abstract: Object hallucinations in Large Vision-Language Models (LVLMs) significantly impede their real-world applicability. As the primary component for accurately interpreting visual information, the choice of visual encoder is pivotal. We hypothesize that the diverse training paradigms employed by different visual encoders instill them with distinct inductive biases, which leads to their diverse hallucination performances. Existing benchmarks typically focus on coarse-grained hallucination detection and fail to capture the diverse hallucinations elaborated in our hypothesis. To systematically analyze these effects, we introduce VHBench-10, a comprehensive benchmark for evaluating LVLMs across ten fine-grained hallucination categories. Our evaluations confirm encoders exhibit unique hallucination characteristics. Building on these insights and the suboptimality of simple feature fusion, we propose VisionWeaver, a novel Context-Aware Routing Network. It employs global visual features to generate routing signals, dynamically aggregating visual features from multiple specialized experts. Comprehensive experiments confirm the effectiveness of VisionWeaver in significantly reducing hallucinations and improving overall model performance. Our code and benchmark are available at https://github.com/whwangovo/VisionWeaver.

## introduction large vision-language models (lvlms), such as gpt-4v @xcite and llava @xcite , demonstrate remarkable abilities to understand @xcite and generate @xcite content from visual inputs. despite these * equal contribution. ## experiments the present experiments were conducted based on the llava-1.5 @xcite architecture. specifically, the llava-1.5 settings were followed, with clip-vit-l-336px serving as the base visual encoder and a two-layer mlp acting as the visual projector. concurrently, we substituted the llm with the most recent versions of llama3.2-instruct-3b (team, 2024) and qwen2.5-instruct-3b @xcite . this substitution was made to ascertain the applicability of our method to the latest llms. the 3b version was selected due to its suitability for end-side deployment and its prevalent use in devices such as cell phones. for multiple vision encoders in visionweaver, inspired by eagle @xcite , we chose convnext @xcite , eva-02 @xcite , sam @xcite , di-nov2 @xcite , and vary @xcite as task-specific visual encoders, which were pre-trained on different downstream tasks with different visual capabilities. a detailed description of each expert's specialized capabilities is provided in appendix d. in order to align with clip encoders when processing images, we use interpolation to fix the input resolution of all encoders to 336×336 and the output token to 576. the output dimension is fixed to 1024 using a linear adapter. ## discussion vision weaver is a vision-centric, encoder-level module designed to fundamentally improve the quality of visual features at their source. this makes it a complementary, plug-and-play component that can act synergistically with other mitigation strategies, such as improved training data or sophisticated decoding-time algorithms. by providing the language model with a more reliable visual foundation, our method reduces the burden on downstream processes to correct potential errors. this research opens several promising avenues for future work, including the possibility of distilling the collective knowledge of multiple experts into a single, more efficient encoder, or analyzing the routing mechanism more deeply to understand what visual cues trigger specific experts. furthermore, while our experiments indicate that simply increasing the number of experts does not guarantee better performance, determining the optimal quantity and composition of the expert pool remains a valuable open question. ## conclusion in this paper, we present the vhbench-10, a new benchmark that systematically classifies the visual hallucinations of the lvlm into 10 different categories, allowing for their fine-grained analysis. by replacing the visual encoder of lvlm , we found that different encoders lead to diverse hallucinatory tendencies. based on these insights, we propose visionweaver, a powerful lvlm architecture that incorporates a context-aware expert routing mechanism and a knowledge augmentation module to efficiently leverage task-specific visual expertise. extensive experiments demonstrate the effectiveness of our approach, establishing visionweaver as a powerful solution for alleviating hallucinations in lvlms. this work opens new avenues for developing more reliable and accurate lvlms. haoxuan you, haotian zhang, zhe gan, xianzhi du, bowen zhang, zirui wang, liangliang cao, shih-fu chang, and yinfei yang. 2023. ferret: refer and ground anything anywhere at any granularity. arxiv preprint arxiv:2310.07704. zihao yue, liang zhang, and qin jin. 2024. less is more: mitigating multimodal hallucination from an eos decision perspective. preprint, arxiv:2402.14545. yiyang zhou, chenhang cui, jaehong yoon, linjun zhang, zhun deng, chelsea finn, mohit bansal, and huaxiu yao. 2023. analyzing and mitigating object hallucination in large vision-language models. arxiv preprint arxiv:2310.00754.