MRFD : Multi-Region Fusion Decoding with Self-Consistency for Mitigating Hallucinations in LVLM s (2025)

Abstract: Large Vision-Language Models (LVLMs) have shown strong performance across multimodal tasks. However, they often produce hallucinations—text that is inconsistent with visual input, due to the limited ability to verify information in different regions of the image. To address this, we propose **Multi-Region Fusion Decoding (MRFD)**, a training-free decoding method that improves factual grounding by modeling inter-region consistency. MRFD identifies salient regions using cross-attention, generates initial responses for each, and computes reliability weights based on Jensen-Shannon Divergence (JSD) among the responses. These weights guide a consistency-aware fusion of per-region predictions, using region-aware prompts inspired by Chain-of-Thought reasoning. Experiments across multiple LVLMs and benchmarks show that MRFD significantly reduces hallucinations and improves response factuality without requiring model updates.

## introduction large vision-language models (lvlms) have emerged as powerful tools for multimodal understanding, achieving significant advances in image captioning, visual question answering, and visual reasoning @xcite @xcite @xcite . however, these models frequently generate hallucinated content, producing textual outputs inconsistent with the visual input. this phenomenon poses substantial challenges for reliability and practical deployment @xcite . hallucinations typically manifest as misidentified objects, fabricated attributes, or omitted visual information, reducing the factual alignment between image and text. ## conclusion we presented multi-region fusion decoding (mrfd), a training-free approach that improves lvlm reliability by mimicking self-consistency multi-view verification. mrfd identifies salient regions via attention, estimates their reliability using jensen-shannon divergence (jsd), and fuses region-level predictions using consistencyweighted, chain-of-thought-inspired prompts. experiments show that mrfd effectively reduces hallucinations and improves factuality across various lvlms and benchmarks, offering a simple yet robust decoding strategy.