Mitigating Hallucinations in Large Vision-Language Models by Self-Injecting Hallucinations (2025)

Abstract: Large Vision-Language Models (LVLMs) suffer from serious hallucination problems, where the model-generated responses are inconsistent with the visual inputs. Existing hallucination mitigation methods are mainly based on preference alignment and require external human annotations or auxiliary models for preference data collection, which increase costs and limit sustainable improvement. To tackle these challenges, we propose Autonomous Preference Alignment via Self-Injection (APASI), a novel and generalizable method that mitigates hallucinations without external dependencies. APASI leverages the target LVLM to self-inject hallucinations into a generated response, creating a pair of responses with varying preference levels. During the self-injection process, the dis-preferred response is generated based on three key observations of hallucinations, ensuring it simulates real hallucination patterns. This fidelity offers an accurate learning signal for hallucination mitigation. Moreover, APASI incorporates an iterative alignment training strategy combined with curriculum learning to periodically update the preference data with increasing challenge, enabling stable and continuous enhancement of the LVLM. Extensive experiments across six benchmarks show that APASI not only effectively mitigates hallucinations for three baseline models but also achieves comparable or even superior performance to alignment-based methods with external dependency, thereby demonstrating its effectiveness and generalization capability.

## introduction the study of large vision-language models (lvlms) @xcite @xcite has made remarkable progress in recent years. lvlms substantially enhance cross-modal understanding in vision-language tasks and achieve notable performance across various applications. despite their demonstrated efficacy, lvlms frequently encounter hallucination problems which refer to the inconsistency between the factual content of visual input and the corresponding generated textual response @xcite . this problem undermines the reliability of lvlms, making the mitigation of hallucinations a critical area of research. various methods have been proposed to mitigate hallucinations in lvlms through preference alignment techniques, such as reinforcement learning from human feedback (rlhf) @xcite and reinforcement learning from ai feedback (rlaif) @xcite . as shown in fig. 1 (a), the target lvlm is trained using ranking or corrective preference data to align with the preference for no hallucinations or reduced hallucinations. notably, rlaif-based methods alleviate the cost and subjectivity issues in rlhf preference data collection @xcite by replacing human annotators with auxiliary feedback models such as gpt-4 @xcite . methods @xcite further promote these improvements by replacing proprietary models with open-source models. however, as alignment progresses, the capabilities of the target lvlm become comparable to the fixed and finite capabilities of the feedback models @xcite , particularly in handling hallucinations. at this point, the feedback models may struggle to detect minor hallucinations in the lvlm's responses or provide effective corrections. therefore, methods that rely on external models face limitations in achieving sustainable improvement of the target lvlm. to overcome these limitations, we propose autonomous preference alignment via self-injection (apasi), a novel hallucination mitigation method that constructs preference data using the target model itself. though following the common framework of using the direct preference optimization (dpo) @xcite algorithm to iteratively align the target lvlm with the preference for reduced hallucinations, apasi distinctively utilizes self-generated data, as shown in fig. 1(b ). instead of collecting ranking or corrective feedback as typically done in rlhf/rlaif methods, apasi generates its data by injecting hallucinations into model-generated responses, creating effective dis-preferred responses that simulate real hallucination patterns. with the original modelgenerated responses as the preferred, this process forms valid preference pairs for dpo. meanwhile, the straightforward injection of hallucinations is autonomously executed independently of external auxiliary models, facilitating sustainable model improvement. moreover, apasi is easily scalable as it requires no annotations. however, directly injecting hallucinations into responses through bad prompts or visual corruption, as done in @xcite , fails to accurately simulate real hallucination patterns. the self-injection process in apasi is based on three key observations of hallucinations: 1) lvlms are prone to hallucinate objects that frequently co-occur with the existent objects in the image @xcite ; 2) lvlms are prone to generate hallucinated content with an over-reliance on language priors @xcite ; 3) hallucinations in lvlms typically cluster towards the latter part of the response @xcite . accordingly, the target lvlm is guided to fabricate sentences about non-existent co-occurring objects with language-only inputs and then integrates this hallucinated content into the latter parts of the preferred response, forming the dis-preferred re-sponse. in this way, the preference pair facilitates an accurate learning signal for the mitigation of real hallucination patterns. moreover, as learning progresses, apasi incorporates a curriculum @xcite to gradually reduce the injection of hallucinations, making it more challenging to distinguish subtler differences in the preference pairs, thereby helping to refine the lvlm's ability to identify hallucinations smoothly over iterations. our contributions are summarized as follows: ## methodology we first provide preliminaries of the dpo algorithm in section3.1. then in section3.2, we dive into the preference data construction process based on pertinent self-injection of hallucination to provide an accurate learning signal for dpo training. finally, we introduce the training scheme for the apasi framework using iterative alignment strategy with curriculum learning to achieve continuous and stable optimization in section3.3. ## conclusions we propose apasi, a novel method for mitigating hallucinations in lvlms via preference alignment without external dependencies. this self-injecting process, based on key observations, has provided an accurate learning signal for effective preference alignment. apasi has further employed iterative alignment with curriculum learning for improving the training process. extensive experiments have demonstrated our effectiveness and superiority across three baselines and various benchmarks.