Improving Alignment in LVLM s with Debiased Self-Judgment (2025)

Abstract: The rapid advancements in Large Language Models (LLMs) and Large Visual-Language Models (LVLMs) have opened up new opportunities for integrating visual and linguistic modalities. Yet, challenges remain in aligning these modalities effectively, causing issues such as hallucinations, where generated outputs are not grounded in the visual input, and safety concerns in the application of LVLMs across various domains. Existing alignment methods, such as instruction tuning and preference tuning, often rely on external datasets, human annotations, or complex post-processing, which limit scalability and introduce additional costs. To address these challenges, we propose a novel approach that generates the debiased self-judgment score, a self-evaluation metric created internally by the model without relying on external resources. This enables the model to autonomously improve alignment. Our method enhances both decoding strategies and preference tuning processes, resulting in improved alignment, reduced hallucinations, and enhanced safety. Empirical results show that our approach significantly outperforms traditional methods, offering a more effective solution for aligning LVLMs.

## introduction owing to the powerful capabilities of large language models (llms) @xcite @xcite , large visual-language models (lvlms) demonstrate impressive performance by effectively integrating visual inputs into the latent representation space of llms @xcite @xcite . however, similar to llms, lvlms face inherent alignment challenges, including hallucinations (where the generated content is not grounded in the image) @xcite , and safety issues @xcite , which negatively impact the application of lvlms across various domains @xcite @xcite . to address misalignment in lvlms, a growing body of recent research has explored enhancing model alignment by leveraging external tools or human annotations to assist with preference tuning @xcite @xcite and inference @xcite . however, most prevailing approaches rely heavily on powerful external resources-such as advanced models like @xcite or human experts-which can lead to substantial costs during both training and inference. moreover, in a hypothetical future where an ai system requiring alignment surpasses both human intelligence and the capabilities of other models, supervision from humans or existing models may offer only limited effectiveness for such a superintelligent system. in response to these challenges, we draw inspiration from the effective self-reflection abilities ob-served in llms @xcite and explore how lvlms can self-evaluate and enhance their alignment independently. we observe that the internal confidence of lvlms can reflect the faithfulness of their output sentences, but it also incorporates significant textual priors. building on this, we introduce the debiased self-judgment score, a sentence-level evaluation metric generated autonomously by the model without relying on external resources. this score is applied to both decoding and preference tuning. our results show that this approach significantly enhances lvlms' performance, improving faithfulness, safety, and overall capability, as shown in figure 1 . in summary, our contributions are three-fold: ## preliminary observations in this section, we present preliminary findings on the potential and limitations of lvlms' selfjudgment abilities, which serve as the foundation for our proposed debiased self-judgment score. ## method in this section, we propose a method that leverages the model's internal confidence for self-judgment and eliminates text modality bias, resulting in a debiased self-judgment score. this score is used for decoding and preference tuning to enhance lvlms' faithfulness, safety, and overall capability. specifically, section 4.1 describes how to derive the debiased self-judgment score and apply it to generate more faithful descriptions; section 4.2 incorporates the score with a safety prefix to prevent unsafe outputs; and section 4.3 investigates how both sentence-level and instance-level selfjudgment contribute to self-improvement training. ## experiments in this section, we evaluate the performance of the proposed debiased self-judgment score across various applications, aiming to answer the following questions: (1) can dsgd effectively reduce hallucinations in lvlms compared to other baselines? (2) can fgsd reduce unsafe outputs while maintaining the utility of lvlms? (3) can dsr effectively enhance the comprehensive capabilities of lvlms? (4) are the self-judgment method and the debiasing method we designed truly effective? results. the primary experimental results are summarized in ## conclusion in this paper, we propose a novel self-alignment method to solve the alignment problems in large vision-language models. by using a debiased selfjudgment score, our approach enables the model to improve its vision-language alignment on its own, eliminating the need for external data or human intervention. our extensive experiments demonstrate that this method reduces hallucinations and makes lvlms safer and more powerful. the promising experimental results of our method indicate that self-judgment has considerable potential for enhancing alignment in lvlms.