Token Preference Optimization with Self-Calibrated Visual-Anchored Rewards for Hallucination Mitigation (2025)

Abstract: Direct Preference Optimization (DPO) has been demonstrated to be highly effective in mitigating hallucinations in Large Vision Language Models (LVLMs) by aligning their outputs more closely with human preferences. Despite the recent progress, existing methods suffer from two drawbacks: 1) Lack of scalable token-level rewards; and 2) Neglect of visual-anchored tokens. To this end, we propose a novel Token Preference Optimization model with self-calibrated rewards (dubbed as TPO), which adaptively attends to visual correlated tokens without fine-grained annotations. Specifically, we introduce a token-level visual-anchored reward as the difference of the logistic distributions of generated tokens conditioned on the raw image and the corrupted one. In addition, to highlight the informative visual-anchored tokens, a visual-aware training objective is proposed to enhance more accurate token-level optimization. Extensive experimental results have manifested the state-of-the-art performance of the proposed TPO. For example, by building on top of LLaVA and Qwen, our TPO boosts the performance absolute improvement for hallucination benchmarks.

## introduction recently, large vision language models (lvlms) have showcased their remarkable capabilities in handling multimodal information, excelling in tasks such as image captioning, visual questionanswering, and complex visual reasoning @xcite @xcite . specifically, by integrating pre-trained language models with meticulously designed visual encoders, lvlms are capable of ef-which color are the glasses that the man in the image is wearing? visual-anchored rewards of ground truth answer visual-anchored rewards of generated responses . the man is not we aring glass . the man is not we aring glass es +tpo es the man in the image is we aring black glass . ## methodology the schematic illustration of the proposed tpo is demonstrated in figure 2 . in sec. 3.1, we present the preliminaries including the definition and offpolicy optimization of dpo. then we detail the visual-anchored rewards and token preference optimization loss in sec. 3.2 and sec. 3.3, respectively. ## conclusion in this study, we propose a novel pereference alignment method, tpo, to mitigation hallucinations in lvlms. tpo incorporates a self-calibrated visualanchored reward mechanism that automatically identifies "vision-anchored tokens" and adaptively assigns appropriate rewards to them. by adding noise to the visual input and capturing changes in the generation probability of each token, tpo computes a score indicating each token's relevance to visual information. based on the self-calibrated visual-anchored reward, tpo can perform more efficient token-level preference alignment optimization for lvlms. experimental results have proved that tpo not only alleviates the hallucination problem but also strengthens the model's attention to visual input when generating responses.