Watermarking for Factuality: Guiding Vision-Language Models Toward Truth via Tri-layer Contrastive Decoding (2025)

Abstract: Large Vision-Language Models (LVLMs) have recently shown promising results on various multimodal tasks, even achieving human-comparable performance in certain cases. Nevertheless, LVLMs remain prone to hallucinations–they often rely heavily on a single modality or memorize training data without properly grounding their outputs. To address this, we propose a training-free, tri-layer contrastive decoding with watermarking, which proceeds in three steps: (1) select a mature layer and an amateur layer among the decoding layers, (2) identify a pivot layer using a watermark-related question to assess whether the layer is visually well-grounded, and (3) apply tri-layer contrastive decoding to generate the final output. Experiments on public benchmarks such as POPE, MME and AMBER demonstrate that our method achieves state-of-the-art performance in reducing hallucinations in LVLMs and generates more visually grounded responses.

## introduction interest in large vision-language models (lvlms) has surged recently, driven by integration of powerful large language models (llms) with visual encoders. this fusion enables a single model to interpret complex images and generate coherent descriptions. recent lvlms like llava @xcite and instructblip @xcite exemplify this trend: llava connects a vision encoder to an llm via a simple projection, while instructblip uses a dedicated query transformer to bridge modalities. such lvlms have demonstrated impressive performance on tasks including image captioning, visual question answering, and other multimodal benchmarks. a key limitation of lvlms is their tendency to hallucinate-generating details absent from the image, such as naming non-existent objects or misattributing properties (see fig. 1 ). such hallucinations are often caused by the dominance of unimodal (language) priors. a lightweight vision module is often paired (and fine-tuned) with llms, which causes a modality imbalance where the language side can overwhelm the visual side @xcite @xcite , outputting responses based mainly on llms' contextual or statistical biases. thus, mitigating hallucinations is crucial for high-stakes applications, such as autonomous driving, medical imaging, and legal evidence analysis, where hallucinated responses could lead to severe consequences. to mitigate such hallucinations, various approaches have been introduced. a straightforward approach is fine-tuning or specialized training: adjusting model weights on curated datasets that emphasize image-grounded truth @xcite @xcite , or employing reinforcement learning from human feedback (rlhf) or direct preference optimization (dpo) to penalize hallucinated outputs @xcite . more recently, training-free inference-time contrastive decoding methods have emerged as efficient alternatives. for example, vcd @xcite contrasts original and perturbed visual inputs to recalibrate the model's reliance on language priors. m3id @xcite boosts visual relevance via mutual information, while avisc @xcite monitors and adjusts visual attention distributions. octopus @xcite combines these strategies by dynamically selecting contrastive approaches through dpo-trained controllers. however, existing methods often overlook how visual tokens interact with language across layers, assuming final outputs suffice for grounding. to address this, we embed lightweight visual watermarks into input images and evaluate layer-wise consistency via targeted visual queries. this enables the identification of the most visually grounded intermediate layer without retraining or architectural modifications, forming the basis of our tri-layer decoding strategy. in this paper, we propose a novel training-free decoding strategy called tri-layer contrastive decoding (tcd), which employs a watermark to guide the identification of the most visually grounded intermediate layer. to select this layer, we embed the watermark into the input image, query a corresponding ad-hoc question, and compare the probability distributions of an answer token across all layers. we explore maximum probability gain search, which identifies the layer based on the probability gain of the label token prompted by the watermark between adjacent layers. given such visually grounded layer, we decode the model using tri-layer contrastive decoding with two additional layers, i.e., mature layer defined as the top layer and an amateur layer with the maximum jensen-shannon divergence (jsd) compared to the mature layer, inspired by dola @xcite . we evaluate our method on widely-used hallucination benchmarks-pope @xcite , mme @xcite , and amber @xcite and show that the proposed approach achieves state-of-the-art performance across various models and settings. detailed analyses further confirm the validity of our approach, demonstrating that watermark-guided tcd effectively mitigates hallucination. our contributions are as follows: to mitigate hallucinations in lvlms, several approaches have been proposed. some methods frame hallucination as a binary classification task @xcite , while others design post-hoc correction modules @xcite , or apply factually augmented reinforcement learning from human feedback (rlhf) @xcite and direct preference optimization (dpo) @xcite . however, these methods typically require additional training stages and curated data. more recently, training-free, inference-time methods have emerged to re-balance models during decoding. opera @xcite penalizes over-aggregated anchor tokens in beam search. vcd @xcite contrasts outputs from original and distorted visual inputs to reduce overreliance on unimodal priors and statistical biases. icd @xcite suppress hallucinations by contrasting responses to perturbed instructions. m3id @xcite upweights image features during token sampling, and avisc (woo probability green red blue (gt) mature layer green red blue (gt) green red blue (gt) amateur layer visually grounded layer watermark image input image watermark query "what is the last captcha character in the image?" visual encoder watermarked image input query "what color is the parrot on the left?" tokenization & embedding projection llm layer 1 layer 2 mature (top) layer amateur layer visually grounded layer layer 24 layer 16 mature layer layer 1 watermarked image tokens watermark query tokens watermark-guided visual layer selection selected maximum probability w 8 t probability maximum probability gain watermarked image tokens input query tokens amateur layer selection layer 24 mature layer layer 1 layer 16 jsd 0.8 0.3 0.1 selected (gt) green red blue (gt) tri-layer contrastive decoding "the color of the parrot on the left is blue" generated output token probability tri-layer contrastive decoding ## method given a visual context v (e.g., an image) and a textual query x, lvlms generate a textual response y. the response @xmath0, y 2 , . . . , y t } is calculated in an auto-regressive manner, where each token is predicted sequentially based on the preceding tokens, and t represents the total number of to-kens in the generated response. formally, the token probability distribution at each time step @xmath1, t ] can be formulated as follows: despite the effectiveness of these decoding strategies, a critical challenge remains: hallucination. in the context of lvlms, even if the probability distribution p θ assigns a high likelihood, a token y t is considered hallucinated if it lacks sufficient grounding in the provided textual query x or visual context v. to this end, we propose a novel tri-layer contrastive decoding with a watermark-guided visual layer selection scheme. this approach aims to realign the model's token probability distribution with the factual constraints in x and v, thereby reducing the incidence of hallucinations in the generated output. an overview of our proposed method is shown in fig. 2 . ## conclusion in this paper, we introduce tri-layer contrastive decoding (tcd), a training-free framework for reducing hallucinations in large vision-language models (lvlms). rather than assuming the final model output always provides the best visual grounding, we propose a principled approach that embeds lightweight visual watermarks into input images and leverages targeted visual queries to probe layer-wise consistency. by combining this watermark-guided visual layer selection with contrastive decoding across mature, amateur, and visually grounded layers, tcd dynamically recalibrates the model's reliance on vision and language, significantly improving factuality.