Retrieval Visual Contrastive Decoding to Mitigate Object Hallucinations in Large Vision-Language Models (2025)

Abstract: Despite significant advancements in Large Vision-Language Models, Object Hallucination (OH) remains a persistent challenge. Building upon prior studies on contrastive decoding that address this issue without requiring additional model training, we introduce RVCD (Retrieval Visual Contrastive Decoding), an advanced method to suppress OH. RVCD leverages both negative and positive images at the logit level, explicitly referencing AI-generated images designed to represent a single concept. Our approach demonstrates substantial improvements over existing decoding-based methods.

## introduction large vision language models (lvlms) are models designed to generate sophisticated textual responses based on multimodal inputs of images and text. in recent years, successful experiments on integrating vision encoders with language models have demonstrated promising progress in this field @xcite @xcite . however, large vision language models (lvlms) are still not free from the issue of object hallucination (oh), which refers to the phenomenon where lvlms erroneously generate hallucinated objects and descriptions in their outputs @xcite . oh can be categorized into three types: generating descriptions of objects that do not exist in the image (existence), misdescribing the attributes of existing objects (attribute), and incorrectly describing the relationships between objects (relationship) @xcite . previous studies have demonstrated that even more sophisticated and larger lvlms are not free * corresponding author. to address object hallucination (oh), which undermines the reliability of lvlms, various methodologies have been proposed. these include approaches that mitigate oh by modifying the outputs generated by lvlms @xcite , introducing self-correction pipelines @xcite , or employing decoding-based methods @xcite @xcite . among these methodologies, visual contrastive decoding-based approaches are particularly attractive and practical because they mitigate oh without requiring additional training for the models @xcite @xcite . these methods distort the input source images @xcite , or zoom into the local views containing important objects in the source images @xcite to generate logits for regulation. theses logits modify or replace the logits generated from the original input images, contributing to producing better outputs. however, despite the excellence of their methods, they do not fully exploit the potential of visual contrastive decoding -the potential that the images used to generate logits for regulation do not always need to be transformations of the original images. we introduce a novel method called rvcd (retrieval-visual contrastive decoding), which maximizes the regulation strength by retrieving and leveraging multiple explicit images as regulatory targets. the explicit images we use are designed to encapsulate a single concept, enabling the contrastive decoding process to add or subtract the desired or undesired concept effectively, thereby allowing the adjusted logits to clearly align with the target image. our method retrieves multiple explicit reference images, generates negative logits to be regulated, and recovers positive logits lost during the regulation process, utilizing them at every decoding step for token generation. these explicit reference images are created using image generation models to represent single concepts and are ultimately selected based on agreement conditions between the image generation models and lvlms. similar to prior studies, our method can be easily applied to open-source lvlms, such as minigpt-4 @xcite , llava @xcite , and mplug-owl2 @xcite . our contributions are summarized as follows: (1) we propose a novel plug-and-play decoding method called rvcd @xcite . this method is train-free, strongly regulates oh, and simultaneously preserves high output text quality. (2) we provide a retrieval database utilized in rvcd. all images in the database are aligned with the consensus between diffusionbased image generation models and lvlms, and they represent single concepts. this database enhances the explainability of our rvcd and can serve as a resource for future research leveraging explicit concepts in visual contrastive decoding. (3) through comprehensive experiments, we demonstrate the strong oh reduction capability of rvcd, which significantly outperforms existing methods. ## related work object hallucination (oh) refers to the phenomenon where bert-based vision language models (vlms) @xcite , or more recent lvlms @xcite @xcite @xcite , erroneously generate unfaithful contents. @xcite and @xcite categorized oh into three types: existence, attribute, and relationship oh. these correspond to generating descriptions of nonexistent objects, producing misleading descriptions, and generating incorrect descriptions of relationships between existing objects, respectively. the most dominant metric for evaluating oh is chair @xcite . this metric can be used in scenarios where a finite synonym dictionary and a set of ground truth objects mapped to an image are defined. it evaluates the proportion of synonyms appearing in lvlm outputs that are not defined in the ground truth object set (chair i ), and the proportion of sentences where chair i is non-zero (chair s ). another well-known recent metric is pope @xcite , which measures the degree of oh using precision, recall, and accuracy. to do so, pope frames a binary classification problem for the lvlm, evaluating its outputs based on the inclusion of positive or negative assertions (e.g., "yes" or "no"). additionally, the traditional and standard text generation quality metric bleu @xcite is still utilized in recent studies @xcite . bleu serves as an additional indicator to ensure that little sacrifice in text quality occurs while mitigating oh @xcite . efforts to mitigate object hallucination (oh) have been ongoing since the introduction of the chair metric by @xcite , yet oh remains an unsolved challenge despite advancements in lvlms @xcite @xcite . no lvlm to date has completely resolved oh in its outputs. to address oh, recent approaches have explored various strategies. @xcite however, despite the excellence of the core idea of visual contrastive decoding (vcd) @xcite , including the above studies that leverage it, they fail to fully exploit its hidden potential. specifically, they overlook the fact that the types of images used to generate regulated logits are not necessarily restricted to variations of the original input image. building on the intuition that explicit images for regulation can be derived from the external database, we propose retrieval-visual contrastive decoding (rvcd). specifically, we constructed a database by generating explicit ai-created images that best represent the single concept of each word in the finite vocabulary used for oh evaluation @xcite . in our approach, hal (yolo) hal (lvlm) gt (yolo) gt (lvlm) 0.0 0.2 0.4 0.6 0.8 1.0 precision (@xmath0.5 minigpt4 mplugowl2 figure 1: detection precision for yolo and lvlm detectors on mscoco validation 2014 (lin et al., 2014). hal (•) shows the proportion of hallucinated objects from greedy-decoded captions detected by yolo and lvlms vqa that were true hallucinations. gt (•) illustrates the proportion of objects correctly identified as existing by yolo and lvlms. while both perform similarly in detecting existing objects, yolo excels in hallucination detection, motivating us to transfer yolo's strength to lvlms for correcting hallucinated objects. the statistical details are provided in appendix c. multiple regulated and preserved logits are generated from images retrieved from this database. ## methodology the overview of our method is illustrated in figure 2 . first, we generate a greedy-decoded text result for the input image using the lvlm, which we refer to as the draft caption. simultaneously, we obtain a list of objects present in the same image using an object detection (od) model such as yolo. objects mentioned in the draft caption but not detected by the od model are classified as negative objects. objects detected by both the draft caption and the od model are classified as positive objects. our goal is to suppress the generation of tokens related to negative objects while preserving the representation of positive objects at every token generation step, by corresponding logits generated from retrieved explicit images. we describe the details of each step below. ## conclusion we propose rvcd, an advanced train-free decoding-based plug-and-play method that significantly alleviates the object hallucination (oh) problem of lvlms. inspired by prior studies leveraging the idea of visual contrastive decoding (vcd), rvcd maximizes the potential of vcd through negative and positive logits generated from explicit images retrieved from a single-concept image database. comprehensive experiments demonstrate that rvcd significantly outperforms other state-of-the-art (sota) decoding methods.