Black-Box Visual Prompt Engineering for Mitigating Object Hallucination in Large Vision Language Models (2025)

Abstract: Large Vision Language Models (LVLMs) often suffer from object hallucination, which undermines their reliability. Surprisingly, we find that simple object-based visual prompting—overlaying visual cues (e.g., bounding box, circle) on images—can significantly mitigate such hallucination; however, different visual prompts (VPs) vary in effectiveness. To address this, we propose Black-Box Visual Prompt Engineering (BBVPE), a framework to identify optimal VPs that enhance LVLM responses without needing access to model internals. Our approach employs a pool of candidate VPs and trains a router model to dynamically select the most effective VP for a given input image. This black-box approach is model-agnostic, making it applicable to both open-source and proprietary LVLMs. Evaluations on benchmarks such as POPE and CHAIR demonstrate that BBVPE effectively reduces object hallucination.

## introduction lvlms @xcite demonstrate impressive capabilities but often suffer from object hallucination, where they describe objects not present in the image. addressing this issue is vital for real-world deployment, particularly in critical areas like healthcare and assistive technologies @xcite . existing methods try to mitigate object hallucination by collecting datasets @xcite , re-training or fine-tuning @xcite , modifying decoding methods @xcite @xcite , or using costly feedback loops @xcite . however, they often require access to model internals (e.g., attention, logits), making them impractical for proprietary lvlms @xcite . a promising yet under-explored direction is visual prompting, which overlays visual cues like bounding boxes or circles on images to guide model outputs @xcite @xcite . while visual prompting has shown potential in improving visual grounding @xcite , its role in reducing object hallucination remains unclear. this raises two key questions: (q1) can visual prompting mitigate object hallucination in lvlms? (q2) if so, can we systematically learn the optimal vps? our preliminary experiments show that simple object-based vps can significantly reduce object hallucination. interestingly, their effectiveness varies across images and is particularly notable in an oracle scenario, where the best-performing vp for each image is assumed to be known. this finding effectively answers q1 (see fig. 1 ) and suggests the need for a systematic method to identify the optimal vp for each image. to answer q2, we introduce bbvpe, a novel framework designed to systematically identify and apply optimal vps to reduce object hallucination in lvlms. our approach treats lvlms as "black boxes", relying solely on input-output pairs without modifying the model itself. the framework has three key components: (1) a pool of predefined vps, (2) a scoring function to evaluate the effectiveness of each prompt, and (3) a router model that dynamically selects the best prompt based on observed input-output behavior. our method requires no access to model internals, making it applicable to both open-source and proprietary lvlms. our key contributions are: 1) we find that oracle vps exist for images given an lvlm, which, when identified, can greatly reduce object hallucination. automated prompt engineering. prompt engineering refines input prompts (x) to yield better outputs (y * ) without modifying model parameters (θ). while traditionally a manual process, ape automates this refinement and has been widely applied in llms @xcite @xcite to improve text prompts. in the vision-language domain, research has also focused on optimizing textual prompts for clip @xcite ) or text-to-image diffusion models @xcite . with llms evolve into multimodal system, capable of handling both text and visual data, ape's application to visual inputs is still largely unexplored. to our knowledge, this work is the first to extend ape to visual inputs, aiming to reduce hallucinations in lvlms. ## black-box visual prompt engineering applying prompt engineering to the visual domain is challenging due to the vast combinatorial complexity of image space. also, direct optimization over pixel values risks distorting the semantic content of the images. to circumvent this, we use a discrete selection approach, choosing from a predefined vps that enhance images without altering their original meaning. a lightweight router model selects the most suitable vp, which is then applied before input to lvlms, reducing hallucinations. our black-box approach mitigates hallucinations without accessing internal lvlm values (e.g., attention, logits), making it compatible with proprietary models. an overview is shown in fig. 2 . oracle. the oracle represents an ideal scenario where the optimal vp for each image is known during evaluation, setting an upper bound on performance (see fig. 1 right). it is equivalent to adaptively selecting the vp with minimal hallucination per image. our goal is to train the router model to approximate this behavior. object localization. to identify relevant objects within an image i, we first utilize an object localization model l. the model detects and outputs a set of object coordinates @xmath0, o 2 , . . . , o m }. visual prompt pool. we define a pool of candidate vps @xmath1, p 2 , . . . , p n }, which includes visual markers like circles and arrows. each vp p @xmath2, producing i p i . the image-text pair (i p i , t ), where t is a textual prompt, is then fed into the lvlm m to produce a response. quantifying object hallucination. to evaluate a model's robustness to object hallucination, we define a scoring function s that measures response accuracy regarding object presence: dataset construction. for a given image i, the optimal vp p * is chosen to maximize s: to ensure uniqueness, cases where multiple vps achieve the highest score are excluded. this results in a training dataset d train that maps images to unique optimal prompts, including the option of not applying any vp: training a router model. the router model r θ is trained on d train to predict the optimal vp p * for a given image i. it assigns a score ŝp i to each vp: these scores are converted into probabilities via softmax: the router model is trained using cross-entropy loss between the predicted probability distribution p (p i | i) and the one-hot encoded ground-truth optimal vp p * : the trained router model enables efficient vp selection without directly querying the lvlm. lvlm inference. at inference, the trained router model r θ predicts the optimal vp p: @xmath3 applying p to the localized objects o in i produces i p, which, along with the textual prompt t , is fed into lvlm m to obtain a response with reduced object hallucination. ## experiments in all tables, baseline refers to not using visual prompting. we compare our approach against three baselines: (1) selecting random vp for each image, (2) consistently using a fixed best vp that delivers the highest overall performance for the model, and ## conclusion in this work, we proposed bbvpe framework to systematically identify optimal vps that mitigate object hallucinations in lvlms. our findings confirm that: