Guiding Medical Vision-Language Models with Diverse Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations (2025)

Abstract: While mainstream vision-language models (VLMs) have advanced rapidly in understanding image-level information, they still lack the ability to focus on specific areas designated by humans. Rather, they typically rely on large volumes of high-quality image-text paired data to learn and generate posterior attention maps. To address this critical issue, we propose leveraging visual prompts—simple visual markers in various forms—to guide and enhance the formation of region-specific attention. Thus, we introduce **MedVP**, a pioneering framework that integrates medical entity extraction, visual prompt generation, and dataset adaptation for visual prompt-guided fine-tuning. We successfully outperform recent state-of-the-art large models across multiple medical VQA datasets. Extensive experiments and Human evaluation are conducted to analyze the impact of different visual prompt forms and how they contribute to performance improvement. The results demonstrate both the effectiveness and clinical significance of our approach.

## introduction as human beings, we inherently rely on visual cues or prompts to understand complex visual content in greater detail. these visual prompts may take various forms, yet they serve a fundamentally similar purpose: to direct our attention toward critical areas that contain rich information. current mainstream pre-trained vision-language models often lack mechanisms to direct their attention to specific areas of interest. instead, their attention distribution is formed during training on specific datasets. however, these pre-trained attention patterns are not universally applicable, particularly in specialized domains such as biomedical images. consequently, a substantial amount of high-quality image-text paired data is required to * equal contribution. figure 1 : the figure illustrates the difference in the reasoning process of the vision-language model (vlm) with and without visual prompts. by emulating how humans approach vision-language tasks, we use an auxiliary visual prompt, akin to pointing at a specific region, to help the model focus more easily on the relevant details and generate accurate responses. to be specific, vision-language models (vlms) are a class of models designed to process both image and text inputs for cross-modal tasks. depending on the specific task, the architecture and pretraining strategies of these models can vary significantly. for the purposes of this discussion, we will focus specifically on large vision-language models (lvlms), a subset of vlms that are particularly designed for image-to-language tasks, such as visual question answering (vqa) and image captioning. in this line of research, foundation models such as blip-2 @xcite and llava @xcite have significantly influenced the design of current mainstream lvlms. taking llava as an example, these models are typically composed of an image encoder, a projection layer, and a large language model (llm)-driven decoder. in the majority of vision-language models (vlms) @xcite , pretraining primarily involves generating descriptions of image content, which encourages the model to focus on global visual features rather than specific regions. during this phase, the attended regions in images remain unguided, allowing the attention mechanism to autonomously learn an implicit attention distribution. as previously discussed, conventional self-attention mechanisms in these models often struggle to generate region-specific content. however, certain vision-language tasks, such as visual question answering (vqa) and referring expression comprehension (rec), inherently require the model to attend to specific objects or localized regions. learning an effective posterior attention distribution for a given textual query, however, typically demands a substantial amount of training data, making it a challenging problem in vision-language learning. to mitigate this problem, recent research @xcite @xcite @xcite has increasingly recognized the value of visual prompts and their application in vision-language tasks. visual prompts can be categorized as implicit or explicit concerning the visual inputs. in this study, we will focus on explicit visual prompts, which are basically visual markers in various form that can be directly added to the input images. inspired by vip-llava @xcite , we decide to add visual markers into the input images to highlight the region of interest (roi) for the visual-prompt guided vlms to focus on important regions selected by prior knowledge. however, current works @xcite @xcite usually assume these rois are provided interactively by human annotators. given the high cost of acquiring human annotations in medical images, we propose a novel method, medvp (medical visual prompting) that will automatically generate visual prompts for the interested region given the context. specifically, our framework first extracts entities or relevant key-words from the queries, as our focus is on medical vqa tasks. next, we fine-tune an open-set grounding model that generates coordinates for the target area based on the identified medical entities. finally, these coordinates are used to generate visual prompts in various formats-such as scribbles, bounding boxes, or circles-on the input images. along with the visual-prompted images, we adapt the text question-answer pairs from slake, vqa-rad, and pmc-vqa datasets to ensure that the language decoder becomes aware of the presence of our visual prompt markers within the images. for example, we incorporate text such as "carefully observe the area in the red box..." to guide the llm-driven decoder to focus on the visually prompted area, ensuring that the model pays attention to the regions highlighted by the visual prompts. the visual-prompted images are fine-tuned with vision-language models (vlms) to develop our model, medvp-llava-the first medical vlm guided by visual prompts. we find that our medvp-llava performs effectively in medical vqa tasks, significantly enhancing performance across multiple datasets. in conclusion, our medvp method has at least the following contribution: figure 2 : the framework of medvp: an automated, explicit visual prompt-guided approach for medical visionlanguage models. the framework first aligns the model with region-level medical knowledge. additionally, it generates explicit visual prompts by leveraging keywords from the question and visual grounding models, integrating these prompts into medical images to enhance performance on medical vqa tasks. ## method in this section, we outline the proposed method, detailing the training pipeline and objectives for med-vlms, including the visual prompt extraction process and fine-tuning downstream medical visual question-answering tasks. ## impact our approach integrates visual prompts to facilitate the medical vqa task, not only enhancing model performance but also offering clinical relevance for medical image analysis. during training, visual prompts serve as attention guidance, directing the model to focus on the most clinically relevant regions, thereby improving both the efficiency and effectiveness of the learning process. in the inference stage, visual prompts provide contextual reference points, enhancing the interpretability of the model's responses while contributing to improved safety and reliability in medical decision-making. furthermore, the proposed framework is easily adaptable to real-world clinical scenarios, where medical professionals can directly annotate visual prompts onto medical images, seamlessly integrating them into the workflow. these factors collectively underscore the potential of our approach in advancing medical imaging applications, improving both automated analysis and human-ai collaboration. ## limitations during the knowledge pre-training stage, the dataset we utilize contains ai-generated visual markers to guide the model's attention. these contents might not be factually accurate, due to the limitation of grounding models we adopted. therefore, we perform a small-scale error analysis (e) and a human evaluation (f) on the ai-generated visual prompts, with detailed results provided in the appendix. due to the high cost and scarcity of human-annotated medical image data, we rely on ai-generated annotations to augment our training corpus. in future work, we aim to develop more rigorous data filtering and validation strategies and integrate retrieval-augmented generation (rag) techniques into our model to mitigate factual inaccuracies. ## conclusion in this work, we propose the medvp method, which introduces explicit visual prompts into medical images to guide med-vlms in focusing on specific regions of interest (rois) for medical vqa tasks. our approach includes a medical visual prompt extraction process. first, we use llm to identify medical entities from the questions. next, we train a grounding model to locate the rois in the images based on the extracted entity names. finally, we blend the visual prompts with the medical images. our medvp-llava achieves significant performance improvements across three medical vqa datasets, demonstrating the effectiveness of incorporating visual prompts to enhance medical vlms' capability in medical question answering.