V ivid M ed: Vision Language Model with Versatile Visual Grounding for Medicine (2025)

Abstract: Recent advancements in Vision Language Models (VLMs) have demonstrated remarkable promise in generating visually grounded responses. However, their application in the medical domain is hindered by unique challenges. For instance, most VLMs rely on a single method of visual grounding, whereas complex medical tasks demand more versatile approaches. Additionally, while most VLMs process only 2D images, a large portion of medical images are 3D. The lack of medical data further compounds these obstacles. To address these challenges, we present VividMed, a vision language model with versatile visual grounding for medicine. Our model supports generating both semantic segmentation masks and instance-level bounding boxes, and accommodates various imaging modalities, including both 2D and 3D data. We design a three-stage training procedure and an automatic data synthesis pipeline based on open datasets and models. Besides visual grounding tasks, VividMed also excels in other common downstream tasks, including Visual Question Answering (VQA) and report generation. Ablation studies empirically show that the integration of visual grounding ability leads to improved performance on these tasks. Our code is publicly available at https://github.com/function2-llx/MMMM.

## introduction medical data encompasses a broad spectrum of modalities, such as medical images, radiology reports and genomics. synthesizing these diverse data is essential for building a holistic view of the health condition of a patient, enabling precision diagnostics and treatment planning. the emergence of large multimodal models (lmms), particularly vision language models (vlms), has initiated a transformative paradigm shift in ai for * equal contribution. lmms overcome the limitations of task-specific specialist medical models confined to single input and output modalities, paving the way for the development of generalist medical ai (gmai) models @xcite . these models are anticipated to handle dynamically specified tasks and incorporate versatile input and output modalities, and realize many unprecedented clinical use cases. a prominent prospective use case of gmai models is to draft grounded radiology reports @xcite . specifically, beyond providing textual reports for given radiology images, gmai models could further ground the anatomical structures and abnormality findings mentioned by specific phrases in reports with localized visualizations, typically highlighting them with bounding boxes or segmentation masks. compared to plain textual reports, visually grounded reports possess significantly improved clinical utility by facilitating intuitive user interaction, effective interpretation of radiology images, and straightforward verification against harmful hallucinations. grounded report generation is rendered feasible by recent advancements in vlms with visual grounding, which are capable of generating visually grounded detailed conversations @xcite . despite generalpurpose visual grounding vlms have demonstrated impressive performance, they still struggle to interpret medical images with accurate anatomical localization @xcite . to address these limitations, we propose vividmed: vision language model with versatile visual grounding for medicine, which supports diverse downstream tasks and accommodates both 2d and 3d imaging modalities. vividmed implements visual grounding via prompting localization modules based on the segment anything model (sam) @xcite with the hidden em-beddings of the additional special tokens in the vlm's output, which are then decoded into corresponding image regions. replicating the success of general-purpose vlms with visual grounding capability in the medical domain is non-trivial, as several major challenges impede such efforts. (i) medical images are highly heterogeneous, encompassing diverse imaging modalities. however, existing vlms are predominantly developed for 2d natural images and are inherently inefficient at handling 3d medical images. to address this, we draw inspiration from previous works @xcite and dynamically adjust the patch embeddings of the vision encoder. (ii) existing grounding vlms typically generate either segmentation masks or bounding boxes. however, both forms are essential for our tasks. while some anatomical structures and abnormalities are best captured by segmentation masks, others are better delineated using bounding boxes as they are ill-suited for segmentation. therefore, we augment the localization module to generate both. (iii) radiologists sometimes refer to multiple instances in a single phrase, therefore requiring instance segmentation and detection. for example, in a chest x-ray report, a radiologist might note "multiple lung nodules are observed", without specifying each nodule separately. we attend to this problem by adapting sam to generate multiple outputs. (iv) the most significant obstacle is the scarcity of publicly available data. currently, there is no single dataset can support the development of grounded report generation. to tackle this challenge, we propose a three-stage training and automatic data annotation pipeline that make the best use of existing localization and report generation datasets to realize grounded report generation. experiments show that vividmed not only excels in previously unassailable visual grounding tasks, but also exhibits competitive performance on common downstream tasks such as visual question answering (vqa) and report generation. ablation studies also empirically show that the integration of visual grounding capability allows medical vlms to achieve improved performance on other downstream tasks. our main contributions are summarized as follows: ## conclusion in this paper, we present vividmed as a pioneering step towards vision-language models with versatile visual grounding for medical images. through its novel architecture, grounded data annotation pipeline, and the three-stage training procedure, vividmed exhibits superior performance on various downstream tasks, and realizes the visual grounding tasks especially the grounded report generation on mimic-cxr and ct-rate datasets. our em-pirical results show that the integration of visual grounding capabilities boosts the performance of medical vlms on other downstream tasks as well. we believe our work has established a robust baseline in this field, and hope that future research may focus on improving performance further, as well as integrating into reliable clinical applications that benefit patients.