M edi VLM : A Vision Language Model for Radiology Report Generation from Medical Images (2025)

Abstract: Generating radiology reports from medical images has garnered sufficient attention in the research community. While existing methods have demonstrated promise, they often tend to generate reports that are factually incomplete and inconsistent, fail to focus on informative regions within an image, and impose strong annotation assumptions, such as bounding box annotations, image level annotations (which can be challenging to obtain) for model training. In this paper, we propose MediVLM, a vision language model (VLM) for radiology report generation from medical images. The proposed model consists of a pre-trained object detector to extract the salient anatomical regions from the images, an image encoder, a text encoder, a module to align the visual and text representations, a cross attention layer to fuse the two representations and finally, a transformer based decoder to generate the final report. MediVLM can generate radiology reports even when no reports are available for training; this is an extremely useful feature, as curating such reports is a labor-intensive task. Further, it computes a severity score (depicting the seriousness of a patient’s medical condition) from the generated radiology reports, which can be used to prioritize patients who need immediate medical attention. Our extensive empirical analyses on three benchmark datasets corroborate the promise and potential of our method against competing baselines. Our code is open-sourcedin our project webpage at: https://sites.google.com/view/medivlm/home

## introduction clinical radiology (such as x-rays, mri scans etc.) is a common type of medical imaging examination and is critical for identifying common diseases such as pneumonia and lung cancer @xcite . given a radiograph, radiologists manually examine the different anatomical regions and describe both the normal and abnor-figure 1 : for a given medical image, medivlm generates a free text radiology report, together with a severity score denoting the seriousness of the patient's medical condition. best viewed in color. while existing techniques for automated report generation have depicted encouraging performance, they often tend to generate reports that are incomplete (miss important observations in the images) or inconsistent (contain factually incorrect information) @xcite . another drawback of current methods is that most of them utilize imagelevel visual features to generate reports and as a result, fail to focus on specific regions within an image that contain anatomical abnormalities @xcite @xcite . further, some methods require specialized annotations for report generation. the icon method @xcite is based on lesion extraction and requires image-level annotations for model training, which can be challenging to obtain in certain medical settings. the region-guided radiology report generation (rgrg) method requires finegrained annotations, such as bounding boxes, for radiology report generation @xcite , which are expensive to obtain given the dearth of trained radiologists. in this paper, we propose medivlm, a vision language model for radiology report generation from medical images to alleviate these challenges. our proposed method first extracts salient anatomical regions from the given input images using a pretrained object detector. a visual encoder is then used to extract spatial features from these patches; at the same time, a language model is used as a tokenizer to embed the input radiology reports into latent representations. the image and text embeddings are aligned using contrastive learning and fused together with a trainable cross attention module. finally, a transformer based language model (decoder), with trainable attention layers, is used to generate the medical report. medivlm is trained end-to-end using a loss function consisting of a cross entropy loss term and a contrastive loss term. the proposed medivlm framework can be trained in an unsupervised manner (when no reports are available for training). we exploit a pseudolabeling strategy to address the absence of ground truth reports and train the medivlm using the generated reports, without requiring any architectural modifications. this is an extremely useful feature, as curating radiology reports is an expensive process in terms of time, labor and human expertise. further, experienced radiologists are busy, which further underscores the challenge of obtaining high quality radiology reports for model training. our model also generates a severity score s for each generated report, which depicts the criticality of the patient's medical condition. this score can be taken into consideration while scheduling doctors' appointments and can potentially mitigate the long waiting times for patients with serious medical conditions. figure 1 depicts a sample output of our system for a given medical image. our contributions in this paper can be summarized as follows: ## related work automatic radiology report generation has attracted significant research attention in recent years; please refer to @xcite for a detailed survey. conventional methods: earlier efforts adopted cnn-rnn architectures to analyze medical images and generate diagnostic reports @xcite @xcite . more recent studies have used the transformer model due to its effectiveness @xcite . chen et al. @xcite proposed a memory-driven transformer (r2gen) to generate radiology reports, which introduces a memory module and a memory-driven conditional layer normalization module into the transformer decoder architecture. the design of the memory module and layer normalization inspired several subsequent research. chen et al. @xcite proposed cross-modal memory networks (cmn) to enhance the encoder-decoder framework for radiology report generation, where a shared memory was designed to record the alignment between images and texts so as to facilitate the interaction and generation across modalities. a few strategies have been exploited to better attend to abnormal regions in an image, such as iteratively aligning visual features and disease tags @xcite , contrasting normal and abnormal images @xcite and exploiting medical knowledge graphs @xcite . a radiologist-minded report generation framework, x-rgen, was proposed by chen et al. @xcite , which mimics the behavior of human radiologists by breaking the entire process down into four principal phases: initial observation, cross-region analysis, medical interpretation and report generation. yeasin et al. @xcite developed a transformer-based system called auto-rad for lumbar spinal stenosis diagnostics and reporting. hierarchical approaches have also been studied @xcite @xcite , in which high-level concepts are first extracted from the image and subsequently decoded into individual sentences. wang et al. @xcite introduced a multi-head transformer that was applied to patch features from a cnn backbone, with each head assigned to a specific anatomical region, generating sentences exclusively for that region. along similar lines, tanida et al. @xcite proposed an explainable framework called region-guided radiology report generation (rgrg) method that detects anatomical regions and generates individual descriptions for each. other methods: reinforcement learning (rl) based methods have also been exploited for radiology report generation and improving clinical accuracy @xcite . @xcite proposed an rl approach over a cross-modal memory (cmm) to better align visual and textual features for radiology report generation. multiple instance learning has also been used for histopathology report generation by aligning whole slide images and diagnostic reports from local and global granularity @xcite . data augmentation techniques like mixup have been used in this context to ensure that the representations of the semantically equivalent lesions align with the same attributes, so as to maintain inter-report consistency @xcite . a line of research has focused on exploiting the temporal structure, i.e. prior images and reports (if available) for report generation @xcite . researchers have also studied the problem of report generation in the unpaired setting (where paired image-report data is unavailable for training) by leveraging information in two distinct datasets, one containing reports and the other containing images @xcite . while these methods have depicted encouraging performance, they often tend to generate reports that are inaccurate and incomplete @xcite , fail to focus on the informative anatomical regions in the images @xcite @xcite and impose strong an-notation constraints, such as image-level or bounding box annotations @xcite . image captioning: radiology report generation is largely inspired by research in image captioning @xcite @xcite . however, while the key ideas from the image captioning domain can be applied to radiology report generation, there are a few important differences: (i) radiology reports are much longer and more diverse than typical image captions, due to the multiple anatomical regions within an image; (ii) generating descriptions of specific but crucial abnormalities is challenging due to the heavy data imbalance towards normal images and normal reports. ## experiments and results datasets. we used three benchmark datasets to study the performance of medivlm: comparison baselines. we used five recent methods as comparison baselines in our work: (i) r2gen (chen et al., 2020); (ii) x-rgen (chen et al., 2024); (iii) cmn (chen et al., 2021); (iv) histgen (guo et al., 2024); and (v) rl (qin and song, 2022). these baselines were selected to cover a wide range of report generation techniques, including memory-driven transformers, methods that mimic the behavior of human radiologists, methods based on multiple instance learning and reinforcement learning. for the mimic-cxr dataset, we used several other comparison baselines, as detailed in ## visual illustrations: unsupervised training of medivlm figure 3 depicts a couple of low severity examples on the iu x-ray dataset, when medivlm is trained in an unsupervised manner (only images and no reports are available for training). for the first image, the gt report mentions that the heart size is normal, the lungs are clear with no signs of consolidation, the hilar and mediastinal contours are normal, and there are no acute abnormalities. all these findings are appropriately captured in the medivlm report. further, since there are no abnormalities, this has been designated as a low severity case (@xmath0.22). the same observations are evident for the second image, where medivlm correctly captures the important findings and designates it to be a low severity case, as there are no significant abnormalities. these examples demonstrate that medivlm can appropriately capture the important findings in a medical image even when no radiology reports are available for training. it can also compute a severity score which appropriately reflects the seriousness of a patient's medical condition, even when it is trained in an unsupervised manner. these results are extremely important from a practical standpoint, given the shortage of trained radiologists to conduct the labor-intensive task of report transcription. we include an analysis of the values of different parameters (section a.3) and more visual illustrations demonstrating the performance of medivlm and the severity scores (section a.4) in the appendix. ## conclusion in this paper, we proposed medivlm, a vision language model for radiology report generation from medical images. apart from report generation, medivlm also furnishes a severity score for each generated report, depicting the seriousness of the patient's medical condition, which can be used to prioritize patients who need immediate medical attention. further, it can also generate reports when only images and no reports are available for training; this is an extremely useful feature, as curating such reports is a labor-intensive task. our exten-sive empirical results on three benchmark datasets (including a dataset where the reports are provided in french) corroborated the promise and potential of our framework against competing baselines.