Can Medical Vision-Language Pre-training Succeed with Purely Synthetic Data? (2025)

Abstract: Medical Vision-Language Pre-training (MedVLP) has made significant progress in enabling zero-shot tasks for medical image understanding. However, training MedVLP models typically requires large-scale datasets with paired, high-quality image-text data, which are scarce in the medical domain. Recent advancements in Large Language Models (LLMs) and diffusion models have made it possible to generate large-scale synthetic image-text pairs. This raises the question: Can MedVLP succeed using purely synthetic data? To address this, we use off-the-shelf generative models to create synthetic radiology reports and paired Chest X-ray (CXR) images, and propose an automated pipeline to build a diverse, high-quality synthetic dataset, enabling a rigorous study that isolates model and training settings, focusing entirely from the data perspective.Our results show that MedVLP models trained exclusively on synthetic data outperform those trained on real data by 3.8% in averaged AUC on zero-shot classification. Moreover, using a combination of synthetic and real data leads to a further improvement of 9.07%. Additionally, MedVLP models trained on synthetic or mixed data consistently outperform those trained on real data in zero-shot grounding, as well as in fine-tuned classification and segmentation tasks.Our analysis suggests MedVLP trained on well-designed synthetic data can outperform models trained on real datasets, which may be limited by low-quality samples and long-tailed distributions[^1].[^1]: All data and code will be released upon acceptance.

## introduction in medical image analysis, learning representative features typically requires labor-intensive and costly image annotations @xcite . medical vision-language pretraining (medvlp) addresses this challenge by aligning vision and language content using paired datasets of images and clinical reports, reducing the need for manual annotations @xcite @xcite . however, existing medvlp models rely heavily on large-scale, high-quality paired data @xcite , which is scarce in practice. real-world datasets often contain noisy data, such as low-quality images and unpaired imagetext samples, degrading model performance @xcite . recent advancements in large language models (llms) and diffusion models enable the generation of large-scale synthetic image-text datasets, offering an alternative to traditional data collection. although these techniques have shown promise in medical tasks, they are primarily used as auxiliary support for real data via augmentation @xcite @xcite , and are often limited to single-modality settings. to the best of our knowledge, no studies have fully explored the potential of using synthetic multimodal data for medvlp or considered training exclusively on synthetic data @xcite . to bridge this gap and showcase synthetic data's potential for medvlp, our contributions are: ## experiments configurations for pre-training, we apply the official configurations provided by convirt @xcite and gloria @xcite on the mimic-cxr dataset to our synthetic cxr image-text dataset, syncxr. ## analysis effect of balanced entity sampling in generating synthetic reports. we evaluate the impact of balanced sampling entities when generating synthetic reports using llms. for the synthetic dataset without balanced sampling, we adjust entity frequencies to match their distribution in mimic-cxr, leading to a long-tailed distribution. as shown in tab 4a, for both medvlp methods, the performance improves significantly when using synthetic datasets generated from balanced sampled entities. this demonstrates that balanced sampling of entities leads to a more representative dataset, benefiting medvlp performance. evaluating the contribution of synthetic images and reports. we aim to assess the individual impact of synthetic images and synthetic reports on medvlp performance. as shown in tab 4b, we generate two partially synthetic datasets by replacing either the image or the text with synthetic data, while keeping the other components real, to evaluate their respective contributions. according to tab 4b, for both medvlp methods, using real images with synthetic reports results in decreased performance, likely due to the persistent long-tailed distribution, as the synthetic reports are generated based on real images. however, using real reports with synthetic images slightly improves performance, as synthetic images can be curated using our image filtering procedure to ensure high quality, avoiding issues commonly found in real datasets. using both synthetic images and synthetic reports achieves the highest performance, indicating that a well-curated synthetic dataset can significantly enhance medvlp performance. furthermore, we evaluate the medvlp method on the cleaned mimic-cxr dataset, as detailed in section e and ## conclusion in this work, we tackle the question: can medvlp succeed using purely synthetic data? our findings demonstrate that the answer is: yes. to the best of our knowledge, this is the first study to comprehensively explore the potential of synthetic data for medvlp models. we also identify key limitations in existing real-world datasets and introduce syncxr-a synthetic dataset of 200,000 imagetext pairs generated without any manual quality checks. our findings show that medvlp models trained on purely synthetic data outperform those trained on real data. moreover, combining synthetic and real data further boosts model performance, demonstrating the potential of synthetic data to overcome limitations in real-world datasets. we systematically analyze key factors in syncxr and validate its effectiveness through extensive ablation studies. in summary, we show that med-vlp achieves strong performance using a purely synthetic image-text dataset and benefits significantly from a combination of real and synthetic data. we believe this work will inspire the community to fully leverage synthetic data and mitigate the challenges posed by noisy and limited real-world datasets.