Towards Injecting Medical Visual Knowledge into Multimodal LLM s at Scale (2024)

Abstract: The rapid development of multimodal large language models (MLLMs), such as GPT-4V, has led to significant advancements. However, these models still face challenges in medical multimodal capabilities due to limitations in the quantity and quality of medical vision-text data, stemming from data privacy concerns and high annotation costs. While pioneering approaches utilize PubMed’s large-scale, de-identified medical image-text pairs to address these limitations, they often fall short due to inherent data noise. To tackle this, we refined medical image-text pairs from PubMed and employed MLLMs (GPT-4V) in an ‘unblinded’ capacity to denoise and reformat the data, resulting in the creation of the **PubMedVision** dataset with 1.3 million medical VQA samples. Our validation demonstrates that: (1) PubMedVision can significantly enhance the medical multimodal capabilities of MLLMs, showing significant improvement in benchmarks including the MMMU Health & Medicine track; (2) manual checks by medical experts and empirical results validate the superior data quality of our dataset compared to other data construction methods. Using PubMedVision, we train a 34B medical MLLM **HuatuoGPT-Vision**, which shows superior performance in medical multimodal scenarios among open-source MLLMs. Our code and data are available at https://github.com/FreedomIntelligence/HuatuoGPT-Vision.

## introduction multimodal large language models (mllms), such as gpt4-v, show limited performance in medical applications, particularly in lacking visual knowledge specific to the medical domain @xcite . although there are some small-scale, high-quality datasets containing medical visual knowledge (lau et al., 2018; liu * benyou is the corresponding author. @xcite , scaling them up is challenging. additionally, there are privacy and licensing issues associated with medical data, further complicating matters. pioneering works @xcite @xcite utilize pubmed foot_0 for largerscale training for medical vision-language alignment. pubmed is favored because it contains medical images and surrounding text, which (i) encapsulate the forefront of human wisdom in medicine and (ii) are well-de-identified @xcite . however, models trained on pubmed are unsatisfactory, as they perform poorly compared to general mllms on medical multimodal tasks @xcite . this can be attributed to data noise in pubmed, which significantly affects multimodal performance @xcite . concurrently, llava-med @xcite ) uses a "blind" large language model (llm) to generate visual question answering (vqa) from the contextual text of pubmed images, achieving notable results. however, this approach might overlook visual information inherent in the medical images themselves as llms cannot perceive images as input, probably leading to the generation of misinterpreted or irrelevant answers. moreover, llava-med is limited to 56k medical vqa entries. thus, creating a higher-quality and larger-scale vision-language alignment dataset for medicine is essential. to close this gap, we meticulously select highquality medical image-text pair from pubmed, employing a proposed refined pipeline. utilizing 914,960 refined medical images and their corresponding text, we apply gpt-4v as the "unblinded" reformatter, contrasting the "blinded" reformatting used in previous works @xcite @xcite , to denoise the pubmed data. our method generates more aligned medical vqa data for medical multimodal alignment. consequently, we constructed a high-quality multimodal medical dataset with 1.3 million samples and name it as pubmedvision. our experiments validated pubmedvision in two key aspects: (1) it significantly enhances the medical multimodal capabilities of mllms, showing notable improvement in benchmarks such as mmmu health & medicine. llava-v1.5-llama-3-8b achieves the strongest performance among open-source mllms with pubmedvision ; ## pubmedvision 3.1 data collection to acquire a comprehensive dataset of pubmed medical images, we integrated previously compiled public data of pubmed images, specifically llava-med pmc (514k) @xcite , pmc-inline (11m) @xcite , and pmc-oa (1m) @xcite . although extensive, the majority of this merged data consists of charts and graphs from papers rather than medical images. therefore, we implemented a rigorous data filtering pipeline: (1) text filtering. a medical vocabulary was used to filter out data where the contextal text contains a sufficient number of medical terms. caption : "density of the cystic lesion is 2.4 hounsfield unit (hu)." inline mention: "only the head and uncinate segment of the pancreas was visualized and the hypodense unilocular cystic lesion was revealed at the head of pancreas ( fig. 3 ) ." than 336x336 pixels). a medical image classification model, trained on 1k manually labeled images and 10k mllm-labeled images, is used to identify medical images. (3) deduplication. using sentence-bert (reimers and gurevych, 2019) as the encoder, we obtained semantic embeddings of the image captions and filtered out images with overly similar contexts. for more details, please see appendix b. ultimately, we filtered out 914,960 medical images and their associated contextual text (captions and inline mentions). figure 3 illustrates the diversity of medical modalities and image regions covered by pubmedvision's images. these medical images are then used to sequentially construct 1.3 million vqa data points for medical alignment. ## conclusion in this study, we refined high-quality data from numerous medical image-text pairs on pubmed. we then employ mllm-powered reformatting method to enhance this data. in this way, we construct pubmedvision, a large-scale, high-quality medical multimodal dataset. experimental results show that pubmedvision significantly boosts the multimodal capabilities of mllms, with marked improvements on benchmarks. this inspires the idea that pubmed holds great potential to advance medical multimodal capabilities, with the key challenge being how to improve data quality, despite the presence of many non-medical images and poor descriptions. we hope that the proposed pubmedvision dataset can aid the development of medical mllms in the future.