Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering (2024)

Abstract: Vision-language models, while effective in general domains and showing strong performance in diverse multi-modal applications like visual question-answering (VQA), struggle to maintain the same level of effectiveness in more specialized domains, e.g., medical. We propose a medical vision-language model that integrates large vision and language models adapted for the medical domain. This model goes through three stages of parameter-efficient training using three separate biomedical and radiology multi-modal visual and text datasets. The proposed model achieves state-of-the-art performance on the SLAKE 1.0 medical VQA (MedVQA) dataset with an overall accuracy of 87.5% and demonstrates strong performance on another MedVQA dataset, VQA-RAD, achieving an overall accuracy of 73.2%.

## introduction vision-language models (vlm), composed of two key elements -vision models and language models, mainly establish a connection between text-based and image-based modalities. in order to accomplish this fusion, vlms undergo training using large volumes of text and images. this training process enables them to understand the correlations between visual and textual data, thus equipping them to handle tasks such as visual question answering (vqa). vision-language models, such as clip @xcite and blip-2 @xcite , have shown impressive performance across various multi-modal applications. nevertheless, these vlms have not displayed similar levels of performance when applied to the medical vqa (med-vqa) task @xcite . the complexity of medical questions in medvqa often requires a deep understanding of medical terminology and image context that may not be adequately captured by a generic vlm. therefore, recent approaches, such as pubmedclip @xcite , med-flamingo @xcite , llava-med @xcite , and biomed-clip @xcite adapt general-domain vlms to the medical domain by leveraging large datasets containing both medical images and accompanying text, such as roco @xcite . moreover, prior approaches, including pubmed-clip @xcite and the models studied by @xcite , treated medvqa as a classification problem, where the models had to choose the correct answer from a predefined set. this approach not only restricts the ability of vlms to generate free-form responses but also leads to inaccurate evaluation. in this paper, we first define the medvqa task as free-text generation, which is considered a more challenging task compared to classification. next, we present a novel vision-language model that fuses a domain-specific large language model (llm) customized for radiology with a vision model designed for biomedical tasks. in the proposed vision-language model, all parameters of both the vision and language models remain fixed. we propose a parameter-efficient training approach by integrating low-rank adaptation (lora) technique @xcite for training the model. the frozen domain-adapted models and lora training ensure not only stability and consistency during training but also optimize the overall efficiency of the training process. our proposed training approach for the trainable parameters consists of three stages: medical concept alignment through the image-captioning task using pmc-oa dataset @xcite , adaptation to the general medical vqa task using the pmc-vqa dataset @xcite , and fine-tuning on the radiology task specific training dataset, such as vqa-rad @xcite and slake 1.0-english @xcite . we conducted evaluations on two public radiology medvqa evaluation benchmarks, vqa-rad @xcite and slake 1.0 @xcite , to assess the performance improvement achieved by our proposed vlm. our model outperformed existing models from published works on the slake 1.0 benchmark, achieving an impressive overall accuracy of 87.5%. furthermore, our model demonstrated strong performance on the vqa-rad benchmark, highlighting its effectiveness compared to other published models. additionally, we conducted a performance comparison between our model and a version that incorporates a general-domain llm while keeping all other components constant. we observed a big performance improvement with the domain-adapted language model, and thereby demonstrating the advantage of integrating these models into vlms as a promising approach to address the limitations of adapting general vlms to domain-intensive applications. lastly, in our ablation investigation, we evaluated the effect of our proposed multi-stage training approach and found that it led to a significant 25% improvement in accuracy compared to directly finetuning a general-domain vlm on the downstream medvqa task. our analysis underscores the advantages of incorporating a domain-specialized llm into the vlm architecture and highlights the effectiveness of our proposed training strategy in addressing medvqa tasks. our contributions can be summarized as follows: the remaining paper is structured as follows. in section 2, we provide a detailed description of the model with its training schema. in section 3, we describe and discuss the dataset and experiments. in section 4, we discuss the related works. section 5 concludes the study. ## model problem formulation: given a medical image v i and a natural language question q i , a trained vlm model m with parameters θ generates the answer a i for the given question as: figure 1 shows our vlm model architecture. our model includes a vision encoder that takes in the image v @xmath0, where h, w , and c denote the height, width, and channels of the image, respectively. it outputs the encoded image e(v) ∈ r n×m , with an embedding size of m and n number of patches. in our vlm model, the fusion module serves the purpose of mapping the encoded vision features e(v) to the embedding space of the llm. this module acts as a bridge between the vision encoder and the llm. taking inspiration from blip-2 @xcite , we employ a learnable query transformer architecture as the fusion module. its primary function is to extract a predetermined set of features from the output of the vision encoder. the parameters of this module are randomly initialized. the query transformer output is transformed using a multi-layer perceptron network to match the embedding size of the llm, resulting in e(v) ′ ∈ r d . these projected features are then combined with the embedded input text e(q) ∈ r d and fed into the llm to generate the desired output. in order to explore the potential benefits of incorporating radiology domain-adapted language and vision models in medvqa tasks that involve radiology images, questions, and answers, we utilize decoder-only transformer models as the llm module. more specifically, we leverage radbloomz-7b @xcite , which is a radiology domain adaptation of bloomz-7b1 @xcite . the radbloomz-7b model has been continuously pre-trained using the mimic-iv radiology reports dataset @xcite and has demonstrated exceptional performance on the radiology report summarization task, surpassing other models on the mimic-iii @xcite , mimic-cxr @xcite , and chexpert @xcite summarization datasets. we argue that radbloomz-7b offers a highly powerful foundation model and brings valuable advantages to downstream medvqa tasks. to investigate the potential advantages of integrating domain-specific vision models into med-vqa, we utilize the vision encoder models from pmc-clip @xcite and biomedclip @xcite . these models have demonstrated notable performance enhancements in multimodal medical tasks, including question-answering. by employing these models, we not only have access to two different pre-trained vision models but also have the opportunity to explore two distinct architectures: resnet50 @xcite from pmc-clip @xcite and vision transformer (vit) from biomedclip @xcite . in our model, the vision encoder and llm remain as pre-trained models with frozen parameters. instead, we propose using the low-rank adaptation (lora) technique @xcite on the pre-trained llm to align it with the downstream medvqa task. ## background and related work language models (lms) designed for general domains often face difficulties when applied to highly specialized fields. additionally, data scarcity is a prevalent challenge in domain adaptation of lms. various methods have been developed to adapt pretrained lms to specific domains. one method involves continuous pre-training of model parameters using data specific to the target domain @xcite . alternatively, synthetic data can be effectively incorporated into the training process for fine-tuning models to better adapt to specific target domains @xcite . another approach includes using parameter-efficient fine-tuning methods @xcite with task-specific training data. our training schema amalgamates several of these methods like image-caption pretraining, synthetic biomedical mqa, and task-specific med-vqa datasets, all fine-tuned using a parameterefficient technique. among parameter-efficient fine-tuning approaches, the low-rank adaptation (lora) technique (hu et al., 2021) has received considerable interest for adapting large lms (llms). in the biomedical domain, domain-specific llms have been proposed either by fine-tuning the model's parameters @xcite or by utilizing lora techniques @xcite . however, it's important to note that biomedical domain-adapted llms might not perform as effectively in the radiology domain. this is due to the complexity of terminologies in clinical nlp @xcite . thus, there have been recent proposals for radiology domainadapted llms @xcite . the application of domain adaptation is not limited to llms. it also finds utility in the adaptation of multi-modal models like vision-language models (vlms). in line with this, there have been recent proposed biomedical vlms such as @xcite @xcite @xcite . these have been successful in achieving state-of-the-art performance in downstream biomedical tasks, such as medical question-answering. in this study, we concentrate on developing a more efficient domain adaptation technique for vlms within the challenging domain of radiology. ## conclusion we introduce a new vision-language model for medical visual question-answering by integrating a radiology large language model, radbloomz-7b @xcite and a biomedical vision encoder, biomedclip-vit @xcite , in to the vlm. our main objective is to investigate the impact of integrating specialised lms and vision encoders into vlms for domain-specific tasks in the medical domain. for this purpose, we propose a parameterefficient training approach by deploying low-rank adaptation technique @xcite decoder-only llm component in the vlm, which significantly reduces the number of trainable parameters while maintaining the model performance. moreover, the vision encoder is kept frozen in the training process. we then propose a two-stage pre-training approach aiming to align our vlm to medical concepts by pre-training the model on the image-captioning task and acquiring general knowledge for medical visual question answering by pre-training it on a general medvqa dataset. we finally finetune the model on the downstream medvqa tasks. our results demonstrate state-of-the-art performance on a medvqa slake 1.0 dataset and strong performance on the vqa-rad dataset. furthermore, compared to a vlm with a generaldomain llm, we show that our proposed vlm leads to a higher performance using parameterefficient training, while a vlm with generaldomain lm benefits slightly from training the vision encoder as well. finally, our findings suggest that the proposed pre-training approach significantly improves model performance in downstream medvqa tasks.