P ub M ed CLIP : How Much Does CLIP Benefit Visual Question Answering in the Medical Domain? (2023)

Abstract: Contrastive Language–Image Pre-training (CLIP) has shown remarkable success in learning with cross-modal supervision from extensive amounts of image–text pairs collected online. Thus far, the effectiveness of CLIP has been investigated primarily in general-domain multimodal problems. In this work, we evaluate the effectiveness of CLIP for the task of Medical Visual Question Answering (MedVQA). We present PubMedCLIP, a fine-tuned version of CLIP for the medical domain based on PubMed articles. Our experiments conducted on two MedVQA benchmark datasets illustrate that PubMedCLIP achieves superior results improving the overall accuracy up to 3% in comparison to the state-of-the-art Model-Agnostic Meta-Learning (MAML) networks pre-trained only on visual data. The PubMedCLIP model with different back-ends, the source code for pre-training them and reproducing our MedVQA pipeline is publicly available at https://github.com/sarahESL/PubMedCLIP .

## introduction medical visual question answering (medvqa) seeks answers to natural language questions about a given medical image. the development of med-vqa has considerable potential to benefit healthcare systems, as it may aid clinicians in interpreting medical images and obtaining more accurate diagnoses by consulting a second opinion. thus, it has become a very active area of research, with competitive benchmarks and yearly competitions @xcite . yet, visual question answering in the medical domain in particular remains non-trivial, as we suffer from a general lack of large balanced training data, in part due to privacy concerns. to solve the multimodal task of medvqa, a system must understand both medical images and textual questions and infer the associations between them sufficiently well to produce a correct answer @xcite . thus, the success of these solutions is tied to the effectiveness of their visual and question encoders. current approaches for med-vqa adopt deep artificial neural network encoders to interpret the image and the question. previous studies in @xcite @xcite commonly exploit the mixture of enhanced visual features (mevf) model @xcite as their visual encoder to overcome data limitations. however, mevf is custom-tailored for the particular challenges encountered in the vqa-rad @xcite ) dataset, i.e., specifically designed for the organs present in this dataset, limiting its generalizability to other settings. in non-medical settings, recent work @xcite @xcite @xcite has shown improvements of visual encoders when learning from multimodal image-text pairs in comparison to learning from just visual images. among these approaches, the contrastive pre-training of language-image data in openai's clip @xcite has been particularly prominent. clip is trained using a vast number of image-text pairs acquired from the internet with close to zero additional human annotation. we argue that this is particularly promising for the medical domain, since data annotation requires expert medical knowledge, making it expensive and time-consuming. following clip, we investigate to what extent learning from publicly available medical image-text pairs without any further annotation can aid in the med-vqa task. to this end, we use image-text pairs obtained from pubmed articles to train a new version of clip called pubmedclip. we then examine the outcomes when incorporating pubmedclip into state-of-the-art medvqa methods, investigating whether clip benefits medvqa. to the best of our knowledge, this is the first study introducing a pubmed-optimized clip and assessing the effectiveness of its visual and textual encoders for vqa. unlike prior work on medvqa, pubmedclip is trained using medical images from a diverse range of body regions and is not restricted to only a few organs. we conduct extensive experiments on two medvqa benchmark datasets and employ diverse back-end visual encoders in pub-medclip. our experiments show that using pub-medclip as a pre-trained visual encoder improves previous models by up to 3%. our experiments further reveal question type distributional differences in the two medvqa benchmark datasets that have not been imparted in previous work and cause different back-end visual encoders in pubmedclip to exhibit different behavior on these datasets. ## pubmedclip our first step is to fine-tune the original generaldomain clip using medical image-text pairs. we refer to the fine-tuned version as pubmedclip. figure 1 (a ) shows an overview of the training procedure for pubmedclip. texts and images are encoded separately using clip, which we denote by e @xmath0, e @xmath1, respectively, for a batch of size b. for each image-text pair, a label @xmath2. the cosine similarities between text and image features are computed to represent the respective visual and textual logits ŷv , ŷt , i.e., following clip, we set λ = 0.5 to obtain the average of vision and language losses. for training pubmedclip, we drew on the radiology objects in context (roco) dataset @xcite . previous work @xcite @xcite also proposes large-scale multimodal datasets in the medical domain. however, they include images of only one imaging modality, i.e., x-ray, for a very limited number of body regions. in contrast, roco includes over 80k samples of diverse imaging modalities such as ultrasound, x-rays, pet scans, ct scans, mri, angiography, from various human body regions, e.g., head, neck, spine, chest, abdomen, hand, foot, knee, and pelvis. learning visual representations of diverse organs with various imaging modalities is valuable for a medvqa system, as it is expected to interpret images given such diversities. the imagetext pairs in roco stem from pubmed articles. the texts are taken from the relatively short captions (average length of 20 words) associated with images in the articles, which provide rich explanatory information about the content of images. in this work, the training and validation data splits from the original paper @xcite were used to train pubmedclip, with vit-b/32 vision transformer @xcite , resnet rn-50 @xcite , and rn-50x4 visual encoder back-ends. with respect to the maximum text length accepted by clip, which is 76, we trimmed any longer captions, while zero-padding shorter ones. pubmedclip was trained for 50 epochs with a batch size of 64, and adam optimization (kingma and ba, 2014) with a learning rate of 10 -5 . the trained models, source code as well as further implementation details are available online at https://github.com/sarahesl/pubmedclip . figures 2 and 3 show pca visualizations of the caption and image embeddings, respectively, for the roco validation set. comparing clip and pubmedclip embeddings, pubmedclip appears to obtain more semantic-aware visual and textual features with regard to body locations. for instance, looking at chest, abdomen, and head body locations, the corresponding embeddings form clusters for pubmedclip. however, the original clip embeddings are scattered without much separation. foot_0 4 pubmedclip for medvqa given a medvqa training dataset represented as our goal is to investigate the effect of employing pubmedclip as the pre-trained visual encoder in medvqa. to this end, we considered two prominent medvqa methods, mevf @xcite and qcr @xcite , that adopt maml as their pre-trained visual encoder and glove word embeddings followed by a recurrent neural network (rnn) as their question encoder. we substitute the pre-trained maml module in mevf and qcr with the pre-trained visual encoder from pub-medclip. a schematic architecture of our pipeline is shown in figure 1 (b ). the representative visual feature f v in this solution is the concatenation of the outputs of the pubmedclip network and the cdae encoder. the objective of cdae's encoder is to robustly encode the noisy version v ′ i of an image v i while the decoder learns to reconstruct the original non-noisy images. denoting the reconstructed image as v rec i , equation 3 defines the image reconstruction loss of cdae as the mean squared error. the multimodal pooling mechanism for combining f v and f q is ban @xcite to obtain the answer feature vector f a , as illustrated in figure 1 (b ). for answer prediction, which is a classification task in our case, a sigmoid layer preceding a binary cross-entropy loss is utilized in order to allow multiple correct answers per question. eq. 4 formulates the answer classification loss function. here, âi,@xmath3, where σ represents the sigmoid function. following ban @xcite ), the answer classifier m is a two-layer feed-forward network with relu activation. the objective of medvqa is to simultaneously minimize the error of answer classification and image reconstruction, denoted as: ## conclusion this work introduces pubmedclip, a pre-trained vision-language encoder for the medical domain trained via contrastive learning of medical imagecaption pairs from pubmed articles. we demonstrated that pubmedclip results in organ-aware vision and language embeddings and evaluated its effectiveness for the task of medvqa in comprehensive experiments across two heterogeneous med-vqa benchmarks. while pubmedclip's text encoder is found to be less powerful for medvqa, we showed that pubmedclip's visual encoder outperforms previously used pre-trained visual encoders by up to 3%, leading to state-of-the-art results.