M ed CLIP : Contrastive Learning from Unpaired Medical Images and Text (2022)

Abstract: Existing vision-text contrastive learning like CLIP aims to match the paired image and caption embeddings while pushing others apart, which improves representation transferability and supports zero-shot prediction. However, medical image-text datasets are orders of magnitude below the general images and captions from the internet. Moreover, previous methods encounter many false negatives, i.e., images and reports from separate patients probably carry the same semantics but are wrongly treated as negatives. In this paper, we decouple images and texts for multimodal contrastive learning, thus scaling the usable training data in a combinatorial magnitude with low cost. We also propose to replace the InfoNCE loss with semantic matching loss based on medical knowledge to eliminate false negatives in contrastive learning. We prove that MedCLIP is a simple yet effective framework: it outperforms state-of-the-art methods on zero-shot prediction, supervised classification, and image-text retrieval. Surprisingly, we observe that with only 20K pre-training data, MedCLIP wins over the state-of-the-art method (using 200K data). The code is available at https://github.com/RyanWangZf/MedCLIP.

## introduction medical images such as x-rays, cts, and mris are commonly used to diagnose, monitor, or treat medical conditions in clinical practice @xcite . with the rapid growth of medical images and the corresponding reports data, researchers have developed various deep learning models to support clinical decision making @xcite . recently, large-scale image-text pre-training, e.g., clip @xcite , has achieved considerable successes in computer vision and natural language processing domains. clip is trained to predict the correct matching of a batch of images 1 our code is available at https://github.com/ ryanwangzf/medclip . ## related works vision-text representation learning was shown to learn good visual representations @xcite @xcite @xcite . but all of them work on paired image and captions from general domain, e.g., flickr @xcite and coco captions @xcite . likewise, these methods do not support cross-modal retrieval hence do not support zero-shot predictions either. many propose to learn visual-semantic embedding for vision-text retrieval @xcite @xcite @xcite by attention or objection detection models; and by vision-text contrastive learning @xcite @xcite or multiple vision and text supervision @xcite . they all work on general domain where near infinite web images and captions are available, which dwarfs the scale of medical image-text data. this challenge hurdles the execution of self-supervised cl for large vision-text transformers. though remedies like data augmentation @xcite and knowledge graph @xcite were proposed, the magnitude of used data is still far larger than medical data. medical image-text representation learning was investigated based on contrastive learning as well @xcite @xcite . nonetheless, they all work on paired medical images and texts so still encounter the lacking data challenge. moreover, they all suffer from the false negative noises when adopting noise contrastive estimation (nce) @xcite to perform instance discrimination @xcite , which undermines the representation quality @xcite . our work bridges the gap by making the full use of all available medical data to support medical image-text pre-training. and we harness medical knowledge tailored to eliminate false negatives in contrastive learning to improve the pre-training data efficiency. ## method in this section, we present the technical details of medclip following the flow in fig. 3 . medclip consists of components (1) knowledge extraction that builds the semantic similarity matrix, (2) vision and text encoders that extracts embeddings, and (3) semantic matching loss that trains the whole model. ## experiments we conduct extensive experiments on four x-ray datasets to answer the following questions: mimic-cxr (johnson et al., 2019) is a large chest x-ray database with free-text radiology reports collected from the beth israel deaconess medical center in boston, ma. we use the training split of this dataset for pre-training. for evaluation, we also sample a mimic-5x200 dataset for the same five tasks above. covid @xcite is a publicly available x-ray dataset with covid v.s. non-covid labels. the positive and negative ratio is roughly 1:1. we use this dataset for evaluation. rsna pneumonia @xcite is a collection of pneumonia cases found in the database of chest x-rays made public by the national institutes of health. this is a binary classification dataset: pneumonia v.s. normal. we sample a balanced subset (i.e., 1:1 positive and negative ratio) and use it for evaluation. ## conclusion in this work, we propose a decoupled medical image-text contrastive learning framework named medclip. it significantly expands the training data size with a combinatorial magnitude. meanwhile, the introduction of medical knowledge sheds light on alleviating false negatives. as a result, medclip yields an excellent pretraining data efficiency: it wins over the state-of-the-art baseline by 1% acc with around 10× fewer data. moreover, we verify the prominence of medclip on zero-shot prediction, supervised classification, and image-text retrieval tasks. it is expected to support a foundational model for the medical domain and handle medical diagnosis when facing diverse diseases with low resource requirements.