Language over Labels: Contrastive Language Supervision Exceeds Purely Label-Supervised Classification Performance on Chest X - R ays (2022)

Abstract: The multi-modal foundation model CLIP computes representations from texts and images that achieved unprecedented performance on tasks such as zero-shot image classification. However, CLIP was pretrained on public internet data. Thus it lacks highly domain-specific knowledge. We investigate the adaptation of CLIP-based models to the chest radiography domain using the MIMIC-CXR dataset. We show that the features of the pretrained CLIP models do not transfer to this domain. We adapt CLIP to the chest radiography domain using contrastive language supervision and show that this approach yields a model that outperforms supervised learning on labels on the MIMIC-CXR dataset while also generalizing to the CheXpert and RSNA Pneumonia datasets. Furthermore, we do a detailed ablation study of the batch and dataset size. Finally, we show that language supervision allows for better explainability by using the multi-modal model to generate images from texts such that experts can inspect what the model has learned.

## introduction multi-modal models that understand text and images, as well as the relations between them, surged in performance due to the pioneering work of clip @xcite . through a contrastive loss based on language supervision, the model embeds matching text-image pairs closely in latent space. this enables various applications, such as image classification @xcite , object detection @xcite , semantic segmentation, @xcite , and text-to-image generation @xcite . as the clip models were trained on data scraped from the internet, they work remarkably well for data of the general domain and excel at tasks such as food @xcite , car brand @xcite , or animal classification @xcite . however, for more specialized tasks such as satellite image @xcite and cancer cell classification @xcite , they do not perform much better than a random guess @xcite . to make these models work for these tasks, they require adaptation to the specific domain. in this paper, we study the adaptation of clip models to the domain of chest x-ray images of the mimic-cxr @xcite @xcite dataset. we show that the clip model pretrained on data scraped from the internet @xcite does not transfer well to mimic-cxr. furthermore, two approaches to adapting the model are compared: contrastive language supervision (cls) and supervised fine-tuning (ft) on labels. we show that cls combined with linear probing performs better than only using ft on labels. furthermore, we show that the same language-supervised model can be used to achieve good performance with only a linear probe on other chest radiograph datasets without retraining. our first ablation study investigates the batch size, as the massive batch size of 32,768 used for the original clip training would impose an obstacle for any cls fine-tuning. we show that a small batch size is sufficient to achieve good cls performance. we also find that a batch size that is too large hurts performance, contrary to the findings of prior work @xcite @xcite . next to the large batch size, clip also used a large dataset of over 400 million image-text pairs. in a second ablation study, we investigate whether cls needs a large dataset size to outperform supervised learning. we show that cls can be superior to ft even with only 20,000 image-text pairs (10% of the mimic-cxr dataset). in the last experiment, we display how to get more interpretable neural network classifiers. the language-supervised model can compare the similarity of the features of a text and an image. through the gradient of the similarity towards the image, an image can be generated purely from a text. this generation allows clinicians and machine learning scientists to visualize model representations. this approach, inspired by clip-based text-to-image approaches such as vqgan-clip @xcite resembles the work of deep-dream @xcite . instead of visualizing classes or neuron activations, it visualizes texts. ## related work in a closely related work named convirt, @xcite train a model using cls on the imagetext pairs of the mimic-cxr dataset and compare it to supervised learning on the labels. their pioneering work partially inspired the creation of clip @xcite . our work is complementary to their work by using the widely adapted architecture and simplified loss function of clip, evaluating the performance of the openai-pretrained clip model on mimic-cxr, running ablation studies on the batch size and dataset size, and introducing the text-to-image visualization of diagnoses. in clip-art @xcite , clip was fine-tuned using cls on a large dataset of museum artworks with descriptions. the features of the fine-tuned clip model do not lead to a significantly better classification performance than the features of the base clip model. more related to the approach of this paper is pubmedclip @xcite . the authors fine-tune clip using the cls objective on image-text pairs from medical papers. they show that the pretrained clip features improve visual question-answering performance over the current state-of-the-art baseline. the continued pretraining using cls only slightly improves the performance over the base clip model. a current preprint follows a similar approach as our paper. @xcite compare the zero-shot performance of a model trained using cls-like loss functions to the performance of supervision on labels. their work confirms the benefits of training using language supervision over labels. however, their work focuses on selecting training data and loss functions. in contrast, our work analyzes batch sizes, dataset sizes, and an explainability approach. ## background this work investigates the effect of pretraining using cls on text-image pairs before ft on labels. we are given a set of images s, corresponding texts t , and labels y . for both the cls and ft stages, a network (in this case, a transformer @xcite network from @xcite first transforms its input into an encoding, leading to the encodings e text and e image . in the cls stage these encodings are improved by training the weights of both transformers using a contrastive loss, whereas the ft stage only uses the vision transformer and its encoding, followed by a linear layer. an overview of the two stages is given in figure 1 . for ft, a prediction ŷni of the target label @xmath0: f (s i ) = y ni . the binary cross entropy loss l bce is calculated per label y n and then averaged over all n labels to get the supervised loss l sl , as was done in previous work in multi-label settings @xcite . the bce loss l bce of label n is calculated using the ground truth y ni and the prediction y ni for sample i. it assigns a loss that is high initially and drops off logarithmically as the prediction approaches the ground truth: a similarity function sim(x, y): the total infonce loss is the average of all individual losses of the samples from the batch. we use the cosine similarity as a similarity function, as in the original clip paper. ## methods the code for training and evaluating is available online 1 . all models were evaluated using the macro average of the area under the receiver-operator curve (roc-auc or auc) @xcite averaged over all labels of the dataset. this metric was 1 https://github.com/notnanton/master_ thesis used to enable a comparison with prior work. for a clinical evaluation, the sensitivity and specificity should be studied in more detail. ## conclusion we show that cls with a simple linear probe outperforms ft on the mimic-cxr dataset, even when using small batch sizes on a single gpu. models trained using cls generalize to datasets of the same domain. cls outperforms ft for all dataset sizes down to 20,000 image-text pairs. the optimal batch size in our experiments was 576. furthermore, cls stopped being performant when using fewer than 20,000 training pairs. future work could investigate how the optimum batch size changes depending on the dataset size and if this critical dataset size is replicable for other datasets.