VPL : Visual Proxy Learning Framework for Zero-Shot Medical Image Diagnosis (2024)

Abstract: Vision-language models like CLIP, utilizing class proxies derived from class name text features, have shown a notable capability in zero-shot medical image diagnosis which is vital in scenarios with limited disease databases or labeled samples. However, insufficient medical text precision and the modal disparity between text and vision spaces pose challenges for such paradigm. We show analytically and experimentally that enriching medical texts with detailed descriptions can markedly enhance the diagnosis performance, with the granularity and phrasing of these enhancements having a crucial impact on CLIP’s understanding of medical images; and learning proxies within the vision domain can effectively circumvent the modal gap issue. Based on our analysis, we propose a medical visual proxy learning framework comprising two key components: a text refinement module that create high quality medical text descriptions, and a stable Sinkhorn algorithm for an efficient generation of pseudo labels which further guide the visual proxy learning. Our method elevates the Vanilla CLIP inference by supplying meticulously crafted clues to leverage CLIP’s existing interpretive power and using the feature of refined texts to bridge the vision-text gap. The effectiveness and robustness of our method are clearly demonstrated through extensive experiments. Notably, our method outperforms the state-of-the-art zero-shot medical image diagnosis by a significant margin, ranging from 1.69% to 15.31% on five datasets covering various diseases, confirming its immense potential in zero-shot diagnosis across diverse medical applications.

## introduction in the realm of vision-language models (vlms) @xcite @xcite @xcite *contributed equally. †corresponding author. refining medical texts brings predicted decision boundaries closer to the optimal. the sinkhorn optimization facilitates a shift from text to visual proxies, mitigating modal gap and aligning predicted boundaries with optimal ones. while our stable sinkhorn optimization, by addressing numerical instability, further enhances the algorithm's capacity for precise boundary approximation, thereby yielding better performance. 2024a; @xcite @xcite @xcite , large-scale pretrained models such as contrastive language-image pre-training (clip) have demonstrated exceptional capabilities across a spectrum of visual and linguistic tasks, particularly excelling in zero-shot recognition tasks @xcite . recent advancements have sought to adapt the principles underlying clip to the field of medical image analysis. for instance, initiatives like medclip signify an effort to tailor large-scale vlms for medical contexts, exploring applications in medical image classification @xcite . a pivotal area of exploration within this domain is zero-shot medical image classification, a task of paramount im-portance in real-world healthcare settings. this is especially relevant where exhaustive datasets covering all possible pathologies are unattainable, and annotated medical images are scarce @xcite @xcite . however, the application of existing vlms like clip to zero-shot medical image classification encounters significant challenges @xcite . first, models such as clip, which are pretrained on internet image-text pairs, may lack the specificity required for medical contexts, leading to suboptimal performance in medical applications @xcite . moreover, the efficacy of zeroshot inference using clip hinges on the model's ability to interpret category texts. these texts in medical imaging are often highly specialized and abstract medical terminologies, posing an intrinsic challenge for existing vlms @xcite @xcite . furthermore, recent studies @xcite have consistently demonstrated that there is a sustained discrepancy between image and text features in zero-shot classification even in general contexts, and current contrastive multi-modal learning paradigms like clip training appear to be insufficient in mitigating this disparity @xcite . these challenges, among others, highlight the imperative for devising specialized methodologies for the adaptation of vlms towards achieving efficacious zero-shot classification in the realm of medical imaging @xcite . efforts to counter the scarcity of medical imagetext pairs in vlms' pre-training datasets and the complexity of medical terminologies include enriching these models with detailed class descriptions @xcite . mirroring the diagnostic procedure of human experts, this involves leveraging human or generative model expertise to create thorough symptoms of medical conditions, unleashing the power of pre-trained vlms. preceding studies emphasize the importance of the text enrichment design, as the model performance is markedly influenced by the precision of descriptions or prompts for querying large language models @xcite @xcite . for our classification task, ensuring medical class descriptions are comprehensive, class-reflective, and distinct is crucial. the vanilla clip inference essentially translates image features into the text feature space @xcite , positioning these image features within the textual domain. however, recent discoveries have highlighted that the inherent discrepancies between image and text feature spaces may impede the efficacy of using text features as classification proxies. @xcite . a more effective approach involves finding proxies in the vision domain @xcite , bypassing the fundamental gap between the two spaces, a gap that is otherwise non-trivial to overcome. in clip inference, image labels are unknown, making the key to success the efficient use of clip-generated image and class text features for obtaining visual proxies. despite its potential, visual proxy learning using these features, particularly in medical image classification, is still underexplored, thus merits further investigation. in this paper, we propose a novel visual proxy learning (vpl) framework for zero-shot medical image diagnosis. specifically, we create specialized prompts for the expert to produce refined text descriptions for each disease, comprehensive and sufficiently distinct to harness the potential of vlms. furthermore, we propose a stable iterative algorithm based on sinkhorn algorithm @xcite that efficiently generates pseudo labels utilizing image features and those of the refined text. proxy learning in the vision space directed by these pseudo labels profoundly enhances the efficacy of text enrichment, taking its impact to a higher level. figure 1 showcases our improvements. the introduction of refined medical text facilitates a shift in proxies and decision boundaries within the text space towards the optimal state. subsequent to the recovery of visual proxies, stable sinkhorn, outperforming the standard, yields superior proxies and decision boundaries in the vision space. we verify the effectiveness of each design through extensive experiments. the main contributions are summarized as: proliferative retinopathy clip:text encoder clip: vision encoder expert ... description 1: the presence of more prominent retinal hemorrhages,venous beading... v 1 t 1 t 2 t 3 t k ... prompt class names coarse text descriptions refined text descriptions t 1 t 2 t 3 t k textual proxy visual proxy predicted labels stable sinkhorn optimization description k: the presence of more extensive areas of retinal hemorrhage, venous beading... text refinement k description 1: moderate cotton wool spots, intraretinal microvascular abnormalities, absence of neovascularization... description k: the presence of more extensive areas of retinal hemorrhage, venous beading... ... k pseudo label logits by textual proxy logits by visual proxy features for different classes predicted/optimal persudo labels / / predicted/optimal boundaries i 1 i 2 i 3 i k ... k i 1 i 2 i 3 i k . figure 2: model pipeline. we initiate by inputting a meticulously crafted prompt into chatgpt, which produces initial coarse text descriptions for each disease. these often contain overlapping information across different classes. then, our textrefinement module transforms these into distinct and clear text descriptions, with similar content highlighted in red and texts with reduced redundancy marked in green. following this, a text encoder generates textual proxies, which merged with the visual features from the image encoder, yield the predicted logits. finally, employing our newly developed stable sinkhorn algorithm, we enhance the visual proxies, culminating in more accurate final predictions. ## conclusion in this work, we focus on enhancing zero-shot classification of clip in medical image diagnosis. our analysis reveals that imprecise texts, along with the vision-text modal gap, are key impediments to current classification performance. to address these, we propose vpl, which consists of a strategy to refine medical texts and a method to recover class proxies in the visual space with the aid of medical textual proxies. specifically, medical category texts are refined via llms to alleviate semantic ambiguities between medical texts and images. then, by improving textual proxy predictions with unlabeled image data, visual proxies are learned through pseudo-labels, circumventing modal discrepancies. experiments across various datasets consistently demonstrate zero-shot accuracy improvement in clip through our approach.