Strengths and Limitations of Word-Based Task Explainability in Vision Language Models: a Case Study on Biological Sex Biases in the Medical Domain (2025)

Abstract: Vision-language models (VLMs) can achieve high accuracy in medical applications but can retain demographic biases from training data. While multiple works have identified the presence of these biases in many VLMs, it remains unclear how strong their impact at the inference level is. In this work, we study how well a task-level explainability method based on linear combinations of words can detect multiple types of biases, with a focus on medical image classification. By manipulating the training datasets with demographic and non-demographic biases, we show how the adopted approach can detect explicitly encoded biases but fails with implicitly encoded ones, particularly biological sex. Our results suggest that such a failure likely stems from misalignment between sex-describing features in image versus text modalities. Our findings highlight limitations in the evaluated explainability method for detecting implicit biases in medical VLMs.

## introduction foundation and vision-language models (vlms) have found many successful applications in the general and medical domains @xcite @xcite @xcite @xcite @xcite @xcite . while powerful, vlms can encode harmful demographic biases and stereotypes @xcite @xcite @xcite , that can also expand to systems that rely on them as backbone structure, such as text-to-image models @xcite . recently, @xcite found similar patterns in the medical domain, showing how general and medical vlms can under-diagnose marginalized demographic groups, adopting bias learned from the training data. analogous evidences were found by multiple studies, which show how different types of machine learning models used in the medical field tend to encode and produce harmful biased predictions against underrepresented demographic groups @xcite @xcite . these results highlight the strong need for mechanisms to trace and quantify possible biased behaviours and knowledge encoded in vlms, especially when a validation set is unavailable for a given task. aside from tracing and mitigating biased distribution in training sets, and using adhoc metrics (see @xcite for a review), instance-level explainability (xai) methods based on saliency maps are among the most adopted methods to trace biases in vlms @xcite @xcite . while instance-based xai methods can effectively and intuitively convey their findings, they struggle to reveal broader patterns on how a model is systematically impacted during a classification task, across a full dataset. these limitations are addressed by conceptbased and task-level xai methods @xcite @xcite @xcite , which focus on gathering descriptions of the differences between a task's classes. since visual explanations can be less effective in conveying cross-category differences, @xcite have proposed a word-based task-level xai methodology leveraging a vlm's joint embedding space. the proposed approach aims at reconstructing the coefficients of a logistic regression, fit to discriminate between images of healthy and clinical patients, by learning a linear combination of word embeddings (see figure 1 ). intuitively, this will result in learning which subset of a pre-defined vocabulary is more descriptive of one category (e.g., disease patient) versus another (e.g., healthy patient). in their work, @xcite show how this approach can capture meaningful aspects of medical diagnosis, such as the one between the roundness of a skin lesion and the high likelihood of it being benign, or its asymmetry and the high probability of such lesion being malignant. in this study, we propose to further test such an approach, to trace and quantify more implicit features and biases encoded in both individual images and overall datasets. we do so with two experiments, both injecting controlled amounts and types of biases in an x-ray-based classification task. in the first experiment, we focus on explicitly quantifiable image characteristics, namely brightness and blurriness, while for the second experiment, we focus on controlling the association between a specific biological sex and the likelihood of such group of patients to be diseased or healthy. using both a general and a medical vlm, our results show how the adopted approach can detect biases that are explicitly encoded in the images (i.e., brightness and blurriness), but fails at detecting more implicit biases connected to biological sex imbalance in the data, producing incoherent predictions, with highly variable and inconsistent patterns that resist straightforward interpretation. these findings highlight the need for more robust methodologies before making definitive claims about bias quantification in medical vlms. ## related work demographic biases in vlms @xcite proposed the first multimodal analysis and metrics to detect and quantify demographic biases in vlms across the two modalities, showing how these biases are not only independently encoded in each separate modality, but can influence and propagate across modalities. @xcite study the effectiveness of data-balancing methods for debiasing vlms. results show that fine-tuning can be effective against some type of biases, though the impact on quality is not always positive. @xcite used gradcam @xcite , to show how clip @xcite encodes societal gender bias, for example by associating concepts like programmer to male figures, and gossipy or homemaker to female ones. @xcite found that a medical vlm for chest x-ray diagnosis consistently underdiagnosed marginalized groups, especially those with intersectional identities like black fe-male patients. crucially, the analysis of the word embedding reveals that the model consistently encoded demographic information with an accuracy exceeding human radiologists, creating bias across multiple pathologies and patient populations. ## bias statement from the medical and diagnostic perspective, we consider as bias the spurious association, created by the model, or contained in the data, between non clinically relevant traits or characteristics and disease likelihood. as demonstrated throughout the previous sections, such conditions appear to afflict medical datasets and ai models, manifesting through systematically different prediction rates across demographic groups when controlling for actual disease prevalence. these biases are harmful because they do not necessarily reflect real-world distributions @xcite , and can perpetuate or amplify existing health disparities through several mechanisms: 1) resource inequality: biased predictions lead to inequitable distribution of healthcare resources, with some demographic groups receiving more accurate diagnoses and timely interventions than others @xcite ; 2) care quality gaps: systematic performance differences compromise care quality for certain groups of people @xcite ; 3) system distrust: consistent misdiagnosis of certain demographic groups undermines trust in ai systems within those communities and potentially exacerbates historical mistrust in healthcare systems @xcite . 4) policy misalignment: if biased ai-systems were used to inform health policies (without awareness/quantification of the underlying biases), their results may fail in appropriately capturing actual population needs and in return might create regulatory gaps that undermine the goal of ensuring equitable healthcare. ## experimental set-up this work has two main experiments, both using the method proposed by @xcite . the first experiment is designed as a proof of concept or stress-test of the original work. the second experiment examines the method's ability to detect gender biases. both experiments have the same core process, models, base dataset, and list of explainable words. these aspects are explained in more detail in the following subsections. ## experiment 2: biological sex bias in this experiment, we test the ability of @xcite 's method to trace sex-based stereotypes. while biological sex may not be as immediately obvious as characteristics like roundness in images, certain sex-based anatomical features may still be detectable in chest x-rays, such as differences in breast tissue. ## discussion and conclusions a consistent body of evidence has shown how many ai models, including vlms, can encode harmful biases and stereotypes based on demographic features, such as ethnicity or biological sex. these biases have been shown to negatively impact the performance of these models, and it is hence essential to trace and quantify their impact at inference time, especially in a crucial field as medical decision-making. in our work, we have focused on a task-level approach to explainability, aiming at understanding if it can coherently trace explicitly (e.g. brightness) or implicit (i.e., biological sex) bias distributions that we have injected in a medical image classification task. our experiments, which use the task-level explainability method proposed by @xcite , reveal important limitations in this method for detecting implicit biases in medical vlms. while experiment 1 demonstrated the method's effectiveness in detecting explicit visual modifications like brightness and blurriness (see figure 2 ), experiment 2 exposed its failure to detect sex-based biases. despite deliberately manipulating the datasets to enhance the association between a specific biological sex and disease presence, the method failed to detect these manipulations in both clip and unimedclip models. such failure could indicate that the models are not using biological sex information in the classifi-cation process, so we performed a detailed analysis. results strongly suggest a fundamental issue: the misalignment between how biological sex is represented in these models versus how humans would interpret it. to start, the prototypical image analysis produced remarkably poor performance (see figure 4 ), indicating the system was essentially guessing patients' biological sex rather than detecting meaningful patterns. moreover, our qualitative investigation showed how clip's image encoder internal activations appear remarkably similar for the two sexes. while this evidence is in line with the basic assumption behind distributional modeling (i.e., similar concepts occupy a close position in the latent space), we notice how the "behaviour" of the model appears poorly aligned with our expectation on where we might focus to make a distinction between biological-sex in a chest image (see figure 6 ). such evidence might seem in contrast with the intuition that vlms might hold better and more grounded knowledge, thanks to their dual-modality modeling. however, recent preliminary evidence suggests that vlms might in fact, be less aligned with human internal representations @xcite . to conclude, this work presented an extensive analysis of the ability of a task-level explainability method based on linear combination of word embeddings to detect implicit and explicit biases by focusing on injecting quantifiable biases, such as brightness and blurriness altering, and more implicit biases, such as patients' biological sex. the first experiment's results are in line with the original work, showing that the system is able to detect imbalances in the data when they are related to explicit features. however, results from the second experiment showed how the method is not able to coherently detect implicitly encoded biases such as the biological sex. our analysis suggested that this is likely due to a misalignment of the concept in the two modalities.