Biases Propagate in Encoder-based Vision-Language Models: A Systematic Analysis From Intrinsic Measures to Zero-shot Retrieval Outcomes (2025)

Abstract: To build fair AI systems we need to understand how social-group biases intrinsic to foundational encoder-based vision-language models (VLMs) manifest in biases in downstream tasks. In this study, we demonstrate that intrinsic biases in VLM representations systematically “carry over” or propagate into zero-shot retrieval tasks, revealing how deeply rooted biases shape a model’s outputs. We introduce a controlled framework to measure this propagation by correlating (a) intrinsic measures of bias in the representational space with (b) extrinsic measures of bias in zero-shot text-to-image (TTI) and image-to-text (ITT) retrieval. Results show substantial correlations between intrinsic and extrinsic bias, with an average \rho = 0.83 \pm 0.10. This pattern is consistent across 114 analyses, both retrieval directions, six social groups, and three distinct VLMs. Notably, we find that larger/better-performing models exhibit greater bias propagation, a finding that raises concerns given the trend towards increasingly complex AI models. Our framework introduces baseline evaluation tasks to measure the propagation of group and valence signals. Investigations reveal that underrepresented groups experience less robust propagation, further skewing their model-related outcomes.

## introduction modern encoder-based vision-language models (vlms) such as clip @xcite and blip-2 @xcite excel at mapping images and text into a shared representational space. these models are the backbone of many zero-shot applications, from object recognition to image retrieval @xcite @xcite @xcite . however, it is increasingly clear that these representations may contain intrinsic biases @xcite @xcite for instance, systematically associating certain social groups with negative concepts. the potential for these biases to manifest in downstream applications can undermine fair and ethical treatment across social groups @xcite . there exists a significant gap in understanding how biases intrinsic to the representational spaces of foundational encoder-based vlms like clip @xcite and blip2 @xcite propagate to tasks they perform effortlessly without retraining or fine-tuning. concretely: the current research quantifies the propagation of biases in encoder-based vlms by comparing (a) intrinsic bias in the representational space learned by these models to (b) the outcomes of downstream zero-shot tasks, specifically text-toimage (tti) and image-to-text (itt) retrieval. in our study, the term "propagation" refers to the directional flow of biases from model representations to outcomes in downstream tasks. by employing a rigorously controlled experimental design, we minimize external confounding factors, enabling a clear observation of how biases are transferred within encoder-based vlms. while we recognise this does not fully establish causality, our findings provide robust evidence of systemic bias transfer, justifying the use of the term "propagation." to conduct our analysis, we curate balanced datasets comprising images from the chicago face database (cfd) @xcite and the open affective standardized image set (oasis) @xcite , capturing social group signals and valence signals, respectively. valence is a primary dimension in our analysis, as it captures biased attitudes critical to understanding bias manifestation in ai @xcite . in parallel, we develop experimental text datasets using semantically-neutral sentence templates (tan and celis, 2019), incorporating target words representing either valence signals from the nrc-vad lexicon @xcite or social group signals from prior research (charlesworth top 500 valence rated sentences retrieved spearman's correlation top 25 positive sentences top 25 negative sentences e.g. cfd image of black woman e.g. cfd image of black woman intrinsic bias in vision-language model representations extrinsic bias in vision-language model zeroshot retrieval outcomes sc-eat (d) , mean valence score sc-eat effect size mean valence score figure 1 : the setup to analyze valence-based bias propagation in image-to-text retrieval. propagation is measured by the correlation between the intrinsic bias (measured by valence sc-eat), and extrinsic bias (measured by mean valence of retrieved text) for images belonging to a social group. the intrinsic sc-eat effect size is computed based on the differential association of each cfd image with the top 25 positive and top 25 negative words in sentence templates. the extrinsic metric is derived from the mean valence of the top 500 words in sentence templates retrieved by each image. ground truth valence scores come from human rated sources @xcite ). spearman's ρ is used to compute the correlation between intrinsic and extrinsic metrics. @xcite . we measure bias propagation by quantifying both intrinsic and extrinsic biases in encoder-based vlms. intrinsic bias is assessed using the single category-embedding association test (sc-eat) @xcite , which evaluates associations between social groups and attribute association signals within the model's representational space. extrinsic bias is evaluated through downstream zero-shot tasks, such as itt and tti retrieval, to assess how these intrinsic biases manifest in practical applications. figure 1 demonstrates the setup to measure valence-based bias propagation in the itt setting. our study uniquely examines the propagation of both social group and valence biases in encoderbased vlms within controlled zero-shot settings, addressing the complexity of bias in multimodal contexts across multiple social groups. it may seem intuitive that intrinsic biases in a model's representations would naturally emerge in its downstream tasks, simply because they share the same embedding space. however, as we will demonstrate, this relationship is not trivial. different social groups (e.g., "black women" vs. "white men") can yield significantly divergent correlations. these systematic differences reflect how complex social cues such as "data marking" and group underrepresentation @xcite ) can affect the flow of biases. accordingly, the present work also contributes empirical clarity to how specific group and valence signals shape the degree of bias propagation in encoder-based vlms. our study makes the following contributions. foot_0 (1) we provide a controlled experimental framework for systematically quantifying bias propagation. the framework reinforces previous work showing biases in the representational spaces of encoder-based vlms. it also, for the first time, shows biases in the downstream zero-shot tasks of tti and itt retrieval as they directly relate to intrinsic biases. moreover, the framework and methods shown here are unique in enabling the study of bias propagation at scale -showing consistent conclusions across 114 scenarios, incorporating 6 social groups, 3 models, and 8 experiments, we show correlations are consistently high and significant (spearman's ρ of 0.83 ± 0.10). a related strand of work by investigates the origins of bias in vlms by examining the influence of pretraining factors. @xcite compare gender bias across 9 clip models and find that larger pretraining datasets often associated with better down-stream performance, also show the least bias. @xcite extend this study across 131 clip models, more than 20 architectures and pretraining datasets and find that the choice of the pretraining dataset and data filtering strategy used to filter it is the most significant predictor of intrinsic bias in clip-based vlms, over and above variables such as architecture choice, size of the pretraining dataset and number of model parameters. @xcite audit clip-based pretraining data filtering strategies and find that data related marginalized groups are subject to higher rates of exclusion compared to historically wellrepresented western-centric demographics. bias propagation in pretrained models. several studies have investigated how biases intrinsically present in pretrained language models can influence downstream model outcomes. @xcite highlighted a weak correlation between intrinsic and extrinsic fairness metrics for contextualized language models in supervised settings. @xcite investigated the bias transfer hypothesis, which posits that biases in large language models (llms) transfer into harmful task-specific behavior after fine-tuning. their findings indicate that reducing intrinsic bias before fine-tuning does little to mitigate discriminatory outcomes. however, it is crucial to note that the downstream tasks examined in such past work are not zero-shot. instead, these tasks involve fine-tuning on specific datasets which can lead the model to learn spurious associations inherent in those datasets. @xcite explored the propagation of political biases in language models. their work revealed that pretrained language models possess political leanings, influenced by politics in training corpora, which affects outcomes in downstream tasks. @xcite showed intrinsic biases do not consistently correlate with extrinsic biases in fine-tuned tasks of encoder-based vlms. they proposed genderneutral pretraining as a mitigation strategy and demonstrated its ability to reduce group disparities. however, their work also focused on finetuning outcomes and lacked a systematic framework to directly investigate bias propagation. in contrast, we focus on zero-shot tasks, where models are directly examined for bias propagation from intrinsic to extrinsic levels without additional fine-tuning, and models directly rely on their pretrained representations. we introduce a controlled framework to investigate biases across multiple vlms and social groups. this enables us to uncover previously overlooked systematic trends in bias propagation. ## data we employ carefully curated datasets that provide experimental control and ground truth values. chicago face database (cfd) @xcite : the cfd dataset consists of 597 self-identified images of men and women participants belonging to 4 different racial demographics, namely, black, white, latino/a and asian. this dataset was selected for its reliable, self-identified human face images with clear group signals, unlike fair-face @xcite , which lacks selfidentified labels and includes visually noisy images, potentially biasing annotations. we first expand the dataset to facilitate the scale and generalization of our experiments by creating within-group image morphs following the approach in @xcite as described in appendix a.1.1. we then randomly sampled images without replacement. open affective standardized image set (oa-sis) @xcite : provides 900 human valence-rated images, enabling the study of how vlms handle non-group-related valence signals from naturalistic image inputs. the images depict a broad spectrum of themes such as animals, humans, objects, and scenes. textual templates and lexica: our research requires controlled text datasets. semantically neutral sentence templates enable understanding how the valence and social group properties of target lexica embedded in them influence biases in vlms. as such, we employ lexica embedded in 6 such templates taken from @xcite for controlled analysis of how words/phrases, particularly those with valence or social group signals, influence biases in extrinsic vlm outputs. for experiments using valence-rated sentences, we employ the nrc-vad lexicon @xcite , comprising approximately 20,000 english words. an example sentence created from the valence lexicon "happy" is, "this is the word happy." for experiments with group signals, we employ a curated list of 864 group-label phrases derived from @xcite . an example sentence created from the group-label "black woman" is "this is the word black woman." we focus on 6 intersectional race and gender groups: "asian men," "asian women," "black men," "black women," "white men," and "white women" in our analyses. summary statistics of cfd, oasis, nrc-vad lexicon, and group-labels are present in for their description and usage details in our experiments, please refer to the appendix a. ## approach social group biases in this study are defined as the association between group signals (e.g., the representation of black women) and valence signals (e.g., the representation of positivity vs. negativity). this approach is motivated by established methods in the field of ai ethics, such as the embedding association test (eat) applied to embedding spaces @xcite , providing a scientifically robust framework for studying biases. valence is also integral to our analysis as it reflects the positive or negative connotations associated with group representations, and is the basic dimension along which biases are evaluated @xcite . ## discussion our study first quantifies the intrinsic biases in encoder-based vlms. next, across 8 experiments and 114 foot_2 analyses, the current work finds consistently across all approaches, positive and generally strong correlations (spearman's ρ of 0.83 ± 0.10) between intrinsic biases in vlm representational spaces and extrinsic bias outputs in zero-shot itt and tti retrieval tasks, with notable systematic differences relating to social group identity. ## conclusion this research investigated intrinsic biases propagation in itt and tti retrieval of encoder-based vlms. across 114 analyses focusing on six social groups, we demonstrated that bias propagate with high and systematically varying correlations (spearman's ρ of 0.83 ± 0.10 on average). larger and more complex models showed greater propagation of biases, a finding that is particularly salient given the deployment of increasingly larger ai models. our results imply detrimental downstream performance implications for marginalised groups that are misrepresented in these models.