Intrinsic Bias is Predicted by Pretraining Data and Correlates with Downstream Performance in Vision-Language Encoders (2025)

Abstract: While recent work has found that vision-language models trained under the Contrastive Language Image Pre-training (CLIP) framework contain intrinsic social biases, the extent to which different upstream pre-training features of the framework relate to these biases, and hence how intrinsic bias and downstream performance are connected has been unclear. In this work, we present the largest comprehensive analysis to-date of how the upstream pre-training factors and downstream performance of CLIP models relate to their intrinsic biases. Studying 131 unique CLIP models, trained on 26 datasets, using 55 architectures, and in a variety of sizes, we evaluate bias in each model using 26 well-established unimodal and cross-modal principled Embedding Association Tests. We find that the choice of pre-training dataset is the most significant upstream predictor of bias, whereas architectural variations have minimal impact. Additionally, datasets curated using sophisticated filtering techniques aimed at enhancing downstream model performance tend to be associated with higher levels of intrinsic bias. Finally, we observe that intrinsic bias is often significantly correlated with downstream performance ( 0.3 \leq r \leq 0.8 ), suggesting that models optimized for performance inadvertently learn to amplify representational biases. Comparisons between unimodal and cross-modal association tests reveal that social group bias depends heavily on the modality. Our findings imply that more sophisticated strategies are needed to address intrinsic model bias for vision-language models across the entire model development pipeline.

## introduction neural network models are prone to learning patterns based on statistical associations between con-* equal contribution. cepts within their training data that might lead to harmful bias when it relates to social groups or model performance @xcite . this phenomenon has been observed in a number of vision and language models, each of which are unimodal and learn information within a single modality @xcite @xcite @xcite . cross-modal models, which learn information from both vision and language modalities, also learn biased information relating to social group associations @xcite @xcite @xcite @xcite @xcite . these results have largely been found using intrinsic bias tests adapted from natural language processing (nlp): evaluations that compare relative distances between a model's representations of stimuli representing different concepts and social groups. despite their prevalence in model evaluation, however, there is limited work connecting them to other factors of model design and optimization. for example, upstream factors such as training datasets, model architectures, and model sizes directly determine the representations learned and consequently may be reflected in intrinsic bias tests. these representations are then directly used for downstream tasks such as zero-shot image classification, which suggests a potential connection between the intrinsic bias of a model and its performance on downstream tasks. by investigating intrinsic bias as it explicitly relates to these two upstream pre-training and downstream zero-shot performance factors, we are able to draw novel insights about the ways in which models can be optimized to reduce harmful or undesirable biases. we measure the associations between social groups and valence, the pleasantness or unpleasantness of a concept @xcite . valence is a robust dimension of human cognition as it relates to shaping attitudes and biases @xcite . social groups and valence associations also exist in unimodal models @xcite . in this work, we examine whether these associations are observed cross-modally and how they relate to upstream factors and downstream model performance. to our knowledge, our work is the first which connects 26 tests of intrinsic bias, including those related to race, gender, age, and baseline associations with respect to non-social group concepts such as flowers and insects, to upstream factors of model training (including 26 training datasets, 55 model architectures, and model sizes ranging between 100 million and 5 billion parameters). while variations on all of these features have been proposed to either mitigate biases or improve performance on downstream tasks, we are the first to address the variance in the magnitude of the effect of these features in clip models. additionally, we connect intrinsic bias tests to a suite of 35 zero-shot image classification and retrieval tasks @xcite . a novel contribution of our work is that we show that optimizing for performance is not sufficient to mitigate intrinsic biases. the scale of our experiments allows us to obtain high statistical power in analyzing the relationship between intrinsic bias and both upstream factors and downstream performance and making the following generalizable knowledge contributions about cross-modal models 1 : flowers insects pleasant unpleasant figure 1: we use embedding association tests, bias evaluation methods for representational or generative models, to quantify biases in 131 clip models. images shown are a subset of the stimuli used to represent the concepts in the image embedding association test (steed and caliskan, 2021) and our controlled attribute stimuli taken from kurdi et al. (2017). distances shown illustrate a stereotype congruent bias (similar to that found in humans), where images representing the concept flower are closer to images representing the concept pleasant, and images representing the concept insect are closer to images representing the concept unpleasant. ## background and related work we now introduce background information on clip models and methods for measuring intrinsic bias in these models. @xcite introduced eats to measure the associations between static word embeddings which encode concepts related to targets (typically social groups) and attributes, similar to implicit association tests (iats) for human associations @xcite . as contextual word embeddings replaced static word embeddings, alternate methods to measure intrinsic bias and associations were also developed @xcite . the sentence encoder association test (seat) introduced by @xcite measures intrinsic bias by operating over a set of target sentences and attribute sentences which are semantically bleached excluding the words which represent the concepts of interest. ## experimental setup we now describe the experimental setup and data for upstream pre-training factors, intrinsic bias, and downstream performance. ## approach we describe the method we use for quantifying biases in clip models, as well as the regression models we use for exploring the relationship between biases, upstream pre-training factors, and downstream performance. measuring intrinsic bias we measure intrinsic bias using eats @xcite @xcite @xcite , which provide a generalizable and principled method for quantifying biases related to a variety of concepts, such as race and gender, grounded in literature in cognitive and experimental psychology @xcite ). an eat compares similarities between four sets of embeddings created by a model: two sets of target embeddings which represent social groups, denoted x and y , and two sets of attribute embeddings which represent valence denoted a and b as described in section 3. each eat gives an effect size d, whose magnitude indicates the strength of the bias, calculated as follows: and cos refers to cosine similarity, a distance metric used in quantifying associations by capturing information overlap between embeddings. we order the sets of stimuli such that a positive d value indicates a bias that is congruent with a stereotype that has been documented in society (i.e. flowers, instruments, women, european american, and young are more associated with pleasantness). ## experiments and results following @xcite and @xcite , we compute the eat effect sizes following the seat and ieat methods across the 131 models and 26 tests for a total of 3,406 data points. our analysis spans four modality combinations: all text, all image, image as target, and text as target, across various bias tests including 'flower-insect/valence', 'instrument-weapon/valence', 'gender/valence', 'race/valence', and 'age/valence' (as introduced in section 3). the eat effect sizes computed using stimuli from the original seat and ieat studies demonstrate a high overall variance of 0.62. in an effort to offer more control in our experiments thereby making the effect sizes more comparable across models and reducing the impact of outliers, we recompute them across using the newly controlled and grounded attribute stimuli for our tests, drawn from the nrc-vad lexicon @xcite and oasis @xcite datasets. we observe reduced effect size variance across models and tests to 0.59 overall (a 4.8% decrease) when replacing seat and ieat attributes with our new attribute stimuli. the decrease in variance was particularly pronounced for the all text modality (a 33.96% reduction in variance), which suggests that our set is less susceptible to noise and idiosyncrasies that may have plagued previous test sets. furthermore, changing the stimuli shows better alignment with human stereotypes, showing a significant effect size (@xmath0.2) in 70.23% of the 3406 instances using the new stimuli, while this number is lesser at 67.88% using the old stimuli. we note that even after using more controlled stimuli, a high aggregate variance is not surprising, due to the scale and nature of the study. our experiments in subsequent sections investigate this variation from known dataset choice and model architectural sources. instead, by employing these carefully curated and grounded stimuli, we gain a clearer lens through which to examine the underlying biases present in various models. consequently, we present all further results using these new stimuli, providing more robust and generalizable insights into bias in vlms. eats as an aggregate measure of bias as shown in figure 2 , 17 out of 20 eats reveal a pattern of associations which aligns in directionality to results from implicit association tests taken by humans (shown by a positive effect size). 2 only in the all image modality for 'age/valence' and the image as target for 'race/valence' and 'age/valence' are the effect sizes negative, representing associations opposite from those of humans. as with humans, 'flower-insect/valence,' and 'instrument-weapon/valence' show the largest effect sizes across modalities (@xmath1, and associations between valence and social groups are weaker but still present. in all cases, the magnitude of effect sizes varies depending on the modality. our findings indicate that biases in clip models generally align with those found in human assessments in 78.86% of the 3,406 cases (where @xmath2. in appendix c.0.1, we show that the direction of effect size across groups is consistent with the original seat and ieat stimuli. figure 2 also contains error bars that represent the standard deviation of effect sizes for the different test categories and modality combinations. we note that there is consistently lower variance for the non-human baselines, such as 'flower-insect/valence' and 'instrument-weapon/valence', compared to the variance observed in social bias categories like 'age/valence' and 'race/valence' indicating that these categories are inherently more susceptible to variability, likely due to the complexity and diversity of social concepts across different training datasets. ## discussion in this work, we explore how intrinsic biases and associations in vlms are influenced by the interaction between modalities and upstream pretraining factors, and their impact on downstream task performance. our findings highlight that the magnitude of bias effect sizes depends on the modality combination and test category being observed. however, across the board, the effect sizes are significantly influenced by dataset selection and correlate with model performance on downstream tasks. these insights highlight existing pitfalls in the data and training pipelines of vlms with respect to fairness considerations and provide important implications for mitigating biases in the future development of these models. bias in cross-modal interactions our analysis reveals that intrinsic biases manifest differently across combinations of text and image modalities. the 'flower-insect/valence' and 'instrument-weapon/valence' tests show consistently high effect sizes across all modality combinations, indicating a strong association that is unaffected by modality. 'gender/valence' biases show positive effect sizes in the all image setting, aligning with previous findings that women are associated with more positive terms than men @xcite . the representation of bias in models also varies substantially depending on the category of bias and the modality combination. notably, for 'age/valence', the direction of the effect sizes differs based on the modality. when analyzing images, older individuals are associated with more positive valence, whereas in text, younger individuals tend to be more positively associated. this discrepancy highlights a critical aspect of modalityspecific bias propagation, where the amount and type of information conveyed through visual modalities can differ from that in textual modalities, leading to distinct biases. such modality-dependent differences in the representation of 'age/valence' suggest that crossmodal vlms are influenced by biases specific to each modality. text stimuli often use older names that are less frequent and therefore may not be represented accurately @xcite , while images can convey richer, more nuanced information about age, potentially leading to different bias patterns. ## conclusion in this work, we conducted the largest analysis to date on the biases in vision-language models, examining 131 unique clip models across 26 datasets and 55 architectures. our study highlights that the choice of dataset during pre-training, particularly those curated using automatic and heuristic-based filtering approaches that optimise downstream vlm performance, significantly influences intrinsic bias, reinforcing existing disparities. additionally, we found that biases in models often correlate with improved downstream task performance, across modality settings, suggesting that the possibility that performance optimization can inadvertently amplify certain intrin-sic biases as vlms learn stronger associations between concepts. these findings emphasize the need for more ethically informed dataset curation and bias mitigation strategies to ensure fairer ai models. we release our code and data at https: //github.com/kshitishghate/clip_bias .