The Bias Amplification Paradox in Text-to-Image Generation (2024)

Abstract: Bias amplification is a phenomenon in which models exacerbate biases or stereotypes present in the training data. In this paper, we study bias amplification in the text-to-image domain using Stable Diffusion by comparing gender ratios in training vs. generated images. We find that the model appears to amplify gender-occupation biases found in the training data (LAION) considerably. However, we discover that amplification can be largely attributed to discrepancies between training captions and model prompts. For example, an inherent difference is that captions from the training data often contain explicit gender information while our prompts do not, which leads to a distribution shift and consequently inflates bias measures. Once we account for distributional differences between texts used for training and generation when evaluating amplification, we observe that amplification decreases drastically. Our findings illustrate the challenges of comparing biases in models and their training data, as well as evaluation more broadly, and highlight how confounding factors can impact analyses.

## introduction breakthroughs in machine learning have been fueled in large part by training models on massive unlabeled datasets such as the pile, c4, and laion @xcite @xcite . however, several studies have shown that these datasets exhibit biases and undesirable stereotypes @xcite @xcite , which in turn impact model behavior. given that models are trained to represent the data distribution, it is not surprising that models perpetuate biases found in the training data @xcite @xcite . to introduce bias amplification, let us take a model that generates images of engineers that are female 10% of the time using a gender-neutral figure 1 : comparing generated and training images for engineer, the model clearly seems to amplify bias by generating 10% female images, as compared to 25% female in training images. however, when looking at the subset of training examples without gender indicators in captions (10% female), similar to our prompts, the model does not amplify bias. the bias amplification paradox given that models learn to fit the training data by maximizing its likelihood, why do models amplify biases found in the data as opposed to strictly representing them? in this paper, we investigate how model biases compare with biases found in the training data. we focus on the text-to-image domain in english and analyze gender-occupation biases in stable diffusion @xcite , as well as its publicly available training dataset laion @xcite , which consists of image-caption pairs ( §2). to select training examples, we identify captions that mention occupations (e.g., engineer) and obtain their corresponding images. we follow previous work @xcite and use prompts that contain a given occupation (e.g., "a photo of the face of an engineer") to generate images. for each occupation, we then classify binary gender to measure bias in corresponding training and generated images, and compare the respective quantities to determine whether the model amplifies biases foot_1 from its training data ( §3). at first glance, it appears that the model amplifies bias considerably (on average, generation bias is 12.57% higher than training bias) using existing approaches ( §4). when comparing training captions and prompts, however, we discover clear distributional differences that impact amplification measurements. for example, one inherent distinction is that captions often specify explicit gender indicators while prompts used to study genderoccupation biases do not. 3 more generally, captions may contain additional context and details that are absent from the prompts we use. based on our observations, it is clear that directly using all training captions that mention a given occupation provides a naive characterization of bias amplification. instead, we propose evaluating amplification on subsets of the training data that reduce distribution shifts between training and generation ( §5). we introduce two approaches to account for distributional differences: (1) excluding captions with explicit gender information and (2) using nearest neighbors (nn) on text embeddings to select training captions that closely resemble prompts. both approaches restrict the search space of training texts to more closely match prompts, which results in considerably lower amplification measures. we then eliminate differences between training captions and prompts by utilizing the captions themselves to generate images ( §6), and show that amplification is minimal. by modifying subsets of captions and prompts used to evaluate amplification, we perform a multi-pronged analysis of distribution shifts that impact evaluation. to summarize, we study gender-occupation bias amplification in stable diffusion and highlight notable discrepancies between texts used for training and generation. we demonstrate that naively quantifying bias provides an incomplete and misleading depiction of model behavior. our work emphasizes that comparisons of dataset and model biases should factor in distributional differences and evaluate comparable distributions. we hope that our work encourages future studies that analyze model behavior through the lens of the data. ## experimental setup before discussing how we define and evaluate amplification in the following section, we first outline the dataset and models in our experiments, as well as how we infer gender from images. ## estimating bias amplification: a baseline we examine the extent to which stable diffusion amplifies gender-occupation biases from the data by selecting training examples that contain a given occupation in the caption (e.g., all captions that contain the word "president"). in practice, we randomly sample a subset of 500 training examples containing each occupation. we find that stable diffusion amplifies bias relative to the training data by 12.57% 9 on average across all occupations and prompts (10.24% for prompt #1, as shown in figure 2 ). this behavior is concerning because instead of reflecting the training data and its statistics, the model compounds bias by further underrepresenting groups. however, when qualitatively inspecting examples, we observe discrepancies in how occupations are presented in captions vs. prompts due to varying levels of ambiguity. for example, we notice the use of explicit gender indicators to emphasize deviations from stereotypical gender-occupation associations, such as female engineers in figure 1 . while gender information is used frequently in captions, we hypothesize that usage is more common for underrepresented groups. if this hypothesis holds, the gender distribution would shift closer towards balanced in resulting training images. as a result, the decision to focus 9 we report values for stable diffusion 1.4 throughout the paper, but results for both model versions are presented in figure 2 : bias is amplified consistently using our baseline approach (by 10.24% for prompt #1). the xaxis corresponds to the % female in training images, and the y-axis corresponds to the % female in generated images (using prompt #1). each point represents an occupation. shading: amplification and de-amplification. on all captions vs. captions without any gender indicators can exaggerate amplification measures, as we see in section 5.1. more generally, prompts commonly used to study gender-occupation bias are intentionally underspecified, or lack detail. underspecification results in the model having to generate images from textual inputs that are vague and open to interpretation @xcite . for instance, the prompt "a photo of the face of a/an [occupation]" does not contain any adjectives or information about surroundings, activities, etc. in contrast, captions from the training data may contain context and details that result in less ambiguous descriptions, as shown in to compare bias at training and generation, we need to consider gender ratios for similar captions and prompts. therefore, we cannot conclude whether differences in gender ratios are due solely to the model amplifying bias, or other confounding factors that contribute to amplification. next, we focus on decreasing the impact of distribution shifts on bias amplification evaluation. ## removing distributional differences: a lower bound the previous approaches reduce discrepancies between training and generation by evaluating amplification with captions that are more similar to prompts. instead, we can focus our efforts in the other direction and modify the prompts we use to align with captions more closely. one way to achieve this is to eliminate differences altogether by making prompts and captions identical. we then ask: does using identical texts to measure training and generation bias lower amplification? we use the original training subset (s o ) from section 4 and make the prompts (p o ) match the captions verbatim. in this setup, we generate 10 images for every prompt in p o , and then compute amplification using p o := s o for each occupation. we hypothesize that enforcing prompts and captions to match yields similar bias measurements, which reduces amplification. as shown in figure 5a , amplification is small when p @xmath0. the average amplification drops to 0.68%, indicating that the model mostly reflects training bias. 13 furthermore, amplification remains consistently low, even for highly imbalanced occupations. 13 however, we reject the null hypothesis that the expected amplification is 0 using a one-sample t-test. for captions that contain either male or female gender indicators, the model generates images that match the gender of corresponding training images (with 98.41% accuracy), since this information is directly provided in the prompt. therefore, we analyze the results separately on the subset of captions without gender indicators. as shown in figure 5b , bias amplification is larger for the no gender indicator subset as compared to all captions. that being said, the average amplification remains low at 2.05% (↓ 84% relative to our naive approach). 13 we also observe similar results when paraphrasing the original training captions and using these texts as prompts, as discussed in appendix a.6, which suggests our results are not simply due to memorization of captions. although practitioners are unlikely to utilize prompts that exactly match training captions, this experiment highlights the impact of distributional similarity between captions and prompts when comparing biases. in addition, it provides a lower bound to the bias amplification problem. in summary, we conclude that the model nearly mimics biases from the data when we eliminate distributional differences. ## related work relating pretraining data to model behavior there is a growing body of work focused on studying pretraining data properties and their relation to model behavior. this type of large-scale data and model analysis provides useful insights into model learning and generalization capabilities @xcite . recent work shows that few-shot capabilities of large language models are highly correlated with pretraining term frequencies, and that models struggle to learn long-tail knowledge @xcite . several works have also explored the relationship between pretraining data and model performance from a causal perspective @xcite @xcite . for example, @xcite comprehensively investigate how various data curation choices and pretraining data slices affect downstream task performance. bias amplification our work is strongly inspired by the findings of @xcite , who show that structured prediction models amplify biases present in the data. however, there are important differences to note. first, their task jointly predicts multiple target labels (including gender), as opposed to generating images. furthermore, their work largely focuses on mitigating amplification, as opposed to investigating underlying factors that affect amplification. @xcite consider how data, training, and modeling choices influence amplification using a classification setup with synthetic bias, but do not examine distribution shifts. @xcite also compare biases exhibited by laion and stable diffusion, and show that the model demonstrates amplification. instead of identifying relevant training examples using captions, they use image-text similarity between training images and prompts. furthermore, their work primarily focuses on bias mitigation, while our work is centered around analyzing confounding factors that impact amplification. bias in text-to-image models while it is wellestablished that language and vision models are prone to biases individually, recent work has shown that text-to-image models display similar biases. several works analyze various biases in text-toimage models, including gender biases @xcite , geographical disparities @xcite , and intersectional biases @xcite . @xcite demonstrate that stereotypes persist even after using counterstereotypes. however, these works solely evaluate model biases, and do not examine the training data. ## discussion our results bring up a number of key issues. generalizability our work demonstrates that using naive procedures to evaluate bias amplification can lead to exaggerated amplification measures. while our analysis does not account for all sources of distribution shift that contribute to amplification, it is meant to be illustrative. we encourage future studies to build on our findings by examining different experimental setups (i.e., datasets, models, and types of bias) to gain a more comprehensive understanding of bias amplification and the impact of confounding factors. variation across prompts as we highlight in figure 6 , small changes to prompts can have a resounding effect on conclusions about bias. for example, "a portrait photo of an attorney" skews heavily male while "a photo of an attorney at work" skews female in generated images. furthermore, reductions in amplification differ based on the prompt (e.g., 89% reduction for prompt #1 vs. 49% for prompt 2), indicating that there are prompt-specific sources of distribution shift. our findings match prior work demonstrating that model biases are highly sensitive to wording and phrasing choices @xcite . amplification baseline our interpretation of amplification is centered around models exacerbating biases in the training data as opposed to real-world statistics @xcite . both approaches are useful to study but answer fundamentally different questions. our approach offers insights into whether model behavior reflects the training data, while real-world amplification captures how well the model reflects reality. ## conclusion in summary, we investigate whether stable diffusion amplifies gender-occupation biases by comparing training data and model biases. we highlight how naive evaluations of amplification fail to consider distributional differences between training and generation, which leads to a misleading understanding of model behavior. although amplification is not eliminated entirely, we observe that reducing discrepancies between captions and prompts during evaluation results in substantially lower measurements. we recommend that analyses comparing training data and model biases, or any data and model properties more generally, account for various distribution shifts that skew evaluations.