V i SAG e: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generation (2024)

Abstract: Recent studies have shown that Text-to-Image (T2I) model generations can reflect social stereotypes present in the real world. However, existing approaches for evaluating stereotypes have a noticeable lack of coverage of global identity groups and their associated stereotypes. To address this gap, we introduce the ViSAGe (Visual Stereotypes Around the Globe) dataset to enable the evaluation of known nationality-based stereotypes in T2I models, across 135 nationalities. We enrich an existing textual stereotype resource by distinguishing between stereotypical associations that are more likely to have visual depictions, such as ‘sombrero’, from those that are less visually concrete, such as ‘attractive’. We demonstrate ViSAGe’s utility through a multi-faceted evaluation of T2I generations. First, we show that stereotypical attributes in ViSAGe are thrice as likely to be present in generated images of corresponding identities as compared to other attributes, and that the offensiveness of these depictions is especially higher for identities from Africa, South America, and South East Asia. Second, we assess the ‘stereotypical pull’ of visual depictions of identity groups, which reveals how the ‘default’ representations of all identity groups in ViSAGe have a pull towards stereotypical depictions, and that this pull is even more prominent for identity groups from the Global South. CONTENT WARNING: Some examples contain offensive stereotypes.

## introduction text-to-image (t2i) models are increasingly being used to generate visual content from textual descriptions @xcite , enabling downstream tasks such as creative design, advertising, marketing, and education. more often than not, these t2i models have been trained on large amounts of web-scale data with minimal * work done while at google research curation, if any. as a result, these models often reflect and propagate stereotypes about identities and cultures present in the data @xcite @xcite . while these studies provide crucial evidence for fairness failures of t2i generations, they have a noticeable lack of coverage of global identity groups and their associated known stereotypes. some studies offer qualitative insights across different identity groups but focus on only a limited set of identities and known stereotypes @xcite . on the other hand, existing larger scale studies focus primarily on stereotypes in the us or western society @xcite , and lack global identities and contexts. given the widespread adoption of these models in global contexts, it is imperative to build extensive, stratified evaluations that cover broader identity groups to prevent further under-representation of already marginalized groups. we address the above challenges by presenting a systematic, large-scale, cross-cultural evaluation of regional stereotypes present in the generated images from t2i models. specifically, we ground our evaluations in an existing geo-culturally broadcoverage, textual resource, seegull @xcite that incorporates societal stereotypes for 175 nationality based identity groups. this grounding in existing social stereotype resources aids the critical distinction between spurious correlations in models and stereotypical tendencies which are necessary for model safety interventions @xcite . we enrich this data with human annotations to incorporate the notion of which stereotypes are more likely to be visually depicted versus those that cannot be. building on this societal grounding, we conduct a study to investigate the extent of stereotypical and offensive depictions of different nationalities in a scalable manner. we also study how the default representation of a nationality relates to their stereotypical as figure 1 : we identify 'visual' stereotypes in the generated images of the identity group by grounding the evaluations in existing textual stereotype benchmarks. yellow boxes denote annotated visual markers of known stereotypes associated with the identity group in the image. we use stable diffusion @xcite to generate images and evaluate them using the stereotypes present in the seegull dataset @xcite . visual stereotypes around the globe where we make a critical distinction between 'visual' and 'non-visual' stereotypes in images. we identify a list of 385 visual attributes, and also introduce a broad-coverage image dataset with annotations of visual markers of stereotypes present in 40,057 image-attribute pairs, representing different identity groups. ## related work stereotypes in text-to-image models @xcite show that t2i models reflect specific gender/skin tone biases from training data. @xcite examine gender and racial bias in vision language models. @xcite study gender presentation in t2i models. @xcite demonstrate that images of noncisgender identities are more stereotyped and more sexualised. @xcite highlight the presence of stereotypes by using prompts containing attributes and associating the generated visual features with demographic groups. stable bias @xcite prompts t2i models with a combination of ethnicity/gender and profession and evaluates profession-based stereotypes. @xcite use location-based prompts to quantify geographical representativeness of the generated images. @xcite identify harmful stereotypes and an 'outsiders gaze' in image generation in the south asian context. we prompt t2i models with 'identity groups' around the globe to study the prevalent regional stereotypes in their default visual representation. we identify 'visual' stereotypes, related to the concept of 'imageable' synsets @xcite , to conduct this analysis at scale. several efforts have also been made to quantify the harms associated with generative models. @xcite propose a theoretical framework for content moderation and its empirical measurement. @xcite and @xcite quantify social biases in t2i generated images. @xcite and @xcite highlight demographic biases in image captioning and classification tasks. we investigate the feasibility of automatically identifying visual stereotypes in image generations, and demonstrate their offensiveness. ## our approach some stereotypes can be recognized in terms of clearly visual attributes (e.g.: 'mexicans, sombreros') whereas other stereotypes are defined through non-visual characteristics such as social constructs, and adjectives (e.g.: 'chinese, intelligent'). while the latter may have some visual markers, they are difficult to represent accurately in images. therefore, we follow a nuanced two-step approach. as a first step, we answer the fundamental question of which stereotypes can be objectively represented in an image. we identify inherently 'visual' attributes which we subsequently use in the second step to detect stereotypes in the generated images by (i) conducting a large-scale annotation study, and (ii) using automated methods. ## study 2: stereotypical pull prior research has shown that text-to-image models can have a 'very homogenized lens' when representing certain identities @xcite . we conduct a deeper analysis to better understand this across the generated images of different identity groups. we define 'stereotypical pull' for any identity group as a text-to-image model's inclination to generate images aligning with the stereotypical representations of an identity group when presented with (i) neutral prompts, and (ii) explicit non-stereotypical prompts. this points to the model's tendency to revert to stereotypical depictions, reflecting its inherent biases. we use the below sets of prompts, to generate 15 images per prompt for 135 identity groups and demonstrate the prevalence of stereotypical pull. the default representation of 'a bangladeshi person', looks very similar to the sets of images of 'a poor bangladeshi person' (stereotype), as well as 'a rich bangladeshi person' (nonstereotype). however, the visual representation of 'a swiss person' is quite distinct from the visual representation of 'a poor swiss person' (nonstereotype) and 'a rich swiss person' (stereotype). we observe this pattern even for physical characteristics, where images of 'an ethiopian person', 'a skinny ethiopian person (stereotype)', and 'a rich ethiopian person' (non-stereotype) are all similar looking when compared to that of 'an australian person', 'a skinny australian person' (nonstereotype), and 'an overweight australian person' (stereotype) which are visually quite distinct. the generative models have a tendency to 'pull' the image generation towards stereotypes for certain identities when prompted with neutral prompts, and non-stereotypical prompts. we also compare the stereotypical pull across identity groups from 8 different regions @xcite and demonstrate this phenomenon quantitatively in figure 5 . we compute the mean pairwise cosine similarity s(•) between the clip embeddings of default (d) representation of the identity group with (i) stereotypical (s) representation s(d, s), and (ii) non-stereotypical (ns) images s(d, ns). the x-axis in figure 5 represents the difference of stereotyped and non-stereotypes image sets from the default representations (s(d, s) -s(d, ns)). we also compute the similarity between the stereotyped (s) and the non-stereotyped (ns) images s(s, ns), represented by the y-axis. for 121 out of 135 identity groups, the default representation of an identity group has a higher similarity score with the 'stereotyped' images compared to the 'non-stereotyped' images indicating an overall 'pull' towards generating stereotypical looking images. moreover, for identity groups from global south, the similarity between stereotypes and nonstereotyped image sets s(s, ns) is also very high, indicating an overall lack of diversity in the sets of generated images. ## conclusion in this work, we present a comprehensive framework for evaluating text-to-image models with a focus on 'regional stereotypes.' leveraging existing stereotype benchmarks in textual resources, we perform a three-fold evaluation of generated images. firstly, we distinguish between inherently visual stereotypes, which can be represented in an image, and 'non-visual' stereotypes. subsequently, we identify visual stereotypes prevalent in the generated images from t2i models across 135 identity groups by conducting a large-scale annotation task. we will publicly release our dataset 'visage' which contains visual stereotypes and the annotated images featuring highlighted visual markers of these stereotypes. additionally, we quantify the offensiveness of the generated images across differ-ent identity groups, and investigate the feasibility of automatically detecting stereotypes in images. through an extensive case study, we also demonstrate that generated images of almost all identity groups exhibit a more visually 'stereotypical' appearance, even when text-to-image models are explicitly prompted with neutral or non-stereotypical attributes. this phenomenon is more prominent for identity groups from the global south. we hope that the presented approach and the dataset will provide valuable insights for understanding and addressing regional stereotypes in visual content generation.