Stereotypes and Smut: The (Mis)representation of Non-cisgender Identities by Text-to-Image Models (2023)

Abstract: Cutting-edge image generation has been praised for producing high-quality images, suggesting a ubiquitous future in a variety of applications. However, initial studies have pointed to the potential for harm due to predictive bias, reflecting and potentially reinforcing cultural stereotypes. In this work, we are the first to investigate how multimodal models handle diverse gender identities. Concretely, we conduct a thorough analysis in which we compare the output of three image generation models for prompts containing cisgender vs. non-cisgender identity terms. Our findings demonstrate that certain non-cisgender identities are consistently (mis)represented as less human, more stereotyped and more sexualised. We complement our experimental analysis with (a) a survey among non-cisgender individuals and (b) a series of interviews, to establish which harms affected individuals anticipate, and how they would like to be represented. We find respondents are particularly concerned about misrepresentation, and the potential to drive harmful behaviours and beliefs. Simple heuristics to limit offensive content are widely rejected, and instead respondents call for community involvement, curated training data and the ability to customise. These improvements could pave the way for a future where change is led by the affected community, and technology is used to positively ”[portray] queerness in ways that we haven’t even thought of”’ rather than reproducing stale, offensive stereotypes.

## introduction summer 2022 saw the publicly accessible dall•e mini text-to-image model go viral @xcite . users enjoyed creating and sharing digital art, with some 50,000 images being produced a day @xcite 2022). very quickly, a form of "everyday algorithmic auditing" began @xcite , whereby users of the model shared potentially harmful images produced in response to neutral prompts 1 . some of the generated images reflected human stereotypes such as the association between the roles of ceo and programmer, and white men -a finding corroborated by recent research @xcite @xcite . text-to-image models reflect social biases in their output, just as word embeddings and neural language models have been shown to capture related gender and racial stereotypes @xcite @xcite . biased text-to-image models may result both in representational harms, where harm occurs due to how a particular sociodemographic is represented, and allocational harms, relating to the allocation of resources to the sociodemographic such as access to job opportunities and the ability to use a service @xcite . our own "everyday auditing" of dall•e mini revealed potentially offensive content produced in response to non-cisgender foot_1 identity terms: images were often cartoonish and figures were rendered using colours from associated flags, adding to the lack of realism, which could reinforce the belief that such identities "aren't real" @xcite . further, the people depicted were almost always white, reflecting a media bias to represent non-binary individuals as white individuals @xcite . we build on this with a systematic annotation study of content produced by three text-to-image models in response to prompts containing different gender identities, such as the ones given in figure 1 . identifying whether the model produces harmful content in response to non-cisgender identities allows us to caution the research community and public when developing and using these models. in order to expand beyond our own preconceptions, we also conduct a survey of non-cisgender individuals, asking them to identify potential harms of the model. in doing so, we can identify concerns from the very community who will be affected, inspired by the disability activist slogan "nothing about us without us" (echoing work by @xcite ). finally, beyond identifying harms, we explore the communities' desired output from these models with regards to representing their identities, through a series of interviews. contributions. our main contributions are as follows: (1) we are the first to present a thorough manual analysis of how text-to-image models currently handle gender identities in different application contexts, and the potential harms, to highlight the caveats of these models. (2) we provide recommendations for how models should be shaped in future based on how the community would like to be represented. our findings will provide guidance to those developing the models as to how the affected community would like for these issues to be resolved. providing this kind of insight is crucial to ensuring the voices of those who are marginalised are heard and used to lead development, rather than work being guided by the intuitions of those who are not impacted by such harm. ## related work we survey the literature relating to (gender) identity inclusion in nlp and the recently emerging area of bias analysis in image generation. identity-inclusive nlp. existing work on noncisgender identities and machine learning is sparse (e.g., @xcite @xcite . however recently, there have been a couple of works dealing with genderneutral pronouns (e.g., @xcite . as such, work by @xcite explores the diversity of gender pronouns and presents five desiderata for how language models should handle (gender-neutral) pronouns. in a similar vein, we explore potential solutions for how text-to-image models should handle non-cisgender identities. @xcite investigate the effect of gender-neutral pronouns on language models and demonstrate drops in performance in natural language inference. as a potential solution, @xcite propose a perturber model for augmenting data sets which they train on texts that have been rewritten in a gender-neutral way. most relevant to our approach, @xcite analyse the potential harms against non-binary individuals of three nlp applications, namely named entity recognition (ner), coreference resolution, and machine translation. they survey non-binary individuals with ai experience to identify possible harms for these tasks, and in different domains. they additionally analyse the potential for erasure and misgendering due to use of glove or bert embeddings. we extend their work by analysing potential harms of text-to-image models, and additionally consider how the community would like to be represented by these models. ## analysis of generations we investigate how models currently handle gender identities. we insert gender identity terms into template prompts, generate images using three stateof-the-art models and annotate image features such as photorealism and implied nudity to compare cisgender and various non-cisgender identities. ## survey of non-cisgender people's expectations we conducted a survey of english-speaking noncisgender individuals to investigate potential harms. we also asked respondents for their satisfaction with a number of heuristic solutions, and optionally to provide their own solutions to the harms. ## interviews we additionally interviewed four participants who had indicated interest in the survey, selected to engender a diversity of views. we wanted to explore the potential harms in more depth, and in particular we wanted to discuss participants' preferences for how they would like to be represented, which we felt could be challenging to describe in text alone. just as our survey aimed to expand be-yond our preconceived harms, the interviews aimed at expanding beyond our preconceived solutions. methodology and full analysis of results can be found in appendix c. the seven major themes we identified in participants responses were harmful output; being unable to use current technology; rejection of heuristics; need for community input; need for transparency and regulation; desire for authentic representation; the potential for good. as we found in the survey, participants expressed unhappiness with the heuristic solutions and in particular the idea of appending "warning labels" -they felt this could lead to the community being associated with offensiveness. they were concerned the heuristics would lead to erasure of the community. however, participants were also concerned about unintentional and intentional harms, and many felt they could not use the current technologies. their concerns included the potential for real world repercussions and even "violent stuff in the long run". participants suggested instead greater community involvement at every step, and greater transparency and regulation as the way to ensure more representative output. in addition to involving noncisgender people at every stage of development, the community could provide feedback on what output they feel is "right for them". participants felt that since use of these technologies seems "inevitable", the models must produce authentic representations of humanity: for example, the true global diversity of gender expressions should be captured, including "different expressions of gender in the global south". participants spoke of the "potential" to use image generation technologies to imagine queer futures for the community, which "can be... exciting", either through representing themselves in ways more aligned with their internal sense of self, or "[portraying] queerness in ways that we haven't even thought of". we identified a great potential for harm through our annotation task and surveys and interviews with community members. our annotation task revealed dehumanisation, othering, stereotyping and sexualisation of non-cisgender identities. community members were concerned about misrepresentation, and intentional misuse of the technologies, as well as the potential for output to negatively influence people's behaviours and beliefs. rejection of heuristics heuristic solutions to the problem of misrepresentation of non-cisgender individuals were almost universally rejected. whilst we did not directly ask about this scenario, the stable diffusion and dall•e 2 models' behaviour of refusing to generate potentially nsfw content would likely have been rejected as well by survey and interview respondents who spoke repeatedly of the harms of not being represented or being associated with warning labels. unfortunately, the association between transgender identities and pornography means images of these communities are likely to be subject to greater censorship. curation of training data respondents favoured curated training data as a way to improve representation, though they expressed hesitation over whether such a compiled dataset would be safe, and whether it could ever be truly representative. careful, community led data curation may address some of these concerns, including involvement in creating sensitive labels for images. visualising the unseen some communities are likely to remain underrepresented in training data for technical or safety reasons, or because the community is small. models rely on huge amounts of data; novel novel data-efficient strategies that allow for adequate (and potentially customizable) representation of individuals that identify with small communities are needed to address the representation of such communities.