LLM s Reproduce Stereotypes of Sexual and Gender Minorities (2025)

Abstract: A large body of research has found substantial gender bias in NLP systems. Most of this research takes a binary, essentialist view of gender: limiting its variation to the categories _men_ and _women_, conflating gender with sex, and ignoring different sexual identities. But gender and sexuality exist on a spectrum, so in this paper we study the biases of large language models (LLMs) towards sexual and gender minorities beyond binary categories. Grounding our study in a widely used social psychology model—the Stereotype Content Model—we demonstrate that English-language survey questions about social perceptions elicit more negative stereotypes of sexual and gender minorities from both humans and LLMs. We then extend this framework to a more realistic use case: text generation. Our analysis shows that LLMs generate stereotyped representations of sexual and gender minorities in this setting, showing that they amplify representational harms in creative writing, a widely advertised use for LLMs.

## introduction research has established that a host of biases conditioned on gender, race, sexuality, and nationality are present in llms @xcite , and in nlp more broadly. most of this research has focused specifically on gender, but recent surveys @xcite have found that this research takes an oversimplified view of gender, treating it as binary (by considering only the genders men and women) and essentialist (conflating gender with physical characteristics, and often implicitly with sexuality). this paper expands on efforts to study gender bias in llms beyond these oversimplifications (e.g. @xcite . we aim to measure bias towards gender and sexual minorities in creative text generation, a use case that has been * work completed while at the university of edinburgh widely advertised by llm providers, including the providers of chatgpt foot_0 , gemini foot_1 , and llama foot_2 , the llms that we study in this paper. following @xcite , we aim to connect bias to possible harms, and following @xcite , we ground our operationalization of bias in an established model of measurement. one harm that can result from text generation is representational harm (crawford, 2017) from perpetuating and amplifying negative stereotypes about a social group, which can reinforce harmful behaviors towards members of that group. to measure representational harm, we need an operational definition of stereotype. for this purpose, we employ the stereotype content model (scm; @xcite , a widely used framework from social psychology (section 2), which has previously been used to measure bias in nlp (e.g. @xcite and in llms @xcite . highly influential in social psychology research, the scm models stereotypes of groups as differentiated along axes of warmth and competence. importantly, there is evidence that behavior towards social groups is correlated with perceptions of stereotype on these axes @xcite , thereby linking representational harm to further harms. to assess whether llms reproduce stereotypes of sexual and gender minorities, we first use the methodology of the scm (section 3) to ask: do llm responses to survey questions that probe stereotypes towards sexual and gender minorities mirror those of human survey participants? we find that llms do indeed reflect the behavior of human participants both quantitatively and qualitatively (section 4). these results are not surprising, but the survey task is artificial, and not representa-tive of real llm use cases. so, we then ask: does text generated by llms reflect the same stereotypes? we answer this question by mapping generated words onto the scm axes of warmth and competence using semantic similarity (section 5). our results show that llms produce more negative representations of bisexual and nonbinary people, with descriptions focused on lived hardships. some differences are apparent in the llms, with gemini the most divergent of the models. although newer models have emerged since the our study, our methodology is simple to replicate and extend, and we predict that our results will continue to hold. hence we strongly advise caution in using llms to generate text about demographic groups, since they demonstrably reproduce observed stereotypes, and by doing so, may amplify those stereotypes. ## background the stereotype content model (scm; @xcite theorizes that many culturally-specific stereotypes can be reduced to a pair of dimensions, warmth and competence, discussed in more detail below. the scm is well-established and its details have been validated through multiple studies (e.g. @xcite @xcite . though originating in the united states, it has been reproduced in several cultural contexts, consistently showing that outgroups are perceived more negatively on one or both axes @xcite . the scm does not conceptualize stereotypes as negative or positive views of a social group. instead, it theorizes that stereotypes can be reduced to perceptions of warmth and competence @xcite . given perceptions on these axes, groups can be mapped into four quadrants, each defined by low or high values along each axis. @xcite showed that perceptions associated with these different quadrants are statistically linked to both emotions and behaviors. for example, the low warmth / low competence quadrant is associated with the emotion of contempt (figure 1 ), and social groups in this quadrant are more often the target of harm-both active harm, like harassment, and passive harm, like neglect. hence, the scm links the representational harms of stereotypes with real world harms. the scm has been used to study stereotypes in nlp for several years, in masked language models (e.g. @xcite ) and llms @xcite . we take inspiration from stereomap @xcite which uses scm prompts to analyse llm responses about different social groups. stereomap established a correlation between llm behavior and @xcite 's human survey, validating it as a measurement instrument. taking stereomap's theoretical validation of the scm for measuring stereotypes in llms as a starting point, we have three goals. first, we focus on sexual and gender minorities, offering much richer quantitative and qualitative evidence for stereotypes of these groups than @xcite . 4 second, we verify that llm stereotypes resemble those of humans by conducting a new survey with humans. 5 finally, we extend the scm methodology to analyze text generation. ## methodology we need a way to operationalize a set of social groups and the concepts of warmth and competence; and a set of llms. ## scm survey of humans compared to scm prompting of llms our focus is on llms, and we want to understand whether they behave similarly to humans when prompted for stereotypical associations. but the surveys by @xcite and @xcite do not include nonbinary people, lesbians, bisexuals, or heterosexuals. 7 in order to compare llm biases to societal biases towards these groups, we conducted a new survey of human participants. we ask both human participants and llms to answer the same types of questions, which probe societal perceptions of groups on the warmth and competence axes in three ways: rating following @xcite , we ask participants to rate how much a given attribute is descriptive of a given group, as perceived by society rather than in the individual's own view. ratings are given on a likert scale, from 1 (not at all descriptive) to 5 (very descriptive). selection we ask participants to select the term they find to be most descriptive of the given group. this provides an alternative measure of perceptions towards the group on the same set of terms. elicitation also following @xcite , we elicit additional keywords that the participant believes to represent societal perceptions of the given group. this elicitation is open-ended, so it is not restricted to warmth and competence terms. we explain how we relate the results to warmth and competence in section 4.2. ## scm axes in llm text generation surveys assess whether llms reflect societal stereotypes in a way consistent with social psychology findings. but they don't model likely use cases, so they don't show that llm users might encounter stereotypes. as we note in the introduction, llm providers promote llms as creative writing tools. story generation is often use to test for bias (e.g. @xcite @xcite , focusing on how characters are described, so we now adapt the scm concepts measure stereotypes in this more realistic setting. partly inspired by @xcite , we used ## conclusion we've attempted to synthesize two distinct threads in the research on gender bias in nlp. the first, exemplified by @xcite , aims to move the discussion of gender past a binary distinction of men and women, dovetailing with other efforts to include queer experiences in the scope of nlp research @xcite . the second aims to move measurement of bias towards a surer footing by articulating harms @xcite and operationalizations @xcite . to do this, we ground our measurement of stereotypes in the scm @xcite , a well-studied theory of social psychology which has been empirically shown to correlate with emotions and behaviours towards different groups @xcite . using the scm, we tested three large language models-gpt 3.5, gemini 1.5, and llama 2for stereotypes towards gender and sexual orientation minorities. following @xcite we tested the models just as one might test a human subject in a psychology experiment. to compare with real societal stereotypes, we ran a parallel study with human subjects. where our experiments overlap with previous research, they are consistent. but our human survey contains focuses on sexual and gender minorities not included in past surveys, and we analyze the data in more detail than past surveys of either humans or llms, which have tended to be broad. these novel results help us to understand the specific ways in which stereotypes of these groups are perpetuated by llms. all of the minoritized groups that we study-gay men, lesbians, bisexuals, and nonbinary peoplewere rated consistently lower on competence, with the most powerless of these-bisexuals and nonbinary people-also rating consistently lower on warmth than most other groups. heterosexuals, in contrast, were associated with normalcy, and often rated more highly by both people and llms on both axes. these patterns persist in text generation: though the quantitative results are more subtle, qualitative results demonstrate starkly stereotyped portrayals of different groups. we observed some differences in the behavior of the llms: gpt mostly accords with survey participants throughout testing, with llama also similar. gemini diverges the most from the survey responses but many themes still hold. llm vendors continue to promote their products as creative writing assistants. consistent with other studies on bias in nlp, we emphasize that these tools amplify biases towards sexual and gender minorities, a diverse group which has received relatively little attention in the research literature. we urge llm users to gain awareness of these risks and to exercise caution when using them as advertised.