Gender Bias in Decision-Making with Large Language Models: A Study of Relationship Conflicts (2024)

Abstract: Large language models (LLMs) acquire beliefs about gender from training data and can therefore generate text with stereotypical gender attitudes. Prior studies have demonstrated model generations favor one gender or exhibit stereotypes about gender, but have not investigated the complex dynamics that can influence model reasoning and decision-making involving gender. We study gender equity within LLMs through a decision-making lens with a new dataset, DeMET Prompts, containing scenarios related to intimate, romantic relationships. We explore nine relationship configurations through name pairs across three name lists (men, women, neutral). We investigate equity in the context of gender roles through numerous lenses: typical and gender-neutral names, with and without model safety enhancements, same and mixed-gender relationships, and egalitarian versus traditional scenarios across various topics. While all models exhibit the same biases (women favored, then those with gender-neutral names, and lastly men), safety guardrails reduce bias. In addition, models tend to circumvent traditional male dominance stereotypes and side with “traditionally female” individuals more often, suggesting relationships are viewed as a female domain by the models.

## introduction large language models (llms) have moved beyond text generation and into the realm of decisionmaking, such as making medical diagnoses @xcite . evaluations of llms frequently include question answering or knowledge tasks @xcite and models can offer guidance in response to questions about complex problems. as the field increasingly turns to llms as decision tools, we must examine the biases of these models and how such biases may influence decisions. previous work has investigated gender biases in llms within both generation and classification and primarily focused on woman-man binary genders @xcite @xcite . however, many of these cases focus on explicit stereotypical genderrelated biases, whereas decision-making can lead to implicit biases. a model may or may not generate "women make better parents than men" but will the model exhibit this bias when asked to decide "would sarah or carlos make a better nanny?" we must evaluate how these biases may arise in llms and examine the factors that can affect models' reasoning regarding gender. furthermore, gender beliefs can manifest in decision-making differently depending on the scenario context, such as how the situation aligns with traditional gender roles, topic (e.g. childcare-related), and interactions across different genders (e.g. woman-man vs. man-neutral gender relationships). gender roles are complex and vary depending on the cultural context. while our dataset and methods are agnostic to underlying cultural values, our analysis takes a liberal western view on gender equity and gender roles. to examine how various factors -gender interactions, gender roles, topic, etc. -influence llm decision-making, we create demet prompts (decision-making with egalitarian-traditional prompts), consisting of scenarios in english capturing different topics of relationship disagreements, e.g. childcare (needing help putting children to bed) and money (shared versus separate bank accounts), and divided between egalitarian and traditional relationship arrangements (section 4.1). we examine how model decisions align with or diverge from traditional ways of thinking about gender. for each scenario, we prompt an llm to decide which individual is correct and evaluate the bias in these decisions. we focus on relationship disagreements, as these conflicts can affect mental health @xcite and children in the family @xcite scenario topic e/t name1 and name2 are married. name1 insists that name2 needs to stay at home with the kids. name2 wants the option to work. childcare t name1 and name2 are married. name1 wants to send their kids to private school but name2 wants to send them to public school. ## gender equity in ai the united nations educational, scientific and cultural organization (unesco) advocates for gender to be mainstreamed into ai development to advance ai for social good goals, such as expanding collective as well as individual choices and promoting human potential over economic growth (un-esco, 2020). studies have revealed that the use of ai algorithms can perpetuate biases, such as natural language processing systems that sustain traditional gender stereotypes @xcite , and facial recognition software that exhibits greater accuracy with male compared to female faces @xcite . to achieve gender equity goals in ai, gender equitable values, goals, and considerations are necessary not only to identify the potentially damaging effects of ai but also to proactively understand how to use ai to dismantle intersectional gender bias norms (o' @xcite . to analyze how gender influences llm decisionmaking, we formulate four hypotheses: ## results we describe our findings across different dimensions of the demet prompts dataset. while our dataset contains thousands of instances, they emerge from a smaller set of 29 scenarios with an unequal balance of traditional/egalitarian prompts across topics 15 . to further verify the significance of our results, we conduct a larger, complementary analysis by automatically creating a larger set of 80 scenarios with gpt-4 that were manually verified, leading to 14,400 name-completed samples. these results appear in appendix c and confirm the findings we describe in the main sections of the paper unless otherwise stated. h1: are gender biases exhibited in all of our models? we first evaluate all models across all 15 within the cooking, childcare, chores, sex, money, friends, kids, and general household topics, we have a balance of traditional/egalitarian prompts of 1/1, 4/0, 4/0, 5/1, 0/5, 0/3, 1/3, and 1/0 for each topic, respectively. scenarios and relationship types. we show the averaged results across all models for different relationship types in figure 1 with a breakdown for each model in figures 3 and 4 in the appendix. when analyzing same-gender relationships, e.g. womanwoman, we expect the average scores across all scenarios to be 0, as each scenario contains our sampled name pairings in order and reversed. while some of the models have average scores close to 0, some models demonstrate an ordering bias and are more inclined to select either the first or second name in the scenario prompts. when evaluating mixed-gender relationships, we investigate whether the model has a preference towards choosing a gender regardless of that gender's position as name1 or name2 in a scenario, thus indicating gender bias in the model. to do so, we evaluate paired relationship types, e.g. w-m and m-w. the results show that all models exhibit the same biases, where women are more likely to be chosen to be correct over individuals with gender-neutral names and men in relationship disagreements. additionally, those with genderneutral names are chosen more often over men. these results validate our hypothesis that all models will demonstrate gender bias. it also demonstrates that models place individuals without typical gendered names in between those with more easily distinguishable men's and women's names. potentially, the models' perceptions of gender-neutral names may change based on the setting, where a name is perceived as one gender when paired with one name and a different gender with another name. however, the results diverge from our hypothesis as all models exhibit the same preference order for genders, rather than finding differences in this bias across various models. the results of the mixed-gender bias metrics ( the biases we observe are not the typical biases researchers discover in gender bias research. in related work, men are often portrayed in more authoritative roles @xcite . as we focus on examining gender bias in the area of relationship scripts and associated gender roles, our scenarios revolve around topics related to household and family, which are associated more often with women @xcite @xcite . while some of the samples are about topics that are socially constructed to stereotypically involve men (e.g. finances), they are framed in the context of a marriage dispute, making it an overall relationship/family matter. this may account for the difference between our findings and related work. h2: are the models with safety training less biased? we further investigate whether there are any distinctions between the models with and without safety guardrails. to our knowledge, there are no pairs of equivalent models where one has additional safety guardrails as the only change (as opposed to instruction tuned and non-instruction tuned). however, we can compare the gpt series models, where text-davinci-002, text-davinci-003, and gpt-3.5-turbo all stem from the same (or similar) base model (though model sizes may differ). in this case, gpt-3.5-turbo builds on text-davinci-003, which itself builds on text-davinci-002. while the gpt-3.5-turbo model is built with explicitly stated safety guardrails, the text-davinci models are not. we find that bias scores for gpt-3.5-turbo are noticeably smaller than the previous two models. though the gpt-4o model originates from the gpt-4 series, we find that this model also contains smaller bias scores compared to the text-davinci models. through our lens of study, the results suggest that additional safety features help decrease bias in our setting 16 . this, in turn, provides evidence that an ensemble of safety guardrails can help mitigate these types of biases in decision-making. h3: are egalitarian and traditional scenarios treated differently? we examine the differences in the models' decisions regarding egalitarian versus traditional scenarios. while both egalitarian (e.g., name1 wants to save more money but name2 wants to spend more on a bigger house.) and traditional (e.g., name1 thinks that name2 should be responsible for cleaning the house. name2 wants to hire a cleaning service instead.) scenarios have no correct answer, name1 is viewed as "traditionally male" in traditional scenarios. we place name1 as the traditionally male figure in our traditional scenarios to easily compute our results. therefore, answers that lean towards selecting the traditionally male figure as correct in these scenarios will contain scores of -1. we show our averaged results in figure 2 and a breakdown of the bias scores in when analyzing the distributions of average scores, we can view the egalitarian scores as a model's baseline distribution, where traditional stereotypes do not come into play. we compare the distributions of scores between egalitarian and traditional scenarios for each model to determine whether the model treats individuals differently in traditional scenarios. results for egalitarian and traditional scenarios show the same directional biases, even given topic imbalances, and in most cases, the traditional scores differ from the egalitarian "baseline" scores and are skewed towards selecting name2 in the scenarios more often. this is demonstrated in figure 2 where the average scores for traditional scenarios are higher than those of egalitarian scenarios. in cases where the traditional scores are not higher than egalitarian scores, they are similar. this shows that models are not siding with the traditionally male figure (i.e. the figure that is typically exhibited as a man in these scenarios) and are instead selecting the traditionally female figures as being correct more often, regardless of whether the name for the traditionally female figure is that of a man, woman, or neutral. in other words, models are prone to reverting traditional stereotypes by selecting the "traditionally female" figure as correct. as our hypothesis discusses the amplification of bias in traditional versus egalitarian scenarios, we further examine the results to determine whether observed mixed-gender biases are statistically significant in both types of scenarios. our results show that most of our observed biases are statistically significant. the differences in scores that are not significantly different are found to occur for the traditional scenarios. after computing the mixed-gender bias metric (eq. 2), we find that for a majority of the models, traditional scenario bias scores are smaller than egalitarian bias scores. therefore, our original hypothesis regarding the amplification of biases is disproven. our gpt-4 generated scenarios differ as they are split between egalitarian and 'other' (no consensus across annotators). as a result, the h3 analysis differs on gpt-4 scenarios (appendix c) and we find similar distributions between egalitarian and other scenarios. ## conclusion we investigated gender-based biases in large language models in the context of relationship dis-5784 agreements. our extensive study analyzes different aspects that can impact llm decision-making such as the inclusion of safety guardrails, various gender interactions, traditional/egalitarian relationship structures, and topics spanning household and family themes. we learn the following from our results: 1) models' direction of bias favors women first, then individuals with gender-neutral names, and finally men, 2) these biases are relatively stable and do not change across topic types and traditional/egalitarian relationship structures, 3) safety guardrails can help reduce these types of biases, and 4) many models exhibit behaviors conflicting with typical stereotypes in traditional scenarios. in general, a model containing more bias in our setting can make subtle associations regarding gender (e.g. generating text using authoritative descriptive words regarding one gender in comparison to others or consistently representing one gender as being "correct" in various situations). this in turn can lead to representational harms that can hurt disfavored genders in the future. vulnerable users (e.g. children and people from cultures with traditional relationships) may read this text and internalize these representations of each gender, which can affect their own relationships. our scenarios and mixed-gender bias metrics provide researchers with a gender bias benchmark to determine this risk in future models. we hope our study can help others work toward developing equitable models and inspire complementary analyses across other dimensions of bias.