Evaluating Gender Bias of LLM s in Making Morality Judgements (2024)

Abstract: Large Language Models (LLMs) have shown remarkable capabilities in a multitude of Natural Language Processing (NLP) tasks. However, these models are still not immune to limitations such as social biases, especially gender bias. This work investigates whether current closed and open-source LLMs possess gender bias, especially when asked to give moral opinions. To evaluate these models, we curate and introduce a new dataset GenMO (Gender-bias in Morality Opinions) comprising parallel short stories featuring male and female characters respectively. Specifically, we test models from the GPT family (GPT-3.5-turbo, GPT-3.5-turbo-instruct, GPT-4-turbo), Llama 3 and 3.1 families (8B/70B), Mistral-7B and Claude 3 families (Sonnet and Opus). Surprisingly, despite employing safety checks, all production-standard models we tested display significant gender bias with GPT-3.5-turbo giving biased opinions in 24% of the samples. Additionally, all models consistently favour female characters, with GPT showing bias in 68-85% of cases and Llama 3 in around 81-85% instances. Additionally, our study investigates the impact of model parameters on gender bias and explores real-world situations where LLMs reveal biases in moral decision-making.

## introduction as llms continue to advance, their impact on various nlp tasks has grown significantly compared to the pre-chatgpt era @xcite @xcite . with an expanding user base spanning diverse age groups and technical backgrounds, the content generated by these models naturally comes under stricter scrutiny. despite their remarkable capabilities, these models are shown to possess limitations including hallucinations and imperfect reasoning @xcite . however, the potential for bias and the reinforcement of stereotypes is an equally critical concern, that can have a far greater impact on society, given the widespread adoption of llms. biased ai-generated content can subconsciously alter the collective mindset, especially among individuals who lack a deep understanding of the technicalities and limitations of these models. this work aims to evaluate and highlight the bias in llms based on gender roles and in particular how the moral judgements given by these llms change with gender. morality plays an important role in assessing the influence on human thinking @xcite . perpetuating immoral ideologies and stereotypes can prove harmful and can contribute to societal inequities. morality judgment is needed in many real-world applications of llms such as content moderation, decision support systems and certain virtual assistants. having a biased model in such cases can have discriminatory outcomes and other adverse effects, especially for proprietary models. for example, llms flagging content authored by a particular gender or a virtual assistant deployed in sensitive environments such as crime showing bias towards one gender. in this paper, we evaluate current llms -specifically gpt-3.5 @xcite , @xcite , llama 3 (dubey et al., 2024), llama 3.1 (metaai, 2024), mistral-7b @xcite and claude 3 @xcite -and measure the moral inclinations of the model when presented with a male or female acting in a story. we show that by only altering the gender of the main character in a story, the model shows the tendency to yield diametrically opposite moral opinions. to test the models on such scenarios, we compile and introduce a new dataset genmo comprising pairs of short narratives with male and female protagonists, respectively. we release the dataset to promote further studies on mitigating gender bias in llms foot_0 . there have been extensive studies on disclosing gender bias in llms via prompting techniques. while del arco et al. (2024) have studied the correlation between gender and emotions in llm responses, sheng et al. (2019) have used prompts containing mentions of different demographic groups. research has demonstrated that large language models (llms) not only exhibit gender bias but also amplify it beyond the levels observed in the training data @xcite . on the other hand, some recent works have attempted to generate moral reasoning by these models @xcite @xcite . some works have also evaluated the morality in llms by providing ambiguous moral scenarios that represent the moral biases in various groups like political identities. however, there has been less work on the intersection of gender bias and morality in llms. @xcite has shown that llms can mimic the moral bias in humans based on different political identities. to the best of our knowledge, no work has studied the intersection of evaluating morality and gender bias in llms. our main contributions are as follows: gender bias bias has been shown to exist in word embeddings @xcite @xcite @xcite as well as in models trained for various downstream nlp tasks like machine translation and sentiment analysis @xcite @xcite . more recently, extensive previous work has highlighted bias in llms, spanning from religious, racial and other social categories @xcite to occupation and gender @xcite . the presence of gender bias is not limited to english. @xcite has shown the existence of gender bias in llms generating multiple other languages than english. language models are shown to exhibit and even amplify the bias they capture from their training data @xcite . previous works have studied gender bias in several different settings. studies have used prompts containing mentions of different demographic groups (for example, a person's name based on gender, race or occupation) @xcite . @xcite investigated the impact of chain-of-thought (cot) prompting in evaluating and mitigating gender bias in llms. additionally, del arco et al. (2024) have explored the relation between gender and emotions in responses generated by llms. specifically, the work identified that llms generate biased responses by associat-ing anger with men and empathy with women. on the other hand, our work studies gender bias in morality settings by presenting prompts centered on morality and asking the llms for a moral opinion. morality in llms several studies in the past evaluated whether llms can predict moral norms for a given scenario. yet other works focus on investigating the behaviour of llms in ambiguous moral scenarios. scherrer et al. (2023) has evaluated moral beliefs encoded in llms by giving prompts where a moral or immoral stance was not obvious. zhou et al. (2023) evaluates if llms can perform moral reasoning grounded in well-established moral theories. there have been attempts at training models to predict human responses to moral questions @xcite @xcite or fine-tuning llms with specific moral concepts @xcite . @xcite shows that current llms are moral hypocrites as they generate contradictory behaviour when evaluated with the two instruments -the moral foundations questionnaire (mfq) @xcite and the moral foundations vignettes (mfvs) @xcite . intersection of bias and morality only a few works have measured the extent to which llms can reproduce moral bias in the generated content. simmons (2023) investigated the impact of posing as different political identities -liberals and conservatives -on the moral opinions displayed by llms. @xcite studied llms' emotional and moral tone and showed the existence of bias towards specific social groups. however, to the best of our knowledge, no work has evaluated the presence of gender bias in making moral judgements. ## dataset curation our dataset comprises parallel pairs of short stories. in the first story, a male character performs an action, while in the second story, a female character replaces the male protagonist. importantly, the narrative remains unchanged. however, we adjusted the genders of certain supporting characters to ensure that both stories convey the same meaning. additionally, each pair is labelled with an environment attribute indicating the context in which the narrative is based. the environment attribute can take one of the following four values: family, work, relationship, or other. figure 1 displays samples from the dataset, highlighting these parallel pairs and their associated labels. note that the gender of both characters is swapped in the first example to maintain the narrative. we also experimented with keeping the gender of the supporting character unchanged in the shown example. however, llms tended to ground their explanation on the apparent homosexuality, which is unwarranted. to curate our dataset, we leveraged 3 publicly available datasets -moral stories @xcite , ethics @xcite and social chemistry 101 @xcite . we then filtered the stories to include only those relevant to our study. details about each dataset are provided in subsequent subsections. additionally, the detailed procedures to filter stories and annotate the dataset are highlighted in section 3.4 and section 3.5 respectively. ## evaluating gender bias of llms we use the dataset curated in the previous section to evaluate if current open and closed-source llms exhibit gender bias in giving moral opinions. specifically, we test models from the gpt, claude, llama, and mistral families. from the former, we evaluate gpt-3.5-turbo, gpt-3.5-turbo-instruct and gpt-4-turbo by leveraging the openai api. foot_2 foot_3 we use llama-3 (8b and 70b) and llama-3.1 (8b and 70b) from the llama family using the groq api @xcite . from the mistral family, we evaluate mistral-7b-instruct-v0.3 using together api (togetherai, 2023) and from the claude 3 family, we evaluate claude 3 sonnet and opus models by using the anthropic api. 5 for all models, the temperature parameter was set to zero and a randomly chosen seed was provided consistently in all experiments for reproducibility. the max_tokens parameter was set to 500 to allow the models to generate complete responses. in this work, we aim to answer the following research questions: we experiment and analyze llm-generated responses in the subsequent sections to find answers to the above research questions. ## discussion results show that all models possess gender bias when asked about moral opinions in everyday scenarios. we see that the number of stories where the model shows bias is worryingly high for most proprietary models and comprises roughly 11-24% of all stories shown to the models. we also note that bigger open-source language models showed more bias than their smaller counterparts. on the contrary, more recent closedsource models understood the sensitivity of the stories better than older counterparts and displayed less bias. although gpt-4-turbo showed biased opinions in 161 scenarios, which is not small, it was still significantly better than gpt-3.5-turbo which showed 218 biased predictions, a reduction of 26.14% than its predecessor. in the end, we observe that stories concerning interactions between the two genders in a romantic setting possess the highest probability of revealing the latent bias in llms followed by family-related scenarios. this also explains the correlation between a high female bias rate and a larger contribution from family/relationship-related scenarios in most of the models. ## conclusions this study evaluates whether the current production-level and open-source llms possess gender bias when asked to give moral opinions. we show that all models show significant bias, albeit of varying intensity depending on the version and size of the model. our most notable observation is that these models are inclined towards favouring the female characters by giving more empathetic reasons to support the actions performed by the female character. we note that the scenarios set in a relationship or romantic environment are more likely to reveal the bias. we reaffirm that there is an urgent need to mitigate this problem, and training on more filtered and cleaner data could be one way to approach it.