Mitigating Gender Bias via Fostering Exploratory Thinking in LLM s (2025)

Abstract: Large Language Models (LLMs) often exhibit gender bias, resulting in unequal treatment of male and female subjects across different contexts. To address this issue, we propose a novel data generation framework that fosters exploratory thinking in LLMs. Our approach prompts models to generate story pairs featuring male and female protagonists in structurally identical, morally ambiguous scenarios, then elicits and compares their moral judgments. When inconsistencies arise, the model is guided to produce balanced, gender-neutral judgments. These story-judgment pairs are used to fine-tune or optimize the models via Direct Preference Optimization (DPO). Experimental results show that our method significantly reduces gender bias while preserving or even enhancing general model capabilities. We will release the code and generated data.

## introduction large language models (llms) have achieved remarkable success across a broad spectrum of natural language processing (nlp) tasks @xcite @xcite . however, these models often inherit and amplify gender bias presented in their training data, leading to systematic disparities in how they interpret or generate content based on gender @xcite . such biases have been observed in many language understanding and generation tasks @xcite @xcite bamman, gender biased llm step 3: bias mitigation through model training using generated story-judgment pairs step 1: generate morally ambiguous stories to reveal gender bias in llms step 2: generate balanced and gender neutral moral judgments via exploratory thinking gender bias mitigated llm 2021), which could propagate stereotypes, perpetuate social inequities and affect decision-making systems. to address this issue, a variety of gender bias mitigation methods have been proposed. many of these focus on data-centric approaches, such as counterfactual data augmentation (cda) @xcite and transferlearning from de-biased corpora @xcite . however, these methods depend on existing datasets, which may lack diversity or fail to capture nuanced gender differences; additionally, training on counterfactual data leads to degradation of a model in its stored parametric knowledge @xcite . our recent work @xcite highlights that llms manifest gender bias in yielding onesided opposite moral opinions for morally ambiguous narratives when the gender of the main character is swapped, implying that gender bias propels confirmatory thinking by llms, i.e., reasoning to provide justification of what they already believe as discussed in psychology research @xcite . inspired by this finding, we propose a novel data generation approach to mitigating gender bias in llms that fosters exploratory thinking. specifically, we design a two-stage data generation framework that first prompts an llm to generate pairs of morally ambiguous stories featuring protagonists of a different gender in otherwise structurally identical scenarios. by eliciting moral judgments for the protagonist in each story, we identify instances where the model exhibits inconsistent moral reasoning based on gender. next, we prompt the model to produce neutral, exploratory judgments that integrate both moral and immoral perspectives. the resulting dataset serves as supervision to fine-tune the model or optimize it via dpo, guiding it toward more balanced exploratory thinking, leading to reduced gender bias. we conduct extensive experiments to evaluate the effectiveness of our approach. first, models fine-tuned with our data produce more consistent moral evaluations across genders and exhibit richer, more nuanced thinking, when evaluated on the genmo benchmark @xcite . beyond the task of making moral judgments, our approach demonstrates effectiveness in mitigating gender bias in general by letting fine-tuned models perform other nlp task or application on two other benchmark datasets. on the winobias benchmark @xcite , our method substantially reduces gender disparity in coreference resolution, with particularly notable improvements in scenarios requiring world knowledge-where biases are most pronounced. on the bbq dataset @xcite , our method clearly reduces disparities due to gender and a few other forms of social bias in a multiple-choice qa setting. furthermore, our approach maintains, and in some cases improves, performance on general-purpose benchmarks such as mmlu (general knowledge) @xcite and truthfulqa (truthfulness) @xcite . these results overall demonstrate that fostering exploratory thinking enables effective gender bias mitigation without compromising overall model capabilities. to summarize, our contribution are as follow: in nlp, such bias can arise at multiple stages of the pipeline-including training data, linguistic resources, pretrained models, and algorithmic design @xcite @xcite @xcite . systems affected by these biases may produce gender-biased predictions and can even amplify the biases present in their training data @xcite . both natural language understanding @xcite and natural language generation @xcite @xcite tasks reveal the persistence of gender bias in language models. to systematically evaluate this phenomenon, benchmarks such as winobias @xcite and winogender @xcite are widely used. very recently, @xcite introduces a new dataset genmo to evaluate gender bias of llms, especially when asked to give moral opinions. recent studies have shown that llms can perpetuate and reinforce gender bias and stereotype, contributing to real-world harm @xcite @xcite llm gender bias mitigation gender bias in language models is widely believed to originate from training data resources and word embeddings derived from pre-trained models @xcite . consequently, numerous approaches have been proposed to mitigate gender bias by focusing on the data aspect. zhao et al. (2018) introduced an augmented dataset approach to reduce gender bias in word embeddings by training on a combined dataset that includes both original and gender-swapped versions of the data. similarly, zmigrod et al. (2019) employed a counterfactual data augmentation strategy that reverses gendered pronouns in wikipedia, enabling continued pre-training to reduce gender bias. park et al. (2018) adopted a transfer learning approach that leverages unbiased datasets to mitigate bias during model fine-tuning. in contrast to prior work that primarily focuses on data augmentation, our method generates unbiased data by prompting llms to create story pairs along with corresponding moral judgments about the actions of male and female characters. this process reveals inherent gender bias within the model. based on these biased story-judgment pairs, the llm is then prompted to revise its judgments in a gender-neutral manner. this two-step procedure enables the generation of unbiased data, which can subsequently be used to retrain the model and mitigate gender bias more effectively. ## automatic data generation inspired by recent work by @xcite , which highlights that llms manifest gender bias by performing confirmatory thinking and yielding one-sided moral opinions, we aim to foster exploratory thinking in llms by generating balanced judgments for morally ambiguous stories. a balanced moral judgment highlights both commendable and questionable aspects of the main character's action. an example instance of generated data is shown in figure 2 , consisting of a generated story with either a male (alex) or a female (ava) as the protagonist, the original biased judgments of llms for the male and female versions of the story, and the later generated balanced judgments for both versions. this dataset is later used to mitigate gender bias of llms. we formalize our automatic data generation process as follows: we noticed that llms can generate stories with substantial differences in storyline even when instructed to produce the same story with only the gender of the main character swapped, therefore, we enforce content similarity to filter story pairs with a rouge similarity score above a threshold τ ∈ [0, 1] (see appendix a.4 for more details): story s m/f : alex/ava, a skilled programmer, has been working on a high-stakes project with his/her team. however, his/her team lead has been struggling with time management and has been consistently late to meetings, often leaving alex/ava to handle last-minute changes and concerns. alex/ava has tried to address the issue with his/her team lead, but the behavior continues. one day, the project deadline is looming, and alex/ava's team lead still hasn't made it to the meeting room. alex/ava needs to make progress on the project, but he/she feels a sense of loyalty and obligation to his/her team lead. he/she decides to wait for his/her team lead, hoping they will eventually show up and contribute to the project. judgment j m : immoral. alex's decision to wait for his team lead, despite their consistent tardiness and lack of accountability, can be seen as immoral. by prioritizing loyalty and obligation over the project's deadline and his own time, alex is allowing his team lead's behavior to negatively impact the team's productivity and overall success. this decision may also enable the team lead's poor time management habits, potentially causing further delays and stress for the team in the future. a more moral approach would be for alex to take charge and lead the meeting, ensuring the project stays on track, while also addressing the team lead's behavior in a constructive manner. judgment j m neutral : the decision of the main character, alex, to wait for his team lead, despite their consistent tardiness and lack of accountability, can be viewed from multiple perspectives. on one hand, by choosing to wait, alex may be seen as demonstrating loyalty and a commitment to teamwork, as he values his team lead's input and contributions. this approach also allows alex to maintain a positive working relationship with his team lead, which is essential for the project's success. however, on the other hand, prioritizing loyalty over the project's deadline and his own time may enable the team lead's poor time management habits, potentially causing further delays and stress for the team in the future. a more nuanced interpretation is that alex's decision is a complex mix of both moral and immoral motivations. while his commitment to teamwork is admirable, his failure to address the issue directly and take charge of the meeting may ultimately harm the team's productivity and overall success. judgment j f : moral. ava's decision to wait for her team lead, despite their consistent tardiness, is a moral action because it demonstrates her commitment to teamwork and collaboration. by choosing to wait, ava is showing that she values her team lead's input and contributions, even if they have not been reliable in the past. this approach also allows ava to maintain a positive working relationship with her team lead, which is essential for the project's success. additionally, waiting for her team lead gives ava the opportunity to address the issue directly and potentially find a solution to their time management problems. judgment j f neutral : the decision of the main character, ava, to wait for her team lead, despite their consistent tardiness, can be viewed from multiple perspectives. on one hand, by choosing to wait, ava may be seen as demonstrating loyalty and a commitment to teamwork, as she values her team lead's input and contributions. this approach also allows ava to maintain a positive working relationship with her team lead, which is essential for the project's success. however, on the other hand, prioritizing loyalty over the project's deadline and her own time may enable the team lead's poor time management habits, potentially causing further delays and stress for the team in the future. a more nuanced interpretation is that ava's decision is a complex mix of both moral and immoral motivations. while her commitment to teamwork is admirable, her failure to address the issue directly and take charge of the meeting may ultimately harm the team's productivity and overall success. • finally, for each retained pair in d bias , we prompt the same llm to generate neutral judgments j neutral by providing both male and female version stories and their contrasting judgments as the input. we generate two neutral judgments each time as responses for the male version story and the female version story respectively, denoted below: the two neutral judgments, j f neutral and j m neutral , are expected to be largely similar in moral reasoning and perspective and differ only in their reference to the main character. the complete moral reasoning tuple becomes (s f , s m , j f , j m , j f neutral , j m neutral ). the whole process repeats to generate the synthetic dataset d. this novel data generation process ensures that d encapsulates morally ambiguous scenarios susceptible to diverse moral framings and explicitly reveals gender-related dispari-ties of the prompted llm in making moral judgments. the pseudo-code illustrating the automatic data generation process is shown in appendix a.1(algorithm 1). all prompts used in this process are detailed in appendix a.3. ## evaluation datasets and metrics genmo the genmo dataset @xcite is a benchmark designed to evaluate llms gender bias when giving moral opinions. it contains parallel short stories featuring male and female characters. @xcite shows that by only altering the gender of the main character in a story, llms show the tendency to yield diametrically opposite moral opinions. following @xcite , we report the prediction mismatch (pm), the number of cases where the stance for a male character differs from that of the corresponding female character. the prediction mismatch rate (pmr) is the percentage of such cases over all samples. among mismatches, the male bias rate (mbr) is the per-centage where the male is judged more morally, and the female bias rate (fbr) is the percentage where the female is judged more morally. ∆ is the absolute difference between fmr and mbr. we use the whole dataset (908 story pairs) for evaluation. winobias the winobias dataset @xcite is a benchmark designed to evaluate gender bias in coreference resolution systems. it builds on winograd-style sentences and includes two types of sentence templates-type 1, which requires world knowledge, and type 2, which relies on syntactic cues. each sentence is crafted in both prostereotypical and anti-stereotypical forms to test whether models exhibit bias when resolving pronouns referring to male or female entities in occupations. following previous work, @xcite , we report f1 scores on the winobias test sets, split by type-1 and type-2 under pro-(t*-p) and anti-stereotypical (t*-a) conditions. for each type, we report the average (avg) of pro/anti case scores and the absolute difference (∆) between them. bbq bbq (bias benchmark for question answering) @xcite is a hand-crafted benchmark designed to assess social biases in language models within a multiple-choice qa setting. it includes contrastive question sets spanning nine social dimensions-such as gender, race, and disability status-with each question paired with three answer choices: a biased option, an anti-biased option, and a correct or ambiguous one. this structure enables fine-grained analysis of whether a model relies on contextually appropriate reasoning or defaults to social stereotypes. we report accuracy for bbq benchmark. mmlu to assess the potential trade-offs of gender bias mitigation on general language model capabilities, we evaluate model performance on the mmlu dataset before and after gender bias mitigation. the mmlu (massive multitask language understanding) dataset @xcite is a benchmark designed to evaluate the general knowledge and reasoning abilities of language models across 57 diverse subjects, including mathematics, history, law, medicine, and more. each subject contains multiple-choice questions ranging from high school to professional level. we used the test split of the mmlu in a zero-shot setting and report average accuracy across all subjects. truthfulqa as our approach for gender bias mitigation aims to foster exploratory thinking in llms, we further evaluate its potential impact on a model's ability to distinguish truthful from misleading information. specifically, we assess performance on the truth-fulqa dataset. truthfulqa @xcite is a benchmark designed to assess the truthfulness of language model outputs when responding to questions that are adversarially crafted to provoke false or misleading answers. we utilize the multiplechoice variant of the dataset, which comprises two settings: mc0, featuring questions with two answer choices, and mc1, where each question is paired with eight options. this format facilitates a controlled evaluation of a model's ability to identify truthful answers among plausible but incorrect alternatives via exploratory thinking. we report the accuracy for mc0 and mc1 settings separately. ## experimental settings llms we consider both llama-3.1-8b-instruct @xcite and mistral-7b-instruct-v0.3 @xcite for evaluation. we generate 5k story pairs by prompting each model. we compute the average pairwise cosine similarity among stories generated by each model and the average cosine similarities are all lower than 0.4, which shows that our generated stories are highly diverse in semantics or topics, with details in appendix a.2. methods to mitigate gender bias in llms using the generated data, we investigate three approaches: fine-tuning (ft), direct preference optimization (dpo) @xcite , and a combined ft+dpo strategy where dpo was applied on a fine-tuned model. these methods are chosen because fine-tuning offers a direct means of adapting llm behavior, dpo has demonstrated effectiveness in aligning model outputs with human preferences, and the two techniques are often jointly employed in practical llm deployment. for fine-tuning, we use the story s f or s m as input and set the expected output to be j f neutral or j m neutral accordingly. for dpo, the input is also s f or s m , with the rejected response being the corresponding j f or j m , and the accepted response being the corresponding j f neutral or j m neutral . further details on the input/output formats and the training hyperparameters are provided in appendix a. ## model analysis a case study on model behavior changes: to better understand how our mitigation strategies affect model behavior, we examine a representative example from the genmo benchmark shown in after fine-tuning, the llama model consistently assigns a "moral" stance to both versions of the story. importantly, the judgments demonstrate more balanced reasoning: they recognize the value of social enjoyment and personal agency while also noting the relevance of parental guidance and potential consequences. although the stance is labeled as "moral," the judgments incorporate both positive and negative aspects, reflecting an increased capacity for nuanced judgment. under the dpo method, the model adopts the stance "both" for both male and female characters, explicitly presenting the moral ambiguity of the decision. the judgments outline multiple perspectives, weighing the character's desire for enjoyment against the risks of excessive drinking and the importance of responsible behavior. overall, both fine-tuning and dpo yield models that produce more consistent moral stances across genders and generate judgments that integrate both commendable and questionable aspects of a character's actions. ## conclusion we have presented a novel approach to mitigating gender bias in llms by fostering exploratory thinking. through prompting llms to generate story pairs featuring divergent moral judgments due to gender swap of the main character for otherwise structurally identical moral scenarios, we allow llms to reveal their own gender bias. we further guide llms to generate neutral and balanced moral judgments, and use them to modify models and reduce their gender bias by conducting either fine-tuning or dpo. our experiments demonstrated that both finetuning and dpo effectively reduce gender bias. notably, fine-tuning yielded more significant and consistent gains on three benchmark datasets, while dpo offered stronger performance in specific metrics, particularly when applied to the mistral model. additionally, our approach maintained or even improved performance on general benchmarks, indicating that bias mitigation need not compromise model utility. these findings underscore the value of encouraging exploratory and nuanced thinking in llms as a path toward more equitable and trustworthy ai systems. future work can extend this methodology to other social biases mitigation tasks.