How Can We Diagnose and Treat Bias in Large Language Models for Clinical Decision-Making? (2025)

Abstract: Recent advancements in Large Language Models (LLMs) have positioned them as powerful tools for clinical decision-making, with rapidly expanding applications in healthcare. However, concerns about bias remain a significant challenge in the clinical implementation of LLMs, particularly regarding gender and ethnicity. This research investigates the evaluation and mitigation of bias in LLMs applied to complex clinical cases, focusing on gender and ethnicity biases. We introduce a novel Counterfactual Patient Variations (CPV) dataset derived from the JAMA Clinical ChallengeUsing this dataset, we built a framework for bias evaluation, employing both Multiple Choice Questions (MCQs) and corresponding explanations. We explore prompting with eight LLMs and fine-tuning as debiasing methods. Our findings reveal that addressing social biases in LLMs requires a multidimensional approach as mitigating gender bias can occur while introducing ethnicity biases, and that gender bias in LLM embeddings varies significantly across medical specialities. We demonstrate that evaluating both MCQ response and explanation processes is crucial, as correct responses can be based on biased reasoning. We provide a framework for evaluating LLM bias in real-world clinical cases, offer insights into the complex nature of bias in these models, and present strategies for bias mitigation.

## introduction despite llms offering promising potential for text generation across various domains, recent studies have shown that these models are prone to exhibiting social biases inherited from their training data @xcite . bias in this context refers to a model's systematic tendency to unfairly discriminate against certain individuals 1 code and dataset available at our github repository: https://github.com/kenza-ily/diagnose_treat_bias_llm or groups in favour of others @xcite . this can manifest as lower prediction accuracy for certain demographic groups or as disparities in the quality of generated content across different populations @xcite . in healthcare, such biases may exacerbate health disparities and unfairly impact certain patient groups, posing significant risks where discriminatory outputs could lead to disparities in patient care and health outcomes @xcite @xcite @xcite . for example, a recent study from @xcite revealed that gpt-4 exhibited a 9% lower likelihood of recommending advanced imaging for black patients and an 8% lower likelihood of rating stress testing as highly important for female patients compared to male patients. current approaches to evaluating llms in medical contexts primarily rely on multiple choice questions (mcq) from standardised exams like the united states medical licensing examination (usmle) @xcite . while some models have achieved scores comparable to or surpassing those of human medical professionals @xcite , excelling at multiple-choice questions does not necessarily equate to superior reasoning skills needed for real-world clinical practice, as highlighted by @xcite @xcite . at the same time, researchers have called for more comprehensive and clinically relevant benchmarks @xcite . in response to these concerns and to address the need for more clinically relevant evaluation methods, @xcite introduced the jama dataset, comprising complex clinical cases that test decision-making skills in realistic clinical scenarios. our work builds on this challenge, using the jama clinical challenge dataset, which provides real-world, complex medical cases along with mcqs and explanations (xpls), allowing us to evaluate the decision-making rationale behind clinical-decision making with llms. we implement counterfactual patient variations (cpvs) to evaluate bias in llms across clinical scenarios (see figure 1 ). our research explores prompt engineering and fine-tuning for bias mitigation, as well as a real-world evaluation without multiplechoice labels given. our framework incorporates a wide array of metrics for bias quantification, including accuracy comparisons, statistical measures, feature importance analysis, and embedding-based assessments. we address three main research questions: rq1: extent of llm bias in cpv across gender and ethnicity in complex clinical scenarios. rq2: effectiveness of prompt and fine-tuning strategies in mitigating bias. rq3: fairness differences between structured mcq and open-ended clinical explanations. we find that llms exhibit pervasive gender and ethnicity biases in outcomes and reasoning, with discrepancies between mcq performance and xpl quality revealing persistent biases despite apparent balanced accuracy. fine-tuning can mitigate some biases but may introduce new ones, particularly across ethnic categories. prompt engineering alone is insufficient for comprehensive debiasing, with effectiveness varying across models and demographics. gender bias in llm embeddings varies considerably across medical specialities, necessitating domain-specific debiasing strategies. our main contributions are: a) a novel cpv framework enabling systematic evaluation of bias in clinical cases. b) a comprehensive bias evaluation in clinical llms, incorporating both mcq performance and explanation quality metrics. c) insights into the complex nature of bias in clinical llms explanations from their embeddings, including the variability across medical specialities and the discrepancy between mcq performance and explanation biases. d) evaluation of various prompting and finetuning strategies for bias mitigation, highlighting their strengths and limitations. clinical case feature extraction to facilitate gender swapping, identify questions asked, and gain insights into the patient population, we conducted extensive preprocessing of the dataset. this process began with a thorough human analysis of numerous clinical cases, which prompted the development of a rule-based system for feature extraction and case exclusion. this preliminary analysis helped identify the gender of cases in the dataset, which were male, female and neutral. preprocessing steps included extracting patient demographics (age, gender, ethnicity) using regex-based pattern matching; detecting gender-specific medical conditions (e.g., pregnancies, women's health issues) for appropriate case exclusion; normalising clinical questions into three standardized formats; and implementing answer option randomisation to mitigate potential selection biases @xcite . the rule-based system was iteratively refined based on human evaluation of its performance on a subset of cases. more details for these processes are available in appendix a.2. ## methodology model selection we selected a diverse range of llms for our experiments, including gpt-3.5 (gpt-3.5-turbo-0301), gpt-4o (gpt-4o-2024-05-13), 2 we note that evaluating biases in the medical domain is particularly challenging due to the intricate interplay between attributes such as sex and hormones, which can significantly influence various biomarkers and complicate the interpretation of research outcomes; for instance, studies have shown that the reliance on male subjects in clinical trials often leads to misleading conclusions about drug efficacy and safety for women, highlighting the necessity for a more nuanced approach that considers these interrelationships @xcite gpt-4 turbo (gpt-4-turbo-2024-04-09), haiku (claude3 haiku), sonnet (claude 3.5 sonnet), gemini (gemini 3.5 flash), llama3 (llama3-70b), llama3.1 (llama3.1-403b) for inference, as well as gpt-4o mini for fine-tuning. inference and prompts we developed multiple prompting strategies to evaluate different approaches to bias mitigation, based on initial work by @xcite and prompting guidelines from @xcite , @xcite , and @xcite . for the exploratory cpv experiment, we enhanced the prompt by incorporating chain-of-thought (cot) reasoning @xcite and follow-up questions about gender and ethnicity relevance. for the prompt bias mitigation evaluation experiment, we implemented three distinct prompts: a baseline question (q), a debiasing prompt adding instruction following (q+if), and a combination of debiasing instructions with chain-of-thought (cot) reasoning (q+if+cot), a framework based on @xcite . finally, the ablation study without multiple-choice used a modified version of the prompt mitigation's baseline prompt adapted not to provide the mcq options. all the prompts are reported in appendix g. to ensure consistent and deterministic outputs across all experiments, we set the temperature parameter to 0 for deterministic generation @xcite . fine-tuning for the fine-tuning experiment, we employed two task-specific paradigms: mcq (multiple choice question) and xpl (explanation). for the mcq task, we fine-tuned models on a dataset with case descriptions and options, outputting only the answer, while for the xpl task, we fine-tuned on a dataset with cases, options, and solutions, outputting only the explanation. we used openai's fine-tuning platform with gpt-4o mini. the datasets for both tasks were carefully curated to ensure a balanced representation across genders and ethnicities, with the mcq dataset containing 1,409 training examples and the xpl dataset containing 4,044 training examples. for the mcq task, we trained for 2 epochs with a batch size of 32 and a learning rate multiplier of 0.8. the xpl task was trained for 3 epochs with a batch size of 2 and a learning rate multiplier of 1.8. these hyperparameters were selected based on multiple iterations and performance on the validation set, balancing between model performance and generalisation. ## experiments our experiments use a system-and-user prompt structure to query llms about clinical cases, evaluating their responses for potential biases. each experiment prompted the models to provide both an mcq response and an accompanying explanation, allowing us to assess bias in both decisionmaking and explanation, in a predict-then-explain framework @xcite . detailed dataset statistics per experiment are available in appendix a. we conducted four main experiments to evaluate and mitigate bias: exploratory cpvs we aimed to assess the extent of bias in llms when presented with cpv across gender and ethnicity: we evaluate how introducing intersectionality through gender and ethnicity cpv may reveal complex bias patterns in llms that may not be apparent when examining gender or ethnicity in isolation. the prompt used incorporated chain-of-thought reasoning and follow-up questions about gender or ethnicity relevance. ## conclusion in this work, we demonstrate the intricate nature of bias in llms for clinical applications through a comprehensive evaluation framework. our findings reveal pervasive gender and ethnicity biases in both mcq performance and explanation quality, with significant discrepancies between surface-level accuracy and underlying reasoning biases. this complexity underscores the need for frameworks that consider multiple bias evaluation metrics, as our multifaceted analysis reveals a much richer picture than simple accuracy assessments. by examining various aspects of llm output, we unveil layers of bias that might otherwise remain hidden. the effectiveness of bias mitigation strategies varied across models and social attributes, while gender bias in llm embeddings showed substantial variability across medical specialities. these nuanced results highlight the limitations of one-size-fits-all approaches and underscore the need for domainspecific strategies, and lack a deeper evaluation of qualitative results, as displayed in figure 5 . our methodology and dataset aim to offer substantive groundwork for future research, providing a foundation to explore the development of more equitable llm-based clinical decision support systems in real-world settings.