Elucidating Mechanisms of Demographic Bias in LLM s for Healthcare (2025)

Abstract: We know from prior work that LLMs encode social biases, and that this manifests in clinical tasks. In this work we adopt tools from mechanistic interpretability to unveil sociodemographic representations and biases within LLMs in the context of healthcare. Specifically, we ask: Can we identify activations within LLMs that encode sociodemographic information (e.g., gender, race)? We find that, in three open weight LLMs, gender information is highly localized in MLP layers and can be reliably manipulated at inference time via patching. Such interventions can surgically alter generated clinical vignettes for specific conditions, and also influence downstream clinical predictions which correlate with gender, e.g., patient risk of depression. We find that representation of patient race is somewhat more distributed, but can also be intervened upon, to a degree. To our knowledge, this is the first application of mechanistic interpretability methods to LLMs for healthcare.

## introduction llms are poised to transform the practice of healthcare in many ways @xcite @xcite , given the volume of unstructured health data and limited provider bandwidth @xcite . such models are capable of a wide range of tasks related to processing and making sense of healthcare data @xcite , from summarizing published medical literature @xcite to extracting key information from the notes within patient electronic health record (ehr) data @xcite . indeed, excitement around such uses is driving fast adoption: epic-a major vendor of ehr software-has hastily integrated gpt-4 1 introduction language models have been at the center of nlp technologies for many years @xcite @xcite @xcite . recently, due to largescale pretraining and human annotation for alignment, they have become commercially valuable @xcite . however, as their commercial value has increased, the largest models have become gated behind proprietary interfaces, with important details left undisclosed. we believe that full access to open language models for the research community is critical to the scientific study of these models, their strengths and weaknesses, and their biases and risks. accordingly, we introduce olmo, a powerful, truly open language model alongside open training data, training and evaluation code, intermediate model checkpoints, and training logs. recent lm releases have varied in their degree of openness. for example, mixtral 8x7b provided model weights and a brief report @xcite , while llama came with in-depth adaptation training instructions @xcite , and mosaic pretrained transformer came with many details, including the dataset distribution, though not the data itself (mosaicml nlp team, arxiv:2402.00838v4 [cs.cl] 7 jun 2024 @xcite we show that we can localize patient gender information in llm representations for clinical tasks. but enthusiasm around the uptake of llms in this space has been tempered by concerns over fairness and the opaque nature of large generative models @xcite . one salient concern-which preliminary work suggests is very much warranted-is that such models might exacerbate existing biases in healthcare. for instance, recent work by @xcite found that gpt-4 exaggerates associations between conditions and sociodemographic groups. specifically, when asked to generate clinical vignettes of patients with particular conditions, gpt-4 will nearly exclusively assume certain demographics (e.g., race, gender). for example, asked to generate vignettes for patients with rheumatoid arthritis, gpt-4 generates cases featuring female patients 97% of the time (the actual percent of individuals with rheumatoid arthritis who are female is about 66%; @xcite . similarly, gpt-4 associates sarcoidosis with black patients and hepatitis b with asian patients more strongly than actual population-wide correlations. in this work, we ask: is the internal llm encoding of patient demographics like gender and race localized? and, can we intervene upon this? to answer this we perform activation patching @xcite in the context of clinical vignette generation. the idea is to identify a small set of internal activations which code for patient characteristics like gender, and then verify these by intervention. we offer the following contributions: to our knowledge, this is the first investigation of mechanistic interpretability methods for healthcare. we focus on the olmo-7b-instruct model 2 following prior work @xcite , we use "gender" instead of "sex" because of the ambiguity in how llms use the terms "male" and "female" (biological vs sociocultural). most (∼ 60%) vignettes generated by olmo state "gender" followed by "male" or "female". our analysis is also limited to binary gender categories, owing to limitations of the data. 3 the actual percent of individuals with rheumatoid arthritis who are female is about 66% @xcite . throughout this paper, but show that our findings generalize across models and scale with three additional models: llama-3.1-8b-instruct @xcite , gemma-2-9b-it @xcite , and olmo-2-0325-32b-instruct @xcite . to find gender-encoding activations, we first prompt the llm to generate a vignette for a condition (strongly) associated with females (or males). specifically, following prior work @xcite , we prompt the llm to provide a succinct description of the patient-including symptoms, medical history, and demographic informationusing the same 10 prompts introduced in this prior study. one of the prompts, e.g., is: compose a brief presentation of a patient presenting with @xcite . please include complete demographic information and past medical history. we run each prompt for every condition through the model 100 times, yielding 1000 vignettes per condition. we pick two conditions for which gpt-4 exaggerates the association between gender and incidence @xcite : sarcoidosis and multiple sclerosis (ms). while these conditions are indeed more prevalent in women than men, 4 gpt-4 generates cases with female patients in 97% and 96% of the cases where gender was specified. we first confirm that olmo-7b-instruct behaves similarly. ## localizing race we have found that patient gender information is localized within llm representations; is race similarly? we repeat the exercise, using two conditions that correlate with race: sarcoidosis and hepatitis b. we again first reproduce zack et al. (2024)'s result using olmo-7b-instruct, confirming that the model disproportionately generates vignettes of black patients in the case of sarcoidosis and asian patients for hepatitis b ( figures 4a and 4b depict the average rewrite scores for hepatitis b and sarcoidosis, respectively. patching in early (ℓ=4) as well as middle (ℓ=18 -20) layers affects racial distribution. figures 5a and 5b show the ratio of the target race (race we aim to flip to) vignettes before and after intervention at layers 4 and 19 respectively. patching a single layer, even with scaling, has a less pronounced effect than for gender. does patching multiple layers at a time help? we explore sliding window patching @xcite , in which neighboring layers are simultaneously patched. for instance, patching at layer ℓ with a window size of 1 implies patching layers ℓ -1, ℓ, and ℓ + 1 simultaneously. in figures 5a and 5b , we show results for window size 5, which results in the highest improvement in the target race ratio (averaged over the 3 target races). we report results of window sizes 1 and 3 in tables 14 and 15 . sliding window patching improves target race ratio, suggesting race information is more dispersed. as we did for gender, we compute average perplexity under llama-3.1-8b over the three target races for 500 vignettes before and after patching for sarcoidosis and hepatitis b. perplexities before and after patching are comparable (figure 8 ). male female ...there is no direct indication that the patient is at risk of depression.... ...the patient's history of anxiety and alcohol withdrawal symptoms, as well as her eviction from her apartment, suggest that she may be at risk of depression.... ...the patient experienced anxiety, agitation, and delirium, but these were managed... ...the patient had persistent difficulties with anxiety over the weeks preceding admission... white black ...there is no indication that she is at risk of depression... ...is at risk of depression, given her anxiety and interest in complementary/alternative medicine for managing her mental health... ...the patient is not at risk of depression... ...at a higher risk of depression...denied suicidality, and she denied prior hospitalizations and incarcerations for which we have documentation... ## clinical applications we have established that certain patient demographic information is localized in llm representations using the task of clinical vignette generation, a straightforward setting that focuses on a single condition and has limited confounding variables. next, we broaden our analysis and look at how mechanistic interventions can be used to detect implicit biases in the context of clinical tasks where llms might be used. 8 ## conclusions we investigated if patient demographic information can be localized in llms. we found that gender information is highly localized. patient race is somewhat localized, but less so (it is somewhat distributed across model activations). we showed that implicit biases in clinical tasks can be studied by mechanistically controlling demographics, pointing to directions for future work, and potentially methods to mitigate bias in clinical tasks.