Bias after Prompting: Persistent Discrimination in Large Language Models (2025)

Abstract: A dangerous assumption that can be made from prior work on the bias transfer hypothesis (BTH) is that biases do not transfer from pre-trained large language models (LLMs) to adapted models. We invalidate this assumption by studying the BTH in causal models under prompt adaptations, as prompting is an extremely popular and accessible adaptation strategy used in real-world applications. In contrast to prior work, we find that biases can transfer through prompting and that popular prompt-based mitigation methods do not consistently prevent biases from transferring. Specifically, the correlation between intrinsic biases and those after prompt adaptation remained moderate to strong across demographics and tasks: gender (rho >= 0.94) in co-reference resolution, and for age (rho >= 0.98), religion (rho >= 0.69), etc., in question answering. Further, we find that biases remain strongly correlated when varying few-shot composition parameters, such as sample size, stereotypical content, occupational distribution and representational balance (rho >= 0.90). We evaluate several prompt-based debiasing strategies and find that different approaches have distinct strengths, but none consistently reduce bias transfer across models, tasks or demographics. These results demonstrate that correcting bias, and potentially improving reasoning ability, in intrinsic models may be reliable ways to prevent propagation of biases to downstream tasks.

## introduction large language models (llms) excel in many tasks and are used in real-world systems @xcite @xcite , including tasks for which models were not (pre-)trained. this means that evaluating the effects of adaptation methods on bias is a growing ethical concern. previous works have studied the correlation between the bias of a pre-trained model and its fine-tuned counterpart @xcite , 2022; @xcite @xcite , with @xcite coining the term bias transfer hypothesis (bth); bth is the theory that social biases (such as stereotypes) internalized by llms during pre-training are also reflected in harmful task-specific behaviors after models are adapted. these works largely find that bth does not hold in masked language models (mlms) when finetuned, but research is notably overlooked regarding causal language models (arguably the most used architecture) under prompt adaptation (an accessible, and sometimes the only available, model adaptation). the notion that bias does not transfer @xcite @xcite poses significant fairness concerns in adapted models as it suggests that the fairness of pre-trained models is inconsequential. we argue that this context-specific conclusion does not generalize to other settings, including those adapted through methods other than fine-tuning; our findings on causal models using prompting reveal that bias can transfer and that accounting for intrinsic biases in pre-trained models before prompt adaptation is crucial to ensure fairness in prompt-adapted downstream tasks. while the term "transfer" can suggest a causal link, existing literature primarily establishes correlation. consistent with prior work, our study makes no claims about causality, and demonstrates bias transfer through correlation. bias transfer in llms must be understood past mlms, as they differ from causal models in their task, learning objective, and size @xcite . causal models are implemented using uni-directional transformers to predict the next token given a context, whereas mlms employ bidirectional architectures to predict masked tokens in input sequences. additionally, causal models have significantly more parameters (e.g. gpt-3: 175b) compared even to the largest mlms (e.g. roberta-large: 355m). these differences may impact models' ability to perpetuate societal biases and highlight the need to study bias transfer in language models beyond mlms. beyond differences in architecture and scale of mlms and llms, the choice of adaptation strategy also shapes how bias transfers in llms. task-specificity of models is not only achieved through full-parameter fine-tuning. prompting has emerged as an important strategy for llm adaptation @xcite to perform downstream tasks (such as multiple-choice question-answering or translation) @xcite @xcite . some factors restricting adoption of fine-tuning based adaptations are lack of compute budget (number of gpus, storage or memory), task-specific data, ml expertise for finetuning, and restricted pre-trained model weights. prompting and fine-tuning are distinct and complementary approaches, as prompting modifies inputs rather than model parameters. studying bias transfer under prompt adaptation is crucial given its widespread adoption @xcite ), yet its bias transfer dynamics are poorly understood; our work directly addresses this gap by investigating bias transfer in causal models under prompting strategies that are accessible to non-expert users. we make four key contributions: 1) a unified metric, selection bias (sb), to analyze both intrinsic and extrinsic biases, departing from prior bth works that used separate metrics for each. by using this single metric, we can directly compare intrin-sic and extrinsic biases, yielding trustworthy bias transfer analysis. 2) we evaluate the correlation of intrinsic with extrinsic biases resulting from zero-, few-shot and cot prompting. we find moderate to strong bias transfer across various prompting strategies, demographics and tasks, indicating a pervasive issue. for instance, this is exemplified by gender (ρ ≥ 0.94) in co-reference resolution, and age (ρ ≥ 0.98) and religion (ρ ≥ 0.69) in question answering. for clarity, and without loss of generality, the main body presents findings on gender bias, while app. f details results for other demographics. 3) we probe the extent to which biases transfer when few-shot composition is systematically varied. we find that few-shot choices, including number of few-shot samples (ranging between 20 and 100), their stereotypical makeup (pro-or anti-stereotypical pronoun with respect to the referent occupation) and occupational distribution (in-or out-of-distribution; balanced or biasweighted resampling) can help reduce bias magnitude, yet models continue to show strong bias transfer (ρ ≥ 0.90). 4) we investigate a suite of existing and novel prompt-based debiasing strategies to mitigate bias transfer in llms. notably, none consistently eliminate bias across all models, tasks or demographics, implying current methods are insufficient to mitigate bias transfer. our findings highlight the critical need for fairness in pre-trained models (before prompt adaptation) to reliably prevent bias transfer. ## related works previous works @xcite @xcite @xcite on bias transfer found intrinsic biases in mlms, like bert @xcite , to be poorly correlated with extrinsic biases on pronoun co-reference resolution. conversely, @xcite found that intrinsic biases do transfer to downstream tasks, and that intrinsic debiasing can improve downstream fairness. @xcite attribute these conflicting findings with incompatibility between intrinsic and extrinsic bias metrics. furthermore, they suggest prompt templates and seed words influence bias transfer, finding no significant correlation between intrinsic and extrinsic biases. while all above works examined the effect of intrinsic debiasing on extrinsic fairness, @xcite study the impact of extrinsic debiasing on intrinsic fairness, and suggest that redesigned intrinsic metrics could better indicate downstream biases than the standard weat metric @xcite . the takeaways from some of the above papers are in direct contradiction with that of others, potentially due to metric inconsistencies. importantly, all of the above works limit their bias transfer research to mlms and fine-tuning, unlike our study of causal models, which differ significantly in implementation and use. despite separate studies on intrinsic biases @xcite and downstream / extrinsic biases under prompt adaptations @xcite @xcite in causal models, the relationship between the two remains unclear. @xcite study the correlation between intrinsic and extrinsic biases on both mlms and causal models and find a lack of bias transfer due to metric misalignment and dataset noise. however, their bias transfer evaluation is limited to the fine-tuning adaptation. @xcite evaluate misinformation biases in mlms and causal models and their relationship with data, intrinsic biases, and extrinsic biases, but do not study stereotypes (generalized and unjustified beliefs about a social group) resulting from prompt adaptations. while @xcite also study bias transfer in causal models, their study differs fundamentally from ours. we examine how prompting affects the transformation of intrinsic biases into extrinsic biases. in contrast, they investigate how fine-tuning transfers intrinsic biases to fine-tuned models, using prompting only as a tool to reveal biases, but do not study the impact that prompting can have on bias transfer. @xcite study bias transfer in causal models under prompting, but differ in their focus on settings where the model gates / rejects responses in the downstream setup. overall, prior work has not shown significant bias transfer from pre-trained models to downstream tasks during fine-tuning. this raises concerns that pre-trained biases might be considered irrelevant to downstream models when other adaptation strategies are used. further, previous approaches have a critical limitation: they measure intrinsic and extrinsic biases independently, using different metrics. this hinders establishing a clear correlation between them, potentially due to either the disparate metrics or a genuine lack of correlation between intrinsic and extrinsic bias. in contrast, we introduce a bias transfer analysis using unified metrics across both intrinsic and extrin-sic biases to effectively examine the relationship between these biases in llms, and demonstrate that biases in the pre-trained models can transfer to downstream tasks. our work focuses on bias transfer in causal models under prompting using unified metrics, by studying bias in various prompting strategies, demographics and tasks.