Neutral Is Not Unbiased: Evaluating Implicit and Intersectional Identity Bias in LLM s Through Structured Narrative Scenarios (2025)

Abstract: Large Language Models often reproduce societal biases, yet most evaluations overlook how such biases evolve across nuanced contexts or intersecting identities. We introduce a scenario-based evaluation framework built on 100 narrative tasks, designed to be neutral at baseline and systematically modified with gender and age cues. Grounded in the theory of Normative-Narrative Scenarios, our approach provides ethically coherent and socially plausible settings for probing model behavior. Analyzing responses from five leading LLMs—GPT-4o, LLaMA 3.1, Qwen2.5, Phi-4, and Mistral—using Critical Discourse Analysis and quantitative linguistic metrics, we find consistent evidence of bias. Gender emerges as the dominant axis of bias, with intersectional cues (e.g., age and gender combined) further intensifying disparities. Our results underscore the value of dynamic narrative progression for detecting implicit, systemic biases in Large Language Models.

## introduction in recent years, there has been growing scholarly interest in understanding how large language models (llms) influence and reflect societal norms, particularly regarding gender roles and other social categories @xcite . as these models are trained on vast corpora of human language, they often inherit and amplify linguistic patterns that reinforce societal stereotypes @xcite . these biases raise serious ethical and societal concerns, as they may contribute to prejudiced or discriminatory outcomes in real-world applications @xcite . prior studies have shown that llms frequently associate occupations with traditional gender roles, for instance, linking "doctor" with men and "nurse" with women, regardless of real-world demographic data @xcite @xcite . research, such as @xcite and @xcite , has focused mainly on explicit gender associations within occupational prompts, often using templatebased datasets or short, controlled sentences, like those in winobias-style evaluations @xcite . while recent efforts have begun to explore implicit bias-biases revealed without overt demographic language-they typically rely on static prompts or narrowly defined task formats @xcite @xcite . for example, @xcite examines implicit and explicit gender cues by designing template-based prompts for professional vs. domestic contexts, and @xcite quantifies stereotype propagation across 106 identity intersections using frequency metrics. however, these studies either remain limited to single-turn or single-attribute comparisons or rely on structured sentence prompts rather than narrative settings. moreover, the interaction between multiple demographic cues-such as gender, age, occupation, or race-is often neglected or treated in isolation. while some recent work explores intersectional biases @xcite , it typically does so in precategorized datasets rather than evolving, contextrich scenarios. very few studies examine how biases shift as demographic cues move from implicit to explicit across narrative contexts, or how such shifts affect role attribution and moral reasoning. to address these limitations, our study proposes a scenario-based evaluation framework using normative narrative scenarios @xcite to reveal both explicit and implicit biases in llm outputs. unlike prior work using short prompts or q&a formats, we embed identity-neutral fictional stories where two individ-uals divide everyday tasks. these scenarios are consistent, role-dividing, and free of overt demographic markers, providing a rich yet controlled setting to assess the emergence of bias. to ensure that observed biases reflect model behavior rather than narrative structure, scenarios were first crafted to be neutral and balanced. this provides a consistent baseline for assessing how identity cues influence role attribution and normative reasoning, and supports future analyses of other intersections, such as race, class, or occupation. in this regard, our main contributions are fourfold: to evaluate llm behavior consistently, we pair each scenario with a universal set of neutrally phrased follow-up questions, avoiding direct reference to demographic categories. these are issued as single-turn prompts to prevent memory or conversational carryover-an improvement over multiturn setups in prior work @xcite . we extend our analysis across multiple foundation models to assess whether observed biases are model-specific or systemic, and propose a feedback-oriented prompting strategy for mitigation-shifting from one-time debiasing to adaptive evaluation. in summary, our study presents a multidimensional, scalable, and theory-grounded approach to understanding how llms reproduce and rationalize social norms. by combining cda with scalable bias testing, we move beyond prompt refinement methods @xcite and present a flexible evaluation strategy for detecting and mitigating bias in llms. our code and data are available at https://github.com/ saba-gh-h/neutral_isnot_unbiased . ## review of the related literature research on gender and social bias in llms has rapidly expanded, with numerous studies demonstrating how these models perpetuate societal stereotypes through their language generation capabilities. a central focus has been occupational gender bias. for example, @xcite shows that llms often associate male-linked professions with higher salaries, while @xcite identifies persistent stereotypes (e.g., "doctor" as male, "nurse" as female) using a modified winobias dataset. @xcite further highlights such associations across languages, underscoring the global scope of the issue. building on these findings, several studies explore bias mitigation strategies. @xcite applies prompt engineering and in-context learning to steer models toward more neutral outputs, while @xcite distinguishes between explicit and implicit bias, proposing selfevaluation and dataset refinement as corrective tools. yet, these methods often rely on short prompts or isolated sentences and do not examine how bias emerges in more naturalistic or narrative settings. recent work has begun to address implicit bias, particularly in non-obvious contexts. @xcite design prompts to elicit implicit stereotypes in professional vs. domestic settings using logit-based metrics and gender-attribute scores. similarly, @xcite and @xcite investigate how subtle linguistic cues perpetuate stereotypes, often using static or template-based prompts. studies like @xcite explore intersectionality by evaluating stereotypes across 106 demographic groups using a stereotype frequency metric (sdeg). while broad in scope, these approaches use categorical labels and fixed inputs, lacking the progressive structure of narrative scenarios. likewise, @xcite assesses gender bias in decision-making using the demet prompts dataset, finding a systematic preference for women and neutral names, but without tracking shifts as identities are incrementally introduced. efforts to align llms via prompt optimization and selfdebiasing have also emerged. techniques such as mecod @xcite , dynamic reweighting @xcite , and self-alignment @xcite aim to reduce bias at the output level, though they focus more on task performance than on how models internalize and reproduce social norms. in contrast, @xcite uses a scenario-based approach to detect escalating offensive content in llms, highlighting inconsistencies in bias expression. however, their focus is on overt harms, while ours targets subtler, normative stereotypes within everyday narratives. work in natural language generation (nlg) and dialogue systems has explored adversarial testing @xcite , counterfactual augmentation @xcite , and structured tools like the prompt association test (p-at) @xcite . these offer valuable methods but typically fall outside role-dividing, multi-character narratives. in domain-specific contexts such as healthcare and education, bias detection efforts stress risks for marginalized users. @xcite and @xcite emphasize transparency and customized benchmarks in clinical llms, while @xcite track bias across educational tool development. though important, these works do not address general-purpose social reasoning in narrative settings. finally, in moral and psychological reasoning areas, @xcite and @xcite examine gender preferences in ethical decision-making and mental health advice. these works highlight embedded values in llms but focus on dilemmas or decision prompts rather than interactive social narratives.prior work has advanced bias detection, but mostly with static prompts, isolated traits, or domain-specific tasks; few track how bias evolves under layered identity cues in everyday, role-sharing narratives-our focus. ## methodology this section presents our multi-step methodology for evaluating implicit and explicit bias in llms through structured narrative scenarios. our approach combines scenario design, controlled prompting, model testing, and discourse analysis to systematically track how biases emerge and shift by introducing social identity cues. see 1. ## conclusions and future work this study demonstrates that llms systematically reproduce gender and age-based biases, even in identity-neutral contexts, with bias intensifying under intersectional conditions. a key contribution is our design of neutral, role-divided narrative scenarios that can be dynamically modified to introduce identity cues-enabling controlled counterfactual comparisons across variations. combined with cda and quantitative linguistic metrics, this framework reveals both surface-level sentiment shifts and deeper discursive asymmetries. we find that intersecting identities-such as being both female and younger-compound disparities in role framing and evaluative language. our method moves beyond static prompts, offering a scalable and contextrich strategy for detecting implicit and systemic bias in llms. future work will extend this framework to more complex social dynamics, additional identity dimensions (e.g., race, class, disability), and explore mitigation strategies such as user-inthe-loop feedback and scenario-based fairness auditing at deployment.