The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Models (2025)

Abstract: Persona prompting is increasingly used in large language models (LLMs) to simulate views of various sociodemographic groups. However, how a persona prompt is formulated can significantly affect outcomes, raising concerns about the fidelity of such simulations. Using five open-source LLMs, we systematically examine how different persona prompt strategies, specifically role adoption formats and demographic priming strategies, influence LLM simulations across 15 intersectional demographic groups in both open- and closed-ended tasks. Our findings show that LLMs struggle to simulate marginalized groups but that the choice of demographic priming and role adoption strategy significantly impacts their portrayal. Specifically, we find that prompting in an interview-style format and name-based priming can help reduce stereotyping and improve alignment. Surprisingly, smaller models like OLMo-2-7B outperform larger ones such as Llama-3.3-70B.Our findings offer actionable guidance for designing sociodemographic persona prompts in LLM-based simulation studies.

## introduction an increasing number of studies employ persona prompts to replace or supplement human input in social surveys, predict voting behavior, or perform subjective annotation tasks @xcite @xcite . persona prompting intends to condition a models' output to reflect the characteristics of specific personas, enabling researchers to simulate opinions, values, and attitudes more effectively. however, llms are known to be susceptible to seemingly minor prompt variations @xcite , and prior work shows that outcomes can vary significantly depending on prompt formulation @xcite . in line with @xcite , we hypothesize that subtle contextual differences in persona prompts can activate different perspectives in llms, which, in turn, figure 1 : evaluation framework for sociodemographic persona prompting. we construct sociodemographic persona prompts using combinations of three different role adoption formats and three strategies for demographic priming. we populate these prompts in conjunction with various sociodemographic groups and systematically evaluate them across both open-and closed-ended tasks using a broad set of bias and alignment measures. may influence their downstream behavior. this may occur because small changes in phrasing lead the model to draw on different aspects of its learned knowledge, depending on which associations are most strongly triggered by the input. as a concrete example, a model may simulate a group better if the demographic attributes are implicitly induced into the llm, e.g., based on names containing demographic cues, rather than through explicit demographic descriptors which can provoke stereotype effects @xcite . however, research on sociodemographic persona prompting is in a nascent and exploratory phase where it remains unclear how different persona prompt strategies affect the representation of the subpopulations being modeled. the lack of clear guidelines has led to considerable variation in the articulation of demographic dimensions ("asian woman" @xcite vs. "ms. huang" @xcite ), and in how the persona's point of view is constructed ("a person can be described as follows: race: x " @xcite vs. "you are a person of race x " @xcite ). to inform the design of our experiments, we conduct a survey of prompt strategies commonly used in llm-based simulations studies. we identify various role adoption formats, i.e., ways of prompting the model to take on a role -through direct role assignment ("you are a black woman"), using the third person singular ("think of a black woman"), or through a structured format where identity elements are introduced in an interview-like format ("interviewer: what is your race? interviewee: my race is black"). additionally, we examine the impact of demographic priming, i.e., how demographic attributes are conveyed to the model -implicitly via names ("ms. gonzalez"), explicitly via demographic terms ("hispanic woman") or explicitly via demographic categories and terms ("a person of race hispanic, and gender female"). using this framework, we evaluate how well llms represent demographic groups in both open-ended and closed-ended tasks -demographic biases in self-descriptions and bios and opinion alignment on survey questions @xcite , respectively (cf. figure 1 ). research questions. we specifically study: results. we find that llms do not simulate all sociodemographic groups equally well. in particular, prompts that invoke nonbinary, hispanic, and middle eastern personas tend to elicit more stereotypical responses than for other personas. however, when systematically comparing differ-ent persona prompting strategies, we observe that interview style and name-based prompting result in less stereotypical associations, better opinion alignment, and reduced disparities between groups. interestingly, larger models such as llama-3.3-70b are less effective at simulating demographic groups, whereas the best simulation performance is achieved by olmo-2-7b. ## sociodemographic persona prompting sociodemographic persona prompting refers to a prompting technique that is used to steer the behavior of an llm to align with that of a specified sociodemographic group or person @xcite . we devise a framework to systematically analyze how different persona prompt types affect the simulation of sociodemographic groups in llms. we begin by identifying common prompt types from prior studies, which we then apply to simulate a variety of sociodemographic personas in both open-and closed-ended tasks. as our foundation, we draw on the dataset curated by @xcite , which compiles studies examining demographic representativeness in llms and includes annotations indicating whether a sociodemographic prompting approach was used. from this dataset, we sample 47 papers and extract their original prompts. our review reveals recurring patterns in persona prompt types across studies, which we distill into two primary axes of variation (see appendix a for further details on the review): based on these dimensions, we construct sociodemographic prompts and systematically examine their effect on llm behavior across several tasks. ## experimental setup our experiments follow a two-part prompt structure. each prompt template consists of a persona segment, where we vary both the role adoption format and demographic priming, followed by a task segment that remains consistent across conditions. to ensure that the observed effects can be attributed to the variations introduced in section 2, and not to wording unrelated to these dimensions, we include two alternative phrasings for each template. we then use them to simulate various demographic personas, covering 15 intersectional race/ethnicity and gender groups (8 groups for the closed-ended task). an overview of all prompt templates and the full set of demographic descriptors can be found in appendix b. we conduct all analyses in english. ## results we now report the results of our analyses of the five different llms by first focusing on (i) demographic representativeness, i.e., how well the llms simulate different identity groups for the two types of tasks and their corresponding evaluation metrics. then, we assess the impact of (ii) strategies for persona prompting. finally, we compare the five different models and explore whether some are more representative than others. for the open-ended tasks, we show results for the self-description task in the main body and report complementary results for the social media bio task in appendix e.2. ## analysis of marked words to complement the quantitative findings in section 4, we also analyze the marked words identified in self-descriptions across all combinations of demographic group and prompt type. to this end, we compute the most frequent marked words per demographic group and present the top 10 in for each category, we calculate the share of personas from the associated groups whose selfdescriptions include such words and compare between prompt strategies. figure 6 shows results for category 1 and 2: using names and/or the interview format consistently lowers the share of stereotyped persona descriptions across models. 11 6 related work personas in llms. personas, widely used in interaction design to represent user archetypes @xcite , have recently been adapted to mold the behavior of language technologies like chatbots and llms. personas in llms span e.g. psychometric, occupational, and demographic dimensions to guide model behavior and tailor responses to contexts or user needs @xcite . inducing personas in llms effectively is an active area of research @xcite @xcite . we study demographic personas which can be used to create synthetic samples of particular human subpopulations @xcite . ## discussion and conclusion our findings show that llms often stereotype marginalized groups, especially nonbinary, hispanic, and middle eastern personas, highlighting ongoing challenges in representing demographic groups @xcite . however, we find that the choice of demographic priming and role adoption strategies significantly group top-10 marked words asian woman heritage, dark, petite, long, modern, almondshaped, cultural, bun, golden, delicate asian man chinese, korean, respect, styled, heritage, martial, vietnamese, slender, japanese, traditional asian nonbinary pers. using last names for demographic priming also shows promise in mitigating biases and aligns with @xcite , who found similar effects using first names. however, using first names to signal gender identity, can entrench biases towards certain groups, e.g, nonbinary people @xcite . we circumvent this by instead relying on titles (i.e, mr., ms., and mx.) to signal gender. nevertheless, we observe that models occasionally interpret last names in unintended ways (for example, some names associated with black identity in the u.s. were linked to french residents), posing challenges to validity @xcite . thus, we suggest treating names as a potentially useful but ethically fraught tool, that requires careful implementation and critical reflection on its limitations. a new observation is that llms switch language during simulation, but only when simulating hispanic personas. to the best of our knowledge, this has not been studied before. investigating whether this extends to other identities and languages, and how it relates to llms' multilingual capabilities, is a promising direction for future work. finally, we stress the importance of clearly documenting and justifying choices around role adoption, demographic priming, and the specific descriptors used in persona prompts. to support future work, we release our code and datasets. 12