Are Economists Always More Introverted? Analyzing Consistency in Persona-Assigned LLM s (2025)

Abstract: Personalized Large Language Models (LLMs) are increasingly used in diverse applications, where they are assigned a specific persona—such as a happy high school teacher—to guide their responses. While prior research has examined how well LLMs adhere to predefined personas in writing style, a comprehensive analysis of consistency across different personas and task types is lacking. In this paper, we introduce a new standardized framework to analyze consistency in persona-assigned LLMs. We define consistency as the extent to which a model maintains coherent responses when assigned the same persona across different tasks and runs. Our framework evaluates personas across four different categories (happiness, occupation, personality, and political stance) spanning multiple task dimensions (survey writing, essay generation, social media post generation, single turn, and multi-turn conversations). Our findings reveal that consistency is influenced by multiple factors, including the assigned persona, stereotypes, and model design choices. Consistency also varies across tasks, increasing with more structured tasks and additional context. All code is available on GitHub.

## introduction personalized large language models (llms) are increasingly deployed in applications where alignment with specific beliefs and values is essential, such as in high-stakes domains like healthcare and education @xcite . while prior research has examined the extent to which llms adhere to their assigned personas in terms of writing style @xcite , less attention has been given to the consistency of persona adherence across different types of tasks and prompting strategies (jiang et al., 1 https://github.com/manon-reusens/persona_ consistency 2024). moreover, it remains unclear how specifying certain persona attributes affects the consistency of other characteristics. for example, does assigning an "economist" persona to an llm ensure stable alignment across other characteristics, such as "extroversion"? additionally, which persona categories lead to the most consistent behavior and does this depend on the task at hand? addressing these questions is essential for understanding how llm personas manifest across diverse contexts and for identifying unintended spillover effects-where defining an assigned persona might reinforce unintended other characteristics. recognizing both the intended and unintended traits associated with a persona is crucial for ensuring reliable and predictable model behavior, especially in high-stake environments. prior work shows that llms can reflect big five personality traits in structured tasks, and that larger models tend to do so more consistently than smaller ones @xcite . however, persona consistency also extends beyond personality to traits like political orientation and social roles @xcite . yet most evaluations rely on ad hoc methods, such as prompt perturbations or a narrow focus on personality-based personas, resulting in an incomplete picture of consistency. interestingly, @xcite investigated whether explicitly assigning personas enhances response consistency. they found that while overall consistency decreased with persona assignment, responses became more consistent along dimensions relevant to the assigned persona. given the recent calls for application-specific evaluations of llm behavior @xcite @xcite , we propose a new standardized framework for analyzing persona consistency across a broader range of realistic tasks. moving beyond prompt perturbations and personality-based personas, our approach @xmath0: the overview of the full methodology. on the left, the persona construction is shown. from these four selected surveys, binary characteristics are selected serving as the base for the different personas. all combinations of these characteristics within a persona category are made, e.g. a character who is introverted, antagonistic, conscientious, neurotic, and open to experience. surveys are evaluated using their respective scoring key. the other dimensions are evaluated using gpt4o as llm-as-a-judge. more specifically, in this paper, we propose a standardized framework for multifaceted persona consistency analysis. we focus on four persona categories-happiness, occupation, personality, and political stance-selected for their relevance in persona-related literature, their variability in the scale of linguistic expression, and the availability of external survey instruments. we evaluate the consistency of persona-assigned llms across multiple evaluation dimensions, including survey answering, social media post generation, essay writing, single-question answering (singlechat), and multi-turn conversations (multichat). we focus on two primary aspects of consistency: (1) intra-persona consistency: whether llms remain consistent within their assigned persona; and (2) inter-persona consistency: whether llms remain consistent across other persona categories than the one assigned. we hypothesize that llms exhibit personadependent consistency effects: some personas (e.g., tied to specific professions) may induce stronger intra-persona consistency, while others (e.g., based on personality traits) lead to more partial or contextdependent patterns. we also explore potential spillover effects-whether traits associated with one persona category influence outputs in another, possibly due to underlying social stereotypes. our findings reveal that specifying a persona leads to high intra-persona consistency, with some persona categories (e.g., happiness and occupation) being more consistent than others (e.g., political stance). we also uncover spillover effects, where persona assignment reinforces inter-persona consistency driven by stereotypes and model defaults. finally, we show that consistency is influenced by task dimensions: clearer tasks and additional context length improve consistency. ## framework and metrics development here, we outline our framework and the evaluation metrics used in our experiments. ## consistency analysis in this section, we present the research questions, the experimental setup, and the results. ## discussion our framework offers a multi-dimensional perspective on the consistency of persona-assigned llms. our results show that the balance between intrapersona and inter-persona consistency varies depending on the evaluation dimension. personabased models are more consistent for attributes explicitly defined in their persona than for unspecified persona attributes, indicating a weaker spillover effect. however, in singlechat and multichat settings, we find no statistical difference between inter-and intra-persona consistency, implying that adding a persona does not significantly increase consistency compared to non-assigned attributes in more realis-tic tasks. this finding urges caution when deploying persona-assigned llms in practical applications, as persona adherence may be less consistent in conversational contexts than in more structured tasks like surveys or essays. finally, we show that longer context, i.e. comparing multichat to singlechat, allows models to express their persona more clearly leading to higher consistency. we identify three main factors influencing consistency across different characteristics. the assigned persona: models generally adhere well to persona-specific instructions, with the degree of adherence varying across categories, models and dimensions. our paper offers a valuable comparison on consistency levels across different persona categories, highlighting how some categories (e.g. happiness and occupation) are easier to consistently express than others (e.g. personality and political stance). the evaluation dimension also highly influences consistency. we show how more structured tasks and longer sequence lengths in the response result in higher consistency, especially for the identified harder persona categories, such as personality and political stance. stereotypical associations with the assigned persona: stereotypical associations play a significant role in inter-persona consistency. characteristics that align with stereotypical traits of a given persona often result in higher consistency scores. for example, persona-assigned llms instructed to be "happy" consistently score higher on extroversion. these tendencies highlight the influence of societal stereotypes embedded within the models. literature confirms this influence of stereotypes in persona-assigned llms @xcite . default persona: when a specific characteristic is not defined and a persona lacks a stereotypical association, the model often reverts to a consistent, pre-defined default persona. models exhibit social desirability bias as was found by @xcite . our findings also reveal default political personas in these models, likely reflecting design choices by developers. as shown by @xcite , ideological stances can be embedded in models, a phenomenon tied to model positionality-the social and cultural perspective developers align the model with (cambo and gergle, 2022). these choices occur throughout the development process and extend beyond training data alone @xcite . ## related work persona-assigned llms. personas can guide llms to generate responses that align with specific values and beliefs @xcite . however, they can also expose stereotypes embedded in the model @xcite , raising concerns about bias and unintended implications. persona adherence is usually evaluated using self-report scales, but @xcite use interview-based testing to capture actual model behavior, showing the need for application-specific evaluations. @xcite examine how different personas from various sociodemographic groups influence writing styles. apart from inference-time persona assignment, it is also possible to further fine-tune the model. for example, @xcite train llms to adopt specific personas using three key components: a profile (a detailed persona description), a scene (a situational context), and interactions. @xcite suggest including dialogues for persona assignment of llms. our framework can evaluate these realistic personas or finetuned models that do not perfectly fit our predefined categories, as shown in appendix j. ## conclusion this paper introduces a multi-dimensional framework for analyzing consistency in persona-assigned llms. our framework encompasses a diverse set of commonly used persona categories, including personality, occupation, political stance, and happiness. it also incorporates application-specific evaluation dimensions, such as survey answering, essay writing, social media post generation, singlequestion answering, and multi-turn conversations. we demonstrate the efficacy of our framework through a comprehensive evaluation of intra-and inter-persona consistency across personas derived from the defined persona categories. additionally, we compare consistency scores across evaluation dimensions. our analysis reveals three key factors influencing consistency in llms: (1) the assigned persona; (2) stereotypes associated with the assigned persona; and (3) model's default personas.