P ersona G ym: Evaluating Persona Agents and LLM s (2025)

Abstract: Persona agents, which are LLM agents conditioned to act according to an assigned persona, enable contextually rich and user-aligned interactions across domains like education and healthcare.However, evaluating how faithfully these agents adhere to their personas remains a significant challenge, particularly in free-form settings that demand consistency across diverse, persona-relevant environments.We introduce PersonaGym, the first dynamic evaluation framework for persona agents, and PersonaScore, a human-aligned automatic metric grounded in decision theory that enables comprehensive large-scale evaluation. Our evaluation of 10 leading LLMs across 200 personas and 10,000 questions reveals significant advancement opportunities.For example, GPT-4.1 had the exact same PersonaScore as LLaMA-3-8b despite being a more recent and advanced closed-source model. Importantly, increased model size and complexity do not necessarily enhance persona agent capabilities, underscoring the need for algorithmic and architectural innovation toward faithful, performant persona agents.

## introduction as the applications of llm agents continue to rapidly diversify (customer service chatbots @xcite , code generation @xcite , robotics @xcite , etc.), a new frontier presents itself in personalizing agents to align with different users. persona agents, i.e., llm agents assigned with a persona, have emerged as the community standard to enable personalized user experiences at scale @xcite @xcite . persona agents are a powerful construct and can imbibe the assigned persona and extrapolate to generate outputs from a persona-specific distribution (figure 1 ). these persona agents have demonstrated potential in diverse and personalized dialogue generation across various contexts @xcite @xcite , enhanced performance in tasks such as mathematical reasoning, physics, and software development @xcite @xcite , and simulating human behavior for scientific research in domains such as psychology @xcite @xcite . however, progress on persona agents has been severely limited by a lack of robust and targeted large-scale evaluation. while recent research @xcite has addressed this to some extent, they exhibit major limitations: (1) they are static and constrain evaluation to predetermined personas. therefore, they fail to capture the full diversity of possible agents and raise concerns about potential data contamination with new sota llms. (2) the persona agents are not initialized in environments relevant to the agent (i.e. a cowboy agent should be tested in farmrelated environments); and (3) these benchmarks are uni-dimensional and fail to holistically evaluate personalized agents @xcite @xcite @xcite . we propose personagym, the first dynamic evaluation framework for persona agents. per-sonagym enables large-scale, multi-dimensional, and targeted evaluation of any arbitrary persona agent assigned to any arbitrary persona. to support this automated evaluation across any persona in any environment, we introduce personascore-the first automatic metric aligned with human judgment that captures how well a persona agent acts in accordance with its assigned persona across diverse tasks and settings. personagym operates through a three-stage evaluation pipeline. 1) dynamic environment selection: an llm reasoner selects relevant environments from a pool of 150 diverse domains based on the agent's assigned persona. 2) persona-task generation: task-specific questions are dynamically generated to probe the agent's persona-consistent behavior across each environment. 3) agent response evaluation: the agent responds using a personaspecific system prompt. then, personascore evaluates these responses using expert-curated rubrics. to align personascore with human preferences, we first generate exemplar responses at each rubric level using llm reasoners, effectively calibrating the evaluators. multiple state-of-the-art llm evaluators then independently score the agent's responses, and we ensemble their judgments to ensure robustness and reduce individual model bias. this dynamic framework avoids the pitfalls of static evaluation-namely, data contamination, lack of personalization, and limited coverage-by tailoring environments, questions, and evaluation criteria to the persona. to support standardized comparisons across research, we also release a static benchmark consisting of 200 personas and 10,000 questions, while preserving personagym's extensibility for custom persona-agent evaluations. to ensure methodological rigor, personagym enforces a strict separation between evaluator models and evaluated persona agents, mitigating circular evaluation concerns. we also adopt a modular architecture for personagym that supports model swapping across roles. this prevents overfitting to any single model's biases and enables fairer evaluation-consistent with prior best practices @xcite @xcite . we benchmark the capability of ten leading open and close source llms (namely gpt-3.5, llama-2-13b, llama-2-70b, llama-3-8b, claude 3 haiku, claude 3.5 son-net, gpt-4.1, gpt-4.5, llama-3.3-70b, and deepseek-v3) to act as persona agents in per-sonagym. these models were evaluated on 200 diverse personas encompassing 10, 000 questions. personagym demonstrates significant weaknesses in even the latest sota models such as claude 3.5 sonnet and gpt-4.5, that fail to outperform less advanced models such as gpt-3.5 at the level they do on other tasks and domains. importantly, our results indicate that a model's increased size or capacity is not a definite indication of its persona agent capabilities. for example, we show that claude 3 haiku is very resistant to generating responses while being a persona agent despite being a sota model. this finding should motivate future studies to carefully study the ability of all sota llms to be persona agents before deployment and to push toward highly capable and faithful persona agents. our main contributions are as follows: ## evaluation tasks in the context of persona agent evaluations, we define the environment as external settings or conditions within which agents operate and interact. understanding how agents interact with their environment is crucial for assessing their performance and capabilities. since agent behavior stems from environment: desert question: if you ran out of water in the desert, what would you do in order to survive? personascore dynamic environment selection persona agent question generation action justification expected action linguistic habits persona consistency toxicity control figure 2: in personagym, relevant environments are selected from a pool of 150 diverse options using an llm reasoner based on persona descriptions. the persona agent is then initialized in these environments and responds to probing questions across five evaluation tasks. final personascore is determined by two strong llm evaluators. rolellm (li et al., 2023) roleeval (xu et al., 2023) incharacter (xu et al., 2024) personagym (ours) arbitrary personas ✗ ✗ ✓ ✓ persona-tailored ques. ✗ ✓ ✗ ✓ multidimensional ✓ ✓ ✗ ✓ open-ended ✓ ✗ ✓ ✓ decision theory provides a comprehensive theoretical foundation for our evaluation through three distinct branches, each directly informing specific evaluation tasks: ## human evaluation human experimental settings to test the alignment of personascore with human judgment, we conducted a human study on a subsample of 100 personas across three models (gpt-3.5, llama-2-13b, and llama-2-70b) in our experiments, totaling 1500 model responses. the annotators were five experts with university-level english proficiency and substantial world knowledge of different personas and their expected characteristics. the annotators were informed that they annotations would be used only to gauge overall alignment with model generated scores and they were provided the same rubric as the evaluator models (see appendix a.4) and asked to indicate their assigned score. these strong correlations validate person-agym's potential for large-scale automated evaluation of persona agents, demonstrating its alignment with human judgment. interestingly, llama-2-13b demonstrates higher correlations with human evaluations compared to gpt-3.5 and llama-2-70b in several key tasks, particularly excelling in persona consistency. this unexpected performance suggests potential ambiguities in responses from larger models, evident in llama-2-70b's lower spearman correlation scores for persona consistency and linguistic habits. further experiments showing the efficacy of our human evaluation is present in appendix f model-human agreement case appendix c illustrates strong alignment between personagym and human evaluations across different llms. for the 36-year-old australian environmental lawyer persona, all models adapted their linguistic style to the courtroom context. llama-2-13b received the highest score (4.5) from both evaluation methods, likely due to its specific references to indigenous peoples and australian colloquialisms that aligned with the persona. gpt-3.5 and llama-2-70b scored 4.0, indicating competent but less tailored performances. this case demonstrates per-sonagym's capacity to assess context-aware linguistic patterns. model-human disagreement case appendix c also presents an evaluation discrepancy case. for a 22-year-old london writer persona, personagym assigned high scores (4.5, 4.5, 4.0) while human evaluators gave substantially lower scores (2.0, 2.0, 3.0). only llama-2-70b incorporated british vernacular, and all responses lacked the sophisticated language expected from a writer describing artwork. this disparity highlights an opportunity to improve personagym's ability to penalize responses that fail to establish and maintain expected linguistic characteristics of a given persona. @xcite developed a framework assessing character fidelity through psychological scales and likert evaluations. @xcite established charactereval, a chinese benchmark containing 1,785 multi-interaction dialogues from novels and scripts. @xcite created roleeval, a bilingual benchmark with 6,000 multiple-choice questions assessing memorization and reasoning across 300 personas. ## conclusion we present personagym, the first dynamic evaluation framework for llm persona agents that assesses performance across five tasks using personaspecific questions. grounded in decision theory, our approach transcends static evaluation by placing agents in contextually relevant environments with tailored questioning. we introduce personascore as a quantitative metric for llm role-playing proficiency, and our evaluation of 10 llms across 200 personas reveals that model size does not necessarily correlate with persona agent capabilities. we find significant performance gaps between sota and less capable models, underscoring the need for targeted research in this domain. strong correlations with human evaluations through spearman and kendall-tau tests validate personagym's effectiveness, establishing a foundation for future persona agent research.