Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents (2025)

Abstract: Role-Playing Agent (RPA) is an increasingly popular type of LLM Agent that simulates human-like behaviors in a variety of tasks. However, evaluating RPAs is challenging due to diverse task requirements and agent designs.This paper proposes an evidence-based, actionable, and generalizable evaluation design guideline for LLM-based RPA by systematically reviewing 1,676 papers published between Jan. 2021 and Dec. 2024.Our analysis identifies six agent attributes, seven task attributes, and seven evaluation metrics from existing literature.Based on these findings, we present an RPA evaluation design guideline to help researchers develop more systematic and consistent evaluation methods.

## introduction llms have yielded human-like performance in various cognitive tasks (e.g., memorization @xcite , reasoning @xcite , and planning @xcite ). these emergent capabilities have fueled growing research interest on role-playing agent (rpa) @xcite : rpas are digital intelligent agent systems powered by llms, where users provide human-like agent attributes (e.g., personas) and task attributes (e.g., task descriptions) as input, and prompt the llm to generate human-like behaviors and the reasoning process. the potential of rpas is promising and far-reaching, as illustrated by the early results of the massive interdisciplinary studies in social science @xcite @xcite , network science (chen figure 1 : rpa evaluation design guideline. to illustrate how to use it in practice, we pretended we were selecting the evaluation metrics for the "stanford agent village" @xcite given agent attributes (yellow) and task attributes (pink). the original authors' selection of evaluation metrics (purple and blue) perfectly aligns with our rpa design guideline, which echoes their work's robustness. more details in sec 5.1 and a bad example in sec 5.2. @xcite , psychology @xcite and juridical science @xcite . despite growing interest in rpas, a fundamental question remains: how can we systematically and consistently evaluate an rpa? how should we select the evaluation metrics, so that the evaluation results can be comparable or generalizable from one task to another task? addressing these challenges is difficult @xcite @xcite . due to the vast diversity of tasks (e.g., simulating an individual's online browser behavior @xcite or simulating a hospital @xcite ), and the high flexibility in rpa design (e.g., an agent persona can be one sentence or 2-hours of interview log @xcite ). another challenge is the inconsistent and often arbitrary selection of evaluation methods and metrics for rpas, raising concerns about the validity and reliability of evaluation results @xcite . as a result, the research community finds it difficult to compare the performance across multiple rpas in similar tasks reliably and systematically. to address this gap, we propose an evidencebased, actionable, and generalizable design guideline for evaluating llm-based rpas. we conducted a systematic literature review of 1, 676 papers on the llm agent topic and identified 122 papers describing its evaluation details. through expert coding, we found that agent attribute design interacts with task characteristics (e.g., simulating an individual or simulating a society requires a diverse set of agent attributes). furthermore, we synthesized common patterns in how prior research successfully (or unsuccessfully) designed their evaluation metrics to correspond to the rpa's agent attributes and task attributes. building on these insights, we propose an rpa evaluation design guideline (fig. 1 ) and illustrate its generalizability through two case studies. ## method we conduct a systematic literature review to address our research question. following prior method @xcite , we aim to identify relevant research papers on rpas and provide a comprehensive summary of the literature. we selected four widely used academic databases: google scholar, acm digital library, ieee xplore, and acl anthology. these databases encompass a broad spectrum of research across ai, humancomputer interaction, and computational linguistics. given the rapid advancements in llm research, we included both peer-reviewed and preprint studies (e.g., from arxiv) to capture the latest developments. below, we detail our paper selection and annotation process. ## survey findings building on the annotated data, we systematically categorized agent attributes, task attributes, and evaluation metrics. we then present a structured rpa evaluation design guideline, outlining how to select appropriate evaluation metrics based on agent and task attributes. ## case study: how to use rpa design guideline to select evaluation metrics we present two case studies to illustrate how our evaluation guidelines can be applied in practice. these examples are not intended to demonstrate superiority but to show the feasibility of aligning evaluation metrics with agent and task attributes, and how such alignment is reflected in existing studies. by adopting the perspective of the original authors, we compare the evaluation outcomes resulting from adhering to or deviating from the rpa evaluation guidelines. ## relationships between agent attributes and downstream tasks both agent attributes and downstream task attributes play a crucial role in selecting appropriate rpa evaluation metrics. researchers predefine these factors when designing and evaluating rpas, yet their interrelation remains an open question. in this section, we analyze how agent attributes correspond to different downstream tasks, uncovering several recurring patterns (fig. 6 ). demographic information and psychological traits are fundamental across all downstream tasks. whether in decision-making, opinion dynamics, or simulated environments, these attributes consistently shape rpa design. as shown in fig. 6 , they are the most frequently incorporated factors, underscoring their central role in modeling agent behavior across diverse applications. for tasks where simulation itself is the primary objective, such as simulated individuals and simulated society, the selection of agent attributes becomes broader. in addition to demographic and psychological factors, these tasks frequently incorporate skills, expertise, and social relationships, reflecting the need for richer agent representations to capture complex social and individual interactions. by contrast, tasks that use simulation as a means to study specific research fields tend to prioritize certain agent attributes. for instance, in opinion dynamics, beliefs and values play a distinctive role, as they directly influence how agents interact and form opinions. similarly, tasks related to educational training and writing exhibit a different pat-tern, emphasizing skills and expertise over broad demographic or psychological considerations. in contrast, attributes such as activity history and social relationships receive significantly less emphasis across tasks. this raises a question: is their impact inherently limited, or are they simply underexplored in current rpa applications? overall, these findings highlight the nuanced interplay between agent attributes and downstream tasks. while demographic information and psychological traits are universally relevant, attributes like beliefs and values gain importance in specific contexts. at the same time, the relative absence of activity history and social relationships in current evaluations presents an open research question, particularly in scenarios requiring long-term modeling and complex social interactions. ## conclusion rpa evaluation lacks consistency due to varying tasks, domains, and agent attributes. our systematic review of 1, 676 papers reveals that taskspecific requirements shape agent attributes, while both task characteristics and agent design influence evaluation metrics. by identifying these interdependencies, we propose guidelines to enhance rpa assessment reliability, contributing to a more structured and systematic evaluation framework.