C haracter B ox: Evaluating the Role-Playing Capabilities of LLM s in Text-Based Virtual Worlds (2025)

Abstract: Role-playing is a crucial capability of Large Language Models (LLMs), enabling a wide range of practical applications, including intelligent non-player characters, digital twins, and emotional companions. Evaluating this capability in LLMs is challenging due to the complex dynamics involved in role-playing, such as maintaining character fidelity throughout a storyline and navigating open-ended narratives without a definitive ground truth. Current evaluation methods, which primarily focus on question-answering or conversational snapshots, fall short of adequately capturing the nuanced character traits and behaviors essential for authentic role-playing. In this paper, we propose CharacterBox, which is a simulation sandbox designed to generate situational fine-grained character behavior trajectories. These behavior trajectories enable a more comprehensive and in-depth evaluation of role-playing capabilities. CharacterBox consists of two main components: the character agent and the narrator agent. The character agent, grounded in psychological and behavioral science, exhibits human-like behaviors, while the narrator agent coordinates interactions between character agents and environmental changes. Additionally, we introduce two trajectory-based methods that leverage CharacterBox to enhance LLM performance. To reduce costs and facilitate the adoption of CharacterBox by public communities, we fine-tune two smaller models, CharacterNR and CharacterRM, as substitutes for GPT API calls, and demonstrate their competitive performance compared to advanced GPT APIs. The code is available at https://github.com/Paitesanshi/CharacterBox.

## introduction role-playing is an advanced capability of large language models (llms) that allows them to mimic human-like behavior within the context of * corresponding author: xu.chen@ruc.edu.cn specific roles. this functionality underpins various practical applications, such as intelligent nonplayer characters (npcs) in video games, digital replicas for personal assistants, and emotional support in mental healthcare. while there are comprehensive benchmarks for evaluating the generalpurpose abilities of llms, including language understanding @xcite , conversation @xcite , and reasoning @xcite , the assessment of role-playing capabilities remains an area that is not as thoroughly explored. current evaluation methods, such as static self-reporting questionnaires @xcite and simple dialogue tasks @xcite , fail to capture the full complexity of role-specific behaviors in real-life scenarios. these methods are limited by their static nature and inability to reflect continuous role-playing interactions @xcite . in reality, a character's actions, attitudes, and emotions are dynamic and evolve in response to the surrounding environment and other individuals. the proverb "a man is judged by his deeds, not by his words" applies here: an llm's true roleplaying ability cannot be fully understood from static dialogues or self-reports alone, but rather is demonstrated during interaction with its environment @xcite . in this paper, we present characterbox, a dynamic, multi-agent virtual world tailor-made for eliciting nuanced human-like behaviors from llms in the context of role-playing evaluations. characterbox crafts immersive scenarios tailored to specific roles, incorporating detailed role specifications, contextual backgrounds, and interactions that mirror real-world complexity. llms are assigned roles and interact with both the environment and other characters through dialogue and actions that reflect their role-specific traits. a comparison between previous methods and characterbox is presented in fig 1 . to track the evolving states of both characters reaction analyze influence (c) characterbox scene info interaction result updated environment q: what method would you use to counter the fire attacks? options: a) somersault cloud b) water from guanyin's vase b) water from guanyin's vase (a) question-answering (fiercely): "red boy, release my master or face the consequences!" (provocatively): "monkey king, you dare cause trouble at my fire mountain. today you won't leave alive!" (b) dialogue-based interaction name: sun wukong state: singed but undaunted, planning his next move position: retreated to a nearby cloud, observing the battlefield. and their surroundings, we incorporate a narrator component, typically powered by advanced models like gpt-3.5-turbo. the narrator monitors character actions and environmental changes, generating behavior trajectories used to assess llm roleplaying performance. ## evaluation framework based on text-based interactive virtual world in this section, we present an in-depth exploration of the interactive evaluation framework, characterbox. the characterbox workflow is structured around three pivotal phases: scene crafting, autonomous story play, and evaluation. ## enhancing role-playing ability with trajectories characterbox facilitates the efficient generation of character trajectories across diverse scenes, providing valuable insights into character reactions and behaviors within varied contexts. these trajectories offer a unique opportunity to enhance the role-playing capabilities of language models. to leverage this data, we propose two distinct methods for fine-tuning llms using generated trajectories: guided trajectory fine-tuning. we first assess the role-playing capabilities of llms using characterbox, selecting high-performing models as teachers. the trajectories generated by these models are then used to fine-tune student models, resulting in significant improvements in the latter's ability to simulate complex character interactions. reflective trajectory fine-tuning. in this approach, we explore the self-reflective capabilities of llms. models analyze their own generated trajectories, identifying inconsistencies and areas for improvement in character portrayal. the models then rewrite these trajectories to enhance character consistency and depth. these revised trajectories are subsequently used for fine-tuning, further strengthening the model's capacity to simulate realistic and nuanced interactions. ## building for a self-contained evaluation workflow in characterbox, the narrator agent and evaluation agent can be powered by advanced language models like gpt-4 or individuals familiar with the characters. however, these methods are costly and lack scalability. to address this, we develop char-acternr and characterrm to reduce costs and enhance scalability. characternr. characternr acts as the narrator within characterbox. initially, gpt-3.5 is used to generate narrator trajectory data due to its strong instruction-following abilities. to handle both chinese and english scenes, we select qwen2.5-7b as the base model and fine-tune it using lora @xcite with data generated by gpt-3.5. characterrm. we collect evaluation scores from gpt-4 across 100 scenes, incorporating outputs from nine different llms to ensure diversity. to maintain fairness in scoring, we select chatglm3-6b @xcite as the base model, since it is not among the evaluated models. we then fine-tune it using lora on the collected data, resulting in characterrm. five works are in chinese and five in english, with evaluations conducted in both language settings. each scene includes specific environment and character information, with 2 to 4 characters per scene (see appendix a.2 for further details). ‚ llm. we evaluate the role-playing ability of nine llms varying in model size. each llm embodies all character agents in the scene, generating their respective actions and dialogues based on assigned personalities and backgrounds. for closed-source models, we use gpt-4-turbo-1106-preview as gpt-4 @xcite and gpt-3.5-turbo-1106 as gpt-3.5 @xcite . for open-source models, we evaluate baichuan2-7b/13b @xcite , qwen2.5-7b/14b @xcite , mistral-7b-v0.2 @xcite , llama3-8b @xcite , and phi-3.5-mini @xcite . all open-source llms we evaluated are versions that have undergone instruction tuning. ## conclusion in this paper, we introduce characterbox, a dynamic, text-based virtual environment specifically designed to evaluate the role-playing capabilities of llms. by creating immersive scenarios that reflect the complexities of real-world interactions, charac-terbox captures nuanced human-like behaviors in llms, going beyond static evaluation methods. we demonstrate that fine-tuning smaller models with high-quality behavior trajectories significantly enhances their role-playing abilities. additionally, we develope two fine-tuned components, characternr and characterrm, allowing for a cost-efficient and self-sustained evaluation process without relying on expensive api calls. these contributions establish characterbox as a powerful and self-contained tool for assessing and improving llm role-playing performance across diverse scenarios.