Guess What I am Thinking: A Benchmark for Inner Thought Reasoning of Role-Playing Language Agents (2025)

Abstract: Recent advances in Large Language Model (LLM)-based Role-Playing Language Agents (RPLAs) have attracted broad attention in various applications. While chain-of-thought reasoning has shown importance in many tasks for LLMs, the internal thinking processes of RPLAs remain unexplored. Understanding characters’ inner thoughts is crucial for developing advanced RPLAs. In this paper, we introduce ROLETHINK, a novel benchmark constructed from literature for evaluating character thought generation. We propose the task of inner thought reasoning, constructing 6,058 data entries from 76 books, which includes two sets: the gold set that compares generated thoughts with original character monologues, and the silver set that uses expert-synthesized character analyses as references. To address this challenge, we propose MIRROR, a chain-of-thought approach that generates character thoughts by retrieving memories, predicting character reactions, and synthesizing motivations. Through extensive experiments, we demonstrate the importance of inner thought reasoning for RPLAs, and MIRROR consistently outperforms existing methods.

## introduction recent advances in large language models (llms) have enabled the development of role playing language agents (rplas) @xcite , which are now widely used in applications from character chatbots @xcite to game npcs @xcite . while chain-of-thought reasoning @xcite has shown significant success in various llm tasks @xcite , the inner thought process of rplas ## mirror to generate character inner thoughts, we need to address two key challenges: locating relevant key evidence from the character's long-context memory, and reasoning about possible reactions of related objects from the character's perspective. we propose memory integration and role reasoning with observation reflection (mirror), a chainof-thought approach with three steps: memory recall, theory of mind thinking, and reflection & summarization. we describe each step in detail below. ## results in our experiments, we aim to answer two research questions: rq1) can llms generate high-quality character thought data? rq2) does character thought data improve role-playing performance? model comparison from the model perspective, o1 and gemini2.5-pro show balanced performance across all evaluation metrics. models with strong long-text processing capabilities, such as claude-3.5 and gemini-2.5-pro, perform exceptionally well in long-context settings. reasoningfocused models like deepseek-r1, despite their chain-of-thought capabilities, do not necessarily excel in this task. while these models typically perform well in mathematical and coding tasks where knowledge is stored in model parameters, character thought generation relies more on accurately capturing scenario-relevant memories than complex reasoning. ## conclusion in this paper, we present the first systematic study on generating inner thought processes for rplas. we introduce rolethink, which contains both direct character monologues and expert-analyzed character behaviors. our proposed method, mir-ror, shows significant improvements in character thought generation by combining memory retrieval, role reasoning, and observation reflection. our experiments with various llms demonstrate that generating complex psychological processes remains a challenging task. nonetheless, generating these inner thoughts proves crucial for developing rplas with richer character depth and more compelling behaviors.