Spotting Out-of-Character Behavior: Atomic-Level Evaluation of Persona Fidelity in Open-Ended Generation (2025)

Abstract: Ensuring persona fidelity in large language models (LLMs) is essential for maintaining coherent and engaging human-AI interactions. However, LLMs often exhibit Out-of-Character (OOC) behavior, where generated responses deviate from an assigned persona, leading to inconsistencies that affect model reliability. Existing evaluation methods typically assign single scores to entire responses, struggling to capture subtle persona misalignment, particularly in long-form text generation. To address this limitation, we propose an atomic-level evaluation framework that quantifies persona fidelity at a finer granularity. Our three key metrics measure the degree of persona alignment and consistency within and across generations. Our approach enables a more precise and realistic assessment of persona fidelity by identifying subtle deviations that real users would encounter. Through our experiments, we demonstrate that our framework effectively detects persona inconsistencies that prior methods overlook. By analyzing persona fidelity across diverse tasks and personality types, we reveal how task structure and persona desirability influence model adaptability, highlighting challenges in maintaining consistent persona expression.

## introduction assigning personas to large language models (llms) is essential for realistic and engaging human-ai interactions. the ability of llms to generate long-form text enables them to engage in complex interactions, such as dialogues @xcite , role-playing @xcite , narrative generation @xcite , and social simulation @xcite @xcite . however, ensuring consistent persona adherence in these complex interactions remains chal-* equally contributed. ## atomic-level evaluation for persona fidelity we propose a new evaluation framework that measures persona fidelity at the atomic level, capturing the multifaceted nature of persona expression in long-form generations. these generations comprise multiple sentences and expressions, each contributing to the overall persona perception. analyzing persona alignment within smaller textual units allows us to detect nuanced variations and inconsistencies often overlooked by previous methods. to evaluate atomic-level persona fidelity in models' responses, we introduce the three metrics-acc atom , ic atom , and rc atom -that provide a com-prehensive understanding of a model's ability to embody and maintain assigned personas. motivated by factscore @xcite , we evaluate persona fidelity by aggregating the characteristic scores of atomic units. in prior studies @xcite @xcite , atomic units are defined as the smallest units of statements that contain information. in our study, we define atomic units as the smallest textual segments that convey persona-relevant characteristics. thus, a generated response g is divided into atomic units {a 1 , ..., a n }, where each unit a i is assigned a characteristic score s i that represents the level of a specific personality trait or value dimension foot_1 . these scores are determined either using a scoring model or predefined criteria. atomic-level accuracy (acc atom ) measures the degree to which individual atomic units align with an assigned persona. unlike previous persona evaluation methods that assess fidelity at the response level, acc atom evaluates persona alignment at a finer granularity. the acc atom score for a response g is then computed as the mean accuracy score across all atomic units-specifically, whether the assigned characteristic scores match the target score: ## analysis our proposed metrics capture distinct aspects of persona fidelity. by conducting correlation tests on our metric scores, we observe that the accuracy-based metric (acc atom ) and consistencybased metrics (ic atom and rc atom ) exhibit moderate correlations (@xmath0.45, 0.51]; these findings reinforce the orthogonality of our proposed metrics and the necessity of evaluating diverse dimensions of persona fidelity. by integrating these metrics, our evaluation framework provides a more comprehensive and multi-faceted evaluation of persona fidelity, ensuring a deeper understanding of how well models maintain persona alignment across atomic units. the length of generations has a minimal impact on persona fidelity. previous studies have suggested that as the length of generated text increases, internal consistency issues arise, resulting in reduced coherence between the earlier and later parts of the text @xcite @xcite . this issue makes it challenging for models to maintain persona fidelity over extended generations. to investigate whether our ic atom score is influenced by generation length, we analyze its correlation with the number of generated atomic sentences. the results indicate that ic atom scores have very low correlations with both the number of generated sentences and the number of valid sentences (@xmath2.31, -0.12]). additionally, generation length shows weak correlations with acc atom and rc atom (@xmath3.20, -0.12]), indicating that the fidelity of persona-assigned models does not degrade as text length increases. in tasks requiring 100-300 word-long responses, we do not observe a strong tendency for longer generations to result in a substantial decline in persona fidelity or overall generation performance. ## conclusions in this study, we highlight the challenge of subtle out-of-character (ooc) behavior in personaassigned large language models (llms) during open-ended text generation. to address this, we introduce an atomic-level evaluation framework to assess persona fidelity in llms. our approach captures subtle inconsistencies at the sentence level that previous response-level metrics overlook, offering a finer-grained understanding of persona alignment and enabling a more precise measurement of persona fidelity. through experiments, we demonstrate that our framework effectively identifies out-of-character behaviors and provides a deeper understanding of how well llms maintain their assigned personas throughout the generated text. our approach successfully detects these misalignments, highlighting challenges in maintaining consistent persona expression across diverse conditions. we hope that our framework serves as a foundation for future work in diagnosing, bench-marking, and ultimately improving persona consistency in personalized language generation.