Beyond Simple Personas: Evaluating LLM s and Relevance Models for Character-Consistent Dialogue (2025)

Abstract: Dialogue systems often rely on overly simplistic persona representations, limiting their capacity to portray realistic, nuanced characters. In this paper, we explore how well existing persona-grounding methods capture complex personalities using two character-rich domains—Sgt Blackwell (single-character) and Twins (two-character)—described extensively through detailed narratives. We compare early fusion techniques, Retrieval-Augmented Generation (RAG), and relevance-based approaches. Evaluations across entailment, persona alignment, and hallucination metrics reveal distinct trade-offs: Knowledge Graph fusion notably reduces hallucinations and maintains relevance, Persona fusion strongly preserves relevance but has higher hallucination rates, and RAG provides fast, fluent responses. Our findings emphasize the critical role of structured persona grounding in achieving nuanced personality modeling.

## introduction dialogue agents have become central to numerous applications, ranging from virtual assistants to educational tutors and virtual characters in games and simulations. a critical challenge for such agents is the consistent portrayal of nuanced, realistic personalities over extended interactions. although significant progress has been made in personagrounded dialogue systems, current benchmarks frequently rely on overly simplistic persona representations. a prominent example is the per-sonachat dataset @xcite , which offers minimal persona descriptions and has been widely criticized for oversimplification. we discuss its limitations in more detail in section 2. consequently, these models fail to capture the rich, complex nature of interesting human personalities. ensuring persona consistency, where dialogue agents consistently reflect the assigned character traits throughout interactions, remains a persistent challenge @xcite . recent studies, particularly with large language models (llms), show varying levels of persona alignment when conditioned on different prompts or knowledge bases, further emphasizing the difficulty of maintaining persona coherence @xcite . moreover, as dialogue agents increasingly integrate external knowledge sources, the risk of hallucinations, responses unsupported by grounding material, rises significantly, undermining user trust and experience @xcite . retrieval-based approaches that select a response from a pre-authored set, such as the relevance-model based npceditor @xcite , have historically offered robust alternatives to generative models. while these methods reliably provide accurate and consistent replies to familiar questions, they often struggle in openended dialogues requiring creative or adaptive interactions beyond pre-defined responses @xcite . however, @xcite show that some domains, including sgt blackwell, have very high maximum similarity scores, meaning that one can usually find an existing answer that is very similar to a desired human-authored answer, however other domains have much lower scores, indicating a need for novel generation. thus, dialogue system designers face an ongoing trade-off between ensuring accuracy and consistency (as in retrieval-based methods) and enabling flexibility and creativity (as in generative llms). in this paper, we investigate this trade-off by comparing various methods for persona grounding and dialogue alignment across nuanced, richly described character domains. we utilize two carefully developed datasets, each containing extensively detailed persona narratives (>100 lines per character), capturing significantly greater complexity than pre-vious benchmarks. the first domain, "sgt blackwell," @xcite focuses on a singlecharacter persona known for its unique speaking style and complex personality traits. the second domain, "twins," comprises two distinct characters (ada and grace) with different personalities, further increasing the complexity of the dialogue setting @xcite . we specifically evaluate three dialogue strategies: traditional persona fusion (where persona summaries are prepended to the prompt), a novel knowledge graph-based fusion (encoding persona details into structured graph representations), and a retrieval-augmented generation (rag) approach that dynamically integrates relevant context at inference time. we benchmark these approaches against a robust, relevance-focused model (npceditor), thus providing a comparative analysis across generative and retrieval paradigms. our evaluation addresses three key research questions: through this comparative study, we aim to clarify the strengths and limitations of various persona grounding strategies, informing future developments in consistent and believable personalitydriven dialogue systems. ## related work early work in personalized dialogue agents often involved human-authored character utterances (e.g., @xcite ). the sequences were sometimes scripted or constrained in how they were presented to ensure coherence. more general open conversation was facilitated by using informationretrieval techniques to select the most relevant response from pre-written dialogue databases @xcite . however, their conversational flexibility is limited to the pre-scripted responses. use of generative ai allows generation of a broad set of responses tailored to the user's input and specific dialogue context without a large burden on the author to provide detailed and formatted information about the persona, however can be subject to incoherence or deviations from the desired character traits. as noted earlier, persona-based dialogue modeling was significantly influenced by the introduction of personachat @xcite , a benchmark that assigns short persona descriptions (typically 4-5 sentences per character) to dialogue agents. although influential, personachat has been criticized for its oversimplification of real-world personalities. critics have highlighted several limitations: unnatural and forced dialogues, constrained topical scope, repetitive responses, and occasional contradictions within persona descriptions themselves @xcite . subsequent datasets like realpersonachat @xcite , designed using authentic user-generated personas, attempted to improve realism, yet still fall short of capturing nuanced and deeply characterized personas. another important direction employs natural language inference (nli) techniques to explicitly maintain persona consistency. the dialogue nli dataset @xcite introduced nli-based approaches to detect and avoid persona contradictions through entailment checks. subsequent studies @xcite integrated entailment-based criteria into the model training via reinforcement learning, significantly reducing explicit persona contradictions. however, nli approaches can overlook subtler stylistic inconsistencies (e.g., shifts in tone or conversational style), as entailment models primarily detect direct factual contradictions. furthermore, these methods depend heavily on the accuracy of nli classifiers, potentially limiting their reliability in more nuanced interactions @xcite . with the rise of large language models (llms), prompt-based methods have emerged as powerful tools for persona modeling, enabling flexible persona adaptation without extensive persona-specific fine-tuning @xcite . prompt-based approaches allow rapid deployment of varied personas, leveraging the generalized knowledge embedded in pretrained models. however, ensuring long-term consistency remains challenging, as models frequently drift from initial persona prompts over extended interactions @xcite ; frisch and giulianelli 2024). effective persona prompting also demands careful prompt design, with poorly structured prompts potentially leading to inconsistent behavior. recognizing these limitations, recent efforts have explored hybrid approaches, integrating retrieval, entailment, and prompt-based strategies. for example, persona-fact retrieval methods dynamically select additional persona information to enhance grounding and minimize hallucination @xcite . nonetheless, these hybrid methods add complexity and risk introducing errors when retrieved information is marginally relevant or inconsistent with existing persona profiles. in this paper, we address these gaps by presenting a novel knowledge graph fusion approach, explicitly structuring detailed persona information into knowledge graphs to ground dialogue generation effectively. unlike prior simplified benchmarks, our work uses extensively detailed character descriptions (over 100 sentences per persona), allowing a richer and more nuanced representation of personalities. we systematically compare this approach against traditional persona fusion, retrieval-augmented generation (rag), and relevance-based methods across multiple evaluation metrics, explicitly targeting persona alignment, hallucination reduction, and relevance-creativity trade-offs. thus, our approach advances personagrounded dialogue towards capturing the complexity and subtlety inherent in realistic human personalities. ## dataset to evaluate persona consistency and grounding in dialogue models, we use two character-rich domains featuring deeply developed virtual personas: sgt blackwell and the twins (ada and grace). these characters are not only highly nuanced in terms of background, speech style, and domain knowledge, but also serve as long-standing virtual agents in interactive systems designed for public engagement. each was on display in museums, where visitors interacted with them, asking questions, many of which related to their personas, backstories, and individual characteristics. unlike prior benchmarks based on shallow, synthetic persona snippets, these datasets provide over 100 lines of detailed character responses per persona, allowing for a richer and more realistic evaluation of dialogue grounding techniques. ## models to systematically evaluate persona grounding and consistency, we compare four distinct models representing various dialogue strategies: two early fusion methods (persona summary and knowledge graph fusion), a retrieval-augmented generative model (rag), and a purely retrieval-based relevance classifier (npceditor). all models except npceditor are implemented using gpt-4o as the underlying language model. below, we describe each model's methodology and highlight the novel aspects of our proposed knowledge graph fusion approach. ## evaluation methods to comprehensively assess the effectiveness of each persona-grounding strategy, we employ three evaluation metrics that capture complementary aspects of dialogue quality: persona alignment, entailment, and relevance. ## results we present the comparative results for the evaluated models across the two persona-rich domains (sgt blackwell and twins). results are organized into two tables: ## qualitative analysis we analyze how different models respond to the same prompts to better understand their behaviors. we select representative examples from both the sgt blackwell and twins domains, highlighting differences in persona fidelity, topical relevance, and factual grounding. sgt blackwell domain. prompt: "do you like thanksgiving?" analysis: kg-gpt and npceditor respond appropriately by deflecting a personal question outside the character's scope. personagpt and rag hallucinate opinions, breaking persona fidelity with unsupported emotional depth. prompt: "what is your favorite music?" analysis: npceditor provides the gold response, rich with character-specific humor and cultural references. kg-gpt and rag produce relevant but less expressive variants. personagpt breaks immersion with a generic disclaimer. this highlights the value of well-authored responses in capturing stylistic depth. twins domain. prompt: "who is norman cahner?" grace: he was also a supporter of the museum and is the namesake behind the cahners computerplace. grace: that's right! cahners computerplace is named in his honor. it's a fantastic section of the museum that offers exhibits and activities all about computers, robots, and communications. analysis: rag and kg-gpt provide fluent, persona-consistent answers grounded in relevant museum knowledge. personagpt fails to respond meaningfully, despite the answer being part of the scripted domain. npceditor includes accurate factual content, but its response focuses more on cahners computer place than cahners himself. prompt: "who is older?" grace: how can we help you? ada's right! our "birthdate" as virtual guides is pretty close, so neither of us can truly claim to be the older one. it's all part of our unique digital family story. grace: my program finished twenty minutes prior. ada: you were a test run. but... interaction with our knowledge base does replicate conversation with nineteen year olds. grace: correction. with one nineteen year old, and one nineteen and twenty minutes year old. analysis: npceditor provides the best response, drawing directly from a hand-authored exchange in the corpus specifically crafted for this question. rag also accesses the original joke and uses it effectively, producing a fluent and engaging response, while slightly more restrained than npceditor. per-sonagpt falls short by offering a generic, overly diplomatic response. it fails to recognize or reproduce the crafted joke from the underlying data, and the second turn includes an out-of-context prompt ("how can we help you?"), reducing overall coherence. kg-gpt is the weakest, entirely missing the joke and failing to provide a meaningful answer. this highlights its limitation in parsing nuanced humor into structured knowledge graph triplets. ## conclusion this paper introduced kggpt, a novel knowledge graph-based fusion approach for grounding dialogue agents with richly detailed personas. we evaluated on two rich character domains compared to multiple baseline techniques involving retrieval and summarization. results show that kggpt achieves reasonable persona alignment, high topical relevance, and significantly reduces hallucinations compared to traditional small persona and retrieval-augmented generation (rag) methods. by explicitly structuring persona knowledge into triplets, kggpt offers a robust middle ground between the rigidity of retrieval-based models and the factual unreliability of unconstrained generation. our results show that structured persona grounding, as realized in kggpt, not only improves dialogue consistency but also enables greater adaptability to novel prompts while maintaining character fidelity. we hope that this work motivates further exploration of structured grounding techniques for building more believable, trustworthy, and personalitydriven dialogue systems.