( CPER ) From Guessing to Asking: An Approach to Resolving Persona Knowledge Gap in LLM s during Multi-Turn Conversations (2025)

Abstract: In multi-turn dialogues, large language models face a critical challenge of ensuring coherence while adapting to user-specific information.. This study introduces the persona knowledge gap, the discrepancy between a model’s internal understanding and the knowledge required for coherent, personalized conversations. While prior research has recognized these gaps, computational methods for their identification and resolution remain underexplored. We propose Conversation Preference Elicitation and Recommendation (CPER), a novel framework that dynamically detects and resolves persona knowledge gaps using intrinsic uncertainty quantification and feedback-driven refinement. CPER consists of three key modules: a Contextual Understanding Module for preference extraction, a Dynamic Feedback Module for measuring uncertainty and refining persona alignment, and a Persona-Driven Response Generation module for adapting responses based on accumulated user context. We evaluate CPER on two real-world datasets: CCPE-M for preferential movie recommendations and ESConv for mental health support. Using A/B testing, human evaluators preferred CPER’s responses 42% more often than baseline models in CCPE-M and 27% more often in ESConv. A qualitative human evaluation confirms that CPER’s responses are preferred for maintaining contextual relevance and coherence, particularly in longer (12+ turn) conversations.

## introduction human communication fundamentally relies on implicit context and incomplete information, requiring iterative dialogue to bridge knowledge gaps and build shared understanding @xcite . this natural process reveals a critical knowledge gap in human-ai interactions, a systemic disparity between the rich contextual 1 code is available at: https://shorturl.at/www6s information needed for coherent, personalized conversations and the limited context available to large language models (llms). while humans naturally resolve ambiguities through iterative questioning, llms generate responses based solely on immediate input, lacking mechanisms to actively seek missing user-specific context @xcite . this gap impedes their ability to retain and adapt to evolving user preferences, emotional states, or domain-specific context across multi-turn conversations @xcite , leading to incoherent or generic interactions over time @xcite . these challenges are particularly pronounced in multi-turn conversational ai systems, which require persistent memory and adaptive reasoning to sustain coherent user engagement. our research addresses two critical questions: how can llms reduce knowledge gaps related to user-specific context in multi-turn conversations? and to what extent does closing these gaps improve the coherence and relevance of conversational ai systems? building on the self-refine framework @xcite , we propose a novel approach to close knowledge gaps through three connected modules (figure 1 ): contextual understanding module: analyzes and quantifies uncertainty in user preferences (eq. 3); dynamic feedback module: measures knowledge disparities between user persona and llm's context understanding (eq. 7), prompting targeted clarification questions; persona-driven response generation: creates contextually aware responses by integrating accumulated user information. this framework enables llms to mimic human conversation patterns by actively resolving ambiguities while maintaining personal context. evaluations on ccpe-m and esconv datasets show marked improvements in both preference tracking and emotional consistency compared to existing approaches. our key contributions are: ## related work advancements in personalized conversational agents stem from improvements in personalization, recommendation systems, and knowledge gap identification in llms. zhang et al. (2024) introduced a memory-based framework for medical assistants, while raj et al. (2024) proposed k-perm, a persona-driven response model integrating external knowledge. however, maintaining consistency across multiple conversation turns remains a challenge. conversational recommendation systems enhance interactions through dynamic context understanding. dao et al. (2023) developed a descriptive graph model for better item recommendations, and feng et al. (2024) introduced a multi-llm framework that detects uncertainty and abstains from answering when needed. meanwhile, cheng et al. (2024) and wu et al. (2024) focused on evolving personas and preference alignment but often rely on static persona modeling. unlike prior work, our framework dynamically detects and resolves knowledge gaps in multi-turn conversations. by actively identifying missing information and asking clarification questions, our system shifts conversational ai from passive response generation to adaptive, context-aware reasoning. for further details, see §a. ## datasets we evaluate our cper framework on two benchmark datasets: ccpe-m and esconv, which address two key aspects of the persona knowledge gap: the ccpe-m (coached conversational preference elicitation for movies) dataset @xcite ) contains 502 dialogues with over 12,000 annotated utterances, capturing user-assistant interactions in a movie recommendation setting. each dialogue is annotated with entity mentions, preference statements, and descriptive justifications, enabling an assessment of how well a system retains evolving user preferences. traditional llms often struggle with knowledge gaps in this dataset, ## cper framework the cper framework dynamically refines responses through iterative feedback and persona adaptation, as formalized in algorithm 1 (fig. 1 ), ensuring coherent, personalized dialogues. ## experimental setup we evaluate our framework against four baselines: zero shot (0s), chain of thought (cot) @xcite , self-fine (sr) @xcite and rationale of thought (rot) @xcite using greedy decoding with a temperature of 0.7. 0s generates responses based solely on user input without leveraging prior context. cot improves coherence by reasoning through intermediate steps. sr iteratively refines outputs using self-feedback, where a single llm generates, evaluates, and refines responses. rot incorporates intermediate rationales to enhance logical consistency and handle multi-turn dialogues effectively. we evaluate cper across 200 multi-turn conversations (5-13 utterances per conversation) in each dataset through two parallel schemes: automated metrics and human assessment. our automated evaluation employs (1) gpt-4o preference scoring, chosen for its strong alignment with human judgment @xcite , and ( 2 ) nubia (kane et al., 2020) a neural metric trained on millions of human annotations capturing semantic relatedness and logical coherence. for human evaluation, seven nlp experts performed blind a/b testing across a subset of 50 multi-turn utterances in each dataset, selecting optimal responses from five system variants per turn based on six criteria: relevance to user input, conversational engagement, contextual appropriateness , natural dialogue flow , persona alignment , and interaction continuity, detailed annotation guidelines are discussed in §c. while we report traditional metrics (e.g., bleu, rouge, bertscore (zhang et al., 2020)) for completeness, they prove inadequate for capturing cper's dynamic knowledge gap management capabilities. the gpt-4o preference scores serve as our primary automated metric due to their correlation with human understanding, while nubia provides granular analysis of semantic-logical consistency across turns. ## experimental results cper consistently surpassed baseline models on both datasets by actively identifying and addressing knowledge gaps through precise questions, as confirmed by human judges and quantitative metrics. performance on ccpe-m: movie preference understanding: cper achieved 53.59% human preference and 60.78% gpt-pref by refining user preferences iteratively. when a user stated, "i enjoy sci-fi films with strong world-building," baseline models suggested generic titles like star wars, while cper asked, "what aspects appeal most-technology or societal dynamics?" this distinction enabled tailored recommendations (e.g., dune vs. black mirror), which traditional metrics like bleu (0.128 vs. baseline's 0.123) failed to capture due to their focus on lexical overlap rather than contextual relevance. our statistical analysis for the human annotation on ccpe-m dataset showed low inter-annotator agreement (fleiss' @xmath0.183) with no significant bias (chi-square @xmath1.565) and significant annotator variation (kruskal-wallis @xmath2.005), indicating subjective differences in preference interpretation. our human evaluation results demonstrate that cper significantly outperforms baseline models in both the ccpe-m and esconv datasets. specifically, human evaluators preferred cper's responses 42% more often than the strongest baseline (sr) in ccpe-m (53.59% vs. 18.30%) and 27% more often in esconv (41.66% vs. 17.86%). these percentage gains are computed using the formula: percentage @xmath3 this study highlights cper's real-world implications for conversational ai systems. by consistently outperforming baseline methods in both human preference and advanced automated metrics, cper demonstrates its capacity to bridge knowledge gaps and maintain personalized, coherent conversations over multiple turns. for practical applications, this means cper can deliver more engaging, emotionally sensitive, and user-centered interactions and personalized recommendations. the findings also reveal that traditional linguistic metrics like bleu, rouge-l and bert-f1 are inadequate for evaluating conversational systems, as they fail to reflect the nuanced personalization and contextual understanding required in real-world dialogues. in contrast, advanced human and semantic evaluations, such as gpt-pref and nubia, provide a better picture of conversational quality. the results underline the potential of cper to adapt dynamically to user preferences and emotional needs, thus creating truly human-like, personalized interactions.