PRINCIPLES : Synthetic Strategy Memory for Proactive Dialogue Agents (2025)

Abstract: Dialogue agents based on large language models (LLMs) have shown promising performance in proactive dialogue, which requires effective strategy planning. However, existing approaches to strategy planning for proactive dialogue face several limitations: limited strategy coverage, preference bias in planning, and reliance on costly additional training. To address these, we propose PRINCIPLES: a synthetic strategy memory for proactive dialogue agents. PRINCIPLES is derived through offline self-play simulations and serves as reusable knowledge that guides strategy planning during inference, eliminating the need for additional training and data annotation. We evaluate PRINCIPLES in both emotional support and persuasion domains, demonstrating consistent improvements over strong baselines. Furthermore, PRINCIPLES maintains its robustness across extended and more diverse evaluation settings. See our project page at https://huggingface.co/spaces/kimnamssya/Principles.

## introduction recent advances in large language models (llms) have substantially improved the performance of dialogue agents @xcite @xcite . in this context, one primary research focus is improving dialogue agents' ability to plan strategies for achieving goals in proactive dialogue @xcite , such as emotional support @xcite and persuasion @xcite . planners based on small lms via supervised finetuning and reinforcement learning. these methods enable dialogue agents to exhibit more proactive, goal-directed behavior and achieve strong performance across dialogue scenarios. despite the success, existing methods have several limitations. first, most of them rely on a set of pre-defined strategies that are relatively small in size. such limited coverage of strategies constrains the agents' adaptability to diverse real-world scenarios @xcite . second, recent work has highlighted that llms exhibit preference bias when selecting strategies @xcite , which hinders their ability to identify optimal strategies. third, many approaches focus on training external planners, which requires specifically curated datasets and may hinder generalization to unseen situations, thereby falling short of providing diverse and unbiased strategy planning. as illustrated in figure 1 , these limitations underscore the need for an alternative that expands strategy coverage and mitigates bias, while not relying on costly training. to tackle these, we introduce principles: a synthetic strategy memory for proactive dialogue agents, derived through offline self-play simulations. inspired by how humans learn from both successful and failed experiences @xcite , we leverage these two types of experiences to derive fundamental principles that enable effective strategy planning. specifically, when the agent's strategy leads to success (e.g., resolving the user's core issue), we derive princi-ples by analyzing the success factors. in contrast, when the strategy results in failure (e.g., worsening the user's distress), we initiate a revision process where we backtrack to the starting point of the failure. from there, we iteratively revise the strategy and re-simulate the interaction until success. then, we derive principles that capture what determines both successes and failures. this process allows us to accumulate strategies from both positive and negative experiences, structured as follows: when [situation], you should [successful strategy], rather than [failed strategies] 1 , because @xcite . principles has advantages in three key aspects: (i) coverage: although derived from a limited set of simulations, our strategy space effectively covers diverse dialogue scenarios, addressing the limitations of pre-defined strategy sets in prior work. (ii) bias: principles semantically captures contrasts between effective and ineffective strategies (i.e., "you should . . . rather than . . . "), which explicitly helps avoid dialogue agents' harmful bias toward improper strategies. (iii) training: by uncovering the hidden parametric knowledge of llms in a non-parametric form, principles enhances dialogue agents without additional training or reliance on vast amounts of human conversation data. after demonstrating the effectiveness of prin-ciples over common baselines in two commonly used datasets of proactive dialogue (e.g., esconv for emotional support and p4g for persuasion), 1 the rather than clause is included only when princi-ples is extracted from a revision process. ## preliminary: self-play simulation following @xcite , we adopt self-play simulation to model strategic decision-making in proactive dialogue. in this setup, an agent engages in multi-turn conversations with a user simulator, adaptively selecting strategies at each turn and responding accordingly to accomplish a defined goal. at each turn t, the agent observes the current state s @xmath0, u 1 , a 2 , . . . , a t-1 , u t-1 }, which consists of the dialogue history up to that point, i.e., all utterances from turns 1 to t-1. given s t , the agent selects a high-level strategy σ @xmath1: subsequently, the user simulator generates a response based on s t and a t : case 1: successful strategy planning case 2: failed strategy planning ... principle derivation strategy planning strategy planning strategy revision re-simulation via backtracking principle retrieval reinterpretation yes, the patient's issue has been solved. thank you, i'm feeling more hopeful. when the patient acknowledges the need ..., you should guide to develop a structured plan..., rather than simply encouraging them ..., because a structured approach provides ... when the patient describes daily struggles ..., you should guide a practice of deep breathing..., rather than simply encouraging them ..., because a structured approach provides ... i suggest we start with a simple deep breathing exercise. principles-driven strategy planning principles construction self-play on train set principles ... ... when the patient describes experiencing ..., you should validate their emotions and work ..., because offering both emotional affirmation ... when the patient acknowledges the need ..., you should guide to develop a structured ..., rather than simply encouraging them ..., because a structured approach provides ... self-play on test set ## principles inspired by louie et al. ( 2024 ), which elicits qualitative feedback from a domain expert, we propose principles: a synthetic strategy memory derived from offline self-play simulations. we explain how principles is constructed and how it can be applied in real-time conversation (figure 2 ). ## human evaluation evaluating response quality. given the nature of self-play simulations, we evaluate responses generated from the same context (i.e., first user utterance) to ensure a fair comparison across baselines, when the patient expresses feelings of depression and loneliness, you should validate their emotions and collaborate on exploring sources of support and potential coping strategies because this combination of emotional acknowledgment and practical exploration can help ... ## in-depth analysis comparison with strong baseline. to further validate our approach, we compare it with dpdp @xcite , a strong baseline using monte carlo tree search (mcts)-based strategy selection at each turn. as shown in ## related work recent studies have increasingly highlighted the importance of strategy planning in proactive dialogue, such as emotional support @xcite and persuasion @xcite , where an agent should take the initiative to achieve a specific goal. to this end, many approaches rely on external planners to select the most suitable choice from a predefined strategy set @xcite . average turn success rate (%) 80.0 90.0 7.0 70.0 6.0 60.0 5.0 4.0 1 3 5 7 9 number of retrieved principles success rate average turn 66.9 88.3 95.0 89.2 87.5 90.0 73.9 73.9 68.5 80.0 esconv p4g figure 10: correlation between a number of retrieved principles and success rate. formulate strategy planning as a search problem, using monte carlo tree search (mcts) to identify optimal strategies. another line of research focuses on open-ended strategy generation. for instance, fu et al. (2023) prompts llms to elicit improved strategies via iterative ai feedback, while zhang et al. (2024b) use llms as a source of expert knowledge. he et al. (2025a) attempts to eliminate the dependency on simulation environments by discovering latent policies from dialogue (louie et al., 2024 ). yet, they have several limitations, including limited coverage, preference bias, and costly training. this motivates the need of alternative approaches, which we introduce in our work. ## conclusion in this paper, we introduce principles, a synthetic strategy memory for proactive dialogue agents, derived through offline self-play simulations. through extensive experiments across both standard and extended datasets, we show that our approach effectively expands strategy coverage and mitigates preference bias, leading to more balanced strategy planning. furthermore, our approach achieves promising performance without additional training by explicitly uncovering the hidden parametric knowledge into a structured, nonparametric form. we expect our novel approaches to serve as a new foundation for future research efforts towards proactive dialogue agents.