Strength Lies in Differences! Improving Strategy Planning for Non-collaborative Dialogues via Diversified User Simulation (2024)

Abstract: We investigate non-collaborative dialogue agents, which are expected to engage in strategic conversations with diverse users, for securing a mutual agreement that leans favorably towards the system’s objectives. This poses two main challenges for existing dialogue agents: 1) The inability to integrate user-specific characteristics into the strategic planning, and 2) The difficulty of training strategic planners that can be generalized to diverse users. To address these challenges, we propose TRIP to enhance the capability in tailored strategic planning, incorporating a user-aware strategic planning module and a population-based training paradigm. Through experiments on benchmark non-collaborative dialogue tasks, we demonstrate the effectiveness of TRIP in catering to diverse users.

## introduction non-collaborative dialogues, such as negotiation @xcite and persuasion @xcite , occur when the agent and user hold conflicting interests @xcite . typically, both parties need to employ various strategies to achieve an agreement favorable to themselves @xcite . as user resistance varies depending on the agent's strategies @xcite , it is imperative for the agent to perform strategic planning tailored to diverse users. relying on a one-sizefits-all strategy can leave the agent vulnerable to others taking advantage due to its lack of adaptability and flexibility @xcite @xcite . recent efforts have resorted to large language models (llms) as dialogue agents to perform noncollaborative tasks @xcite ., * corresponding author. to provide more evidence for the above analysis, we establish an evaluation protocol, which situates diverse user simulators with varying noncollaborative behaviors. we investigate the limitations of current llm-based dialogue agents on strategic planning (cf. section 3 for details). the evaluation results clearly demonstrate that existing agents struggle to tailor their strategies for diverse users, leading to sub-optimal performances. this limitation compromises the practical utility of these agents, both in functioning as a successful agent in conversational ai and in providing social skills training in pedagogy. the key challenges lie in making dialogue agents aware of diverse non-collaborative user behaviors and devising tailored strategies for individual users. to tackle these challenges, we design a simple yet effective method, called trip, to improve llms' capability in tailored strategic planning. trip includes a user-aware strategic planning module and a population-based training paradigm. specifically, the strategic planning module incorporates user-specific characteristics into strategic planning using the theory-of-mind (tom) @xcite . this involves analyzing users' mental states and future possible actions during interactions to understand their interests @xcite . moreover, instead of relying on a solitary user simulator, our populationbased training paradigm promotes the adaptation of the strategic planning module to various users, achieved by training it with more diverse user simulators. each simulator is equipped with extensive sets of non-collaborative strategies and role-playing personas @xcite . as such, trip essentially manipulates the experience of the dialogue agent, enabling it to recognize the importance of tailoring strategies for individual users. our key contributions are concluded below: ## related work our research is closely tied to the strategic planning and training paradigms to address the non-collaborative tasks in the era of llms. we provide a literature review and highlight our differences. ## strategic planning evaluation we introduce a novel evaluation protocol to analyze the limitations of existing llm-based dialogue agents and highlight their inability to handle users exhibiting various non-collaborative behaviors. the overall evaluation process is illustrated in figure 1 . see more details of our evaluation protocol in appendix a. ## trip: tailored strategic planning to enhance llms' tailored strategic planning, we propose an effective method trip, which develops an external planner by modeling user characteristics and training with diverse user simulators. as illustrated in figure 2 , our trip includes a useraware strategic planning module and a populationbased training paradigm. the former aims to explicitly model user characteristics (e.g., mental states and future actions), while the latter incorporates diverse user simulators for training simultaneously. 5 we find that other baselines also have similar issues, as detailed in section 5. ## experiments this sections aims to evaluate the effectiveness of our trip, following the evaluation protocol proposed in section 3.1. we initially report the overall performances of dialogue agents in section 5.1. next, we conduct an in-depth analysis to reveal the tailored strategies of trip in section 5.2. finally, we perform ablation studies in section 5.3 to sort out the performance variation of different user awareness and training population, and find a dominant predictor for the tailored strategic planning. llm-based baselines. we consider llm-based dialogue agents with two types of strategic planning modules, as discussed in section 2. 1) prompt-based planning, including standard, pro-cot @xcite and icl-aif @xcite , which use mixed-initiative prompts, cot, and ai feedback to select next strategies, respectively. 2) external strategy planners, including gdp-mcts @xcite and ppdpp @xcite , which utilize monte carlo tree search and a trainable plug-in for determining nextstep strategies, respectively. note that all baselines fail to model user-specific characteristics explicitly and are trained using one user simulator. implementation details are presented in appendix b. evaluation metrics. we use the same automatic metrics mentioned in section 3.1. furthermore, we conduct human evaluation to assess the practical effectiveness of these dialogue agents. see more details of human evaluation in appendix c.