Experience as Source for Anticipation and Planning: Experiential Policy Learning for Target-driven Recommendation Dialogues (2024)

Abstract: Target-driven recommendation dialogues present unique challenges in dialogue management due to the necessity of anticipating user interactions for successful conversations. Current methods face significant limitations: (I) inadequate capabilities for conversation anticipation, (II) computational inefficiencies due to costly simulations, and (III) neglect of valuable past dialogue experiences. To address these limitations, we propose a new framework, Experiential Policy Learning (EPL), for enhancing such dialogues. EPL embodies the principle of Learning From Experience, facilitating anticipation with an experiential scoring function that estimates dialogue state potential using similar past interactions stored in long-term memory. To demonstrate its flexibility, we introduce Tree-structured EPL (T-EPL) as one possible training-free realization with Large Language Models (LLMs) and Monte-Carlo Tree Search (MCTS). T-EPL assesses past dialogue states with LLMs while utilizing MCTS to achieve hierarchical and multi-level reasoning. Extensive experiments on two published datasets demonstrate the superiority and efficacy of T-EPL.

## introduction conversational recommender systems (crss) @xcite @xcite @xcite @xcite aim to provide suitable recommendations through interactive multi-turn dialogues. traditional crss typically employ a reactive approach, identifying user interests during the conversation and mapping these preferences to a set of candidate items. while effective in certain contexts, these passive systems fall short in their ability to proactively guide users towards specific items, a capability crucial for promoting new products and increasing sales revenue @xcite . recently, target-driven crs models @xcite have emerged as a more dynamic alternative. these models leverage various dialogue strategies (e.g., chit-chat, question-answering, etc.) to engage users and foster their interest in target items, aiming to recommend them when the timing and context are most appropriate. despite their effectiveness, existing target-driven crs models exhibit a notable shortcoming: the inability to anticipate future user-system interactions, as they focus solely on individual next-turn evaluations. in target-driven recommendation dialogues, anticipating conversational trajectories toward specific target items is essential for successful interactions. hence, the integration of foresight usersystem interactions has the potential to enhance dialogue planning capabilities @xcite . for instance, as illustrated in figure 1 , by considering potential future interactions, the dialogue strategy (q/a, jaychou) is apparently the optimal choice since it effectively steers the conversation toward the desired target kungfu panda 3. previous works attempt to leverage simulated user interactions generated by llms and employ either reinforcement learning (rl) @xcite or open-loop monte-carlo tree search (mcts) @xcite to enhance their dialogue policies. however, these approaches encounter two main challenges. first, fixed pre-trained policies generated during training may not adapt well to new user interactions during deployment. second, constant access to llms for state evaluations can lead to computational inefficiencies. notably, almost all prior work fails to leverage newly obtained interactions during inference to further enhance their performance. recent advancements have demonstrated the benefits of experience reflection in various domains, including multimodal response generation @xcite , decision-making @xcite , and recommendation @xcite . these approaches often utilize analogous examples or historical trials to deepen understanding of current inputs. for example, in recommender systems, similar users tend to exhibit similar preferences and behaviors @xcite , suggesting that past interactions with such users could provide valuable insights for ongoing conversations. furthermore, these similar interactions can reduce the need for expensive simulations to estimate future interactions. inspired by these observations, we leverage similar past dialogue states from previous sessions to enhance target-driven recommendations, addressing key research questions: (1) how can we integrate experienced interactions into future anticipation? (2) how can we establish adaptable policies for new interactions on the fly? to address these challenges, we introduce experiential policy learning (epl), a novel dialogue policy framework for target-driven crss. our proposed epl approximates a target-driven scoring function for future anticipation by utilizing similar past interactions and their associated values rather than relying on expensive online trials, such as rollout simulations. specifically, given a dialogue state, epl retrieves the k most similar past interactions and their corresponding potential values from a dedicated memory structure. these values are then aggregated to approximate the potential value for the given state. to demonstrate the flexibility of the epl framework, we introduce treestructured epl (t-epl) as one possible realization. t-epl is a training-free implementation that leverages llms to assess dialogue states' potentials while integrating our target-driven scoring function into an mcts algorithm for state value estimation, thus reducing the need for costly llm-based evaluations. extensive evaluations demonstrate the superiority of our method over existing approaches. to sum up, our contributions are threefold: target-driven proactive dialogue systems. recent efforts @xcite @xcite @xcite @xcite have increasingly focused on developing proactive dialogue systems designed to guide conversations toward predefined targets, such as negotiation @xcite and recommendation @xcite . within this target-driven paradigm, establishing effective dialogue policies to achieve specific outcomes has become a crucial area of research. several approaches have been proposed, including brownian-motion dialogue planning @xcite and long-short term strategic balancing @xcite . despite their contributions, these methods lack the ability to incorporate foresight user-system interactions, which are essential for conversation anticipation. to address this limitation, deng et al. (2023a) utilized llm-generated interactions to fine-tune a pre-trained policy with reinforcement learning (rl). however, the static nature of their pre-trained policy limits its adaptability to newly encountered situations. alternatively, @xcite proposed using open-loop monte-carlo tree search (mcts) for future interaction estimation, but this approach suffers from computational inefficiencies due to costly llm-based evaluations. furthermore, the aforementioned methods do not leverage valuable interactions during inference, which could significantly enhance their performance. in contrast to existing methods, we introduce experiential policy learning (epl), a novel dialogue policy method that integrates past interactions into the planning process. by utilizing past user-system interactions, epl anticipates conversational trajectories and plans more effectively. ## methodology first, we describe our proposed experiential policy learning (epl) in sections 4.1. subsequently, we introduce tree-structured epl (t-epl), a trainingfree realization of the epl framework in section 4.2. figure 2 illustrates our proposed t-epl. ## conclusion in this work, we propose experiential policy learning (epl), a novel dialogue policy for target-driven recommendation dialogues. our epl framework leverages similar past interactions to enhance its dialogue planning capability. moreover, we introduce tree-structured epl, a training-free realization of epl, utilizing llms and mcts. throughout extensive experiments and analyses on two published datasets, we demonstrate the effectiveness of our proposed method in terms of both performance and efficiency against state-of-the-art approaches.