Adaptive Dialog Policy Learning with Hindsight and User Modeling (2020)

Abstract: Reinforcement learning (RL) methods have been widely used for learning dialog policies. Sample efficiency, i.e., the efficiency of learning from limited dialog experience, is particularly important in RL-based dialog policy learning, because interacting with people is costly and low-quality dialog policies produce very poor user experience. In this paper, we develop LHUA (Learning with Hindsight, User modeling, and Adaptation) that, for the first time, enables dialog agents to adaptively learn with hindsight from both simulated and real users. Simulation and hindsight provide the dialog agent with more experience and more (positive) reinforcement respectively. Experimental results suggest that LHUA outperforms competitive baselines from the literature, including its no-simulation, no-adaptation, and no-hindsight counterparts.

## introduction dialog systems have enabled intelligent agents to communicate with people using natural language. for instance, virtual assistants, such as siri, alexa, and cortana, have been increasingly popular in daily life. we are particularly interested in goaloriented dialog systems, where the task is to efficiently and accurately exchange information with people, and the main challenge is on the ubiquitous ambiguity in natural language processing (spoken or text-based). goal-oriented dialog systems typically include components for language understanding, dialog management, and language synthesis, while sometimes the components can be constructed altogether, resulting in end-to-end dialog systems @xcite @xcite @xcite . in this paper, we focus on the problem of policy learning for dialog management. reinforcement learning (rl) algorithms aim at learning action policies from trial-and-error experiences @xcite , and have been used for learning dialog policies @xcite . deep rl methods (e.g. @xcite ) have been developed for dialog policy learning in dialog domains with large state spaces @xcite @xcite . while it is always desirable for rl agents to learn from the experiences of interacting with the real world, such interactions can be expensive, risky, or both in practice. back to the context of dialog systems, despite all the advances in rl (deep or not), dialog policy learning remains a challenge. for instance, interacting with people using natural language is very costly, and low-quality dialog policies produce very poor user experience, which is particularly common in early learning phases. as a result, it is critical to develop sample-efficient rl methods for learning high-quality dialog policies with limited conversational experiences. in this paper, we develop an algorithm called lhua (learning with hindsight, user modeling, and adaptation) for sample-efficient dialog policy learning. lhua, for the first time, enables a dialog agent to simultaneously learn from real, simulated, and hindsight experiences, which identifies the key contribution of this research. simulated experience is generated using learned user models, and hindsight experience (of successful dialog samples) is generated by manipulating dialog segments and goals of the (potentially many) unsuccessful samples. dialog experience from simulation and hindsight respectively provide more dialog samples and more positive feedback for dialog policy learning. to further improve the sample efficiency, we develop a meta-agent for lhua that adaptively a dialog agent interacts with both real and simulated users while learning a dialog policy from this interaction experience. a simulated user is modeled using real dialog samples, and interacting with this simulated user provides the dialog agent with simulated dialog samples. an adaptive coordinator learns from the dialog agent's recent performance to adaptively assign one user (real or simulated) for the dialog agent to interact with. a hindsight manager manipulates both real and simulated dialog samples (of mixed qualities) to "synthesize" successful dialog samples. learns to switch between real and simulated users in the dialog-based interactions, which identifies the second contribution of this research. an overview of lhua is shown in figure 1 . experiments were conducted using a realistic movie-ticket booking platform @xcite . lhua has been compared with state-of-the-art methods @xcite @xcite in dialog policy learning tasks. results suggest that ablations of lhua produce comparable (or better) performances in comparison to competitive baselines in success rate, and lhua as a whole performed the best. ## related work in this section, we summarize three different ways of improving the efficiency of dialog policy learning (namely user modeling, hindsight experience replay, and reward shaping), and qualitatively compare them with our methods. researchers have developed "two-step" algorithms that first build user models through supervised learning with real conversational data, and then learn dialog policies by interacting with the simulated users @xcite . in those methods, user modeling must be conducted offline before the start of dialog policy learning. as a result, the learned policies are potentially biased toward the historical conversational data. toward online methods for dialog policy learning, researchers have developed algorithms for simultaneously constructing models of real users, and learning from the simulated interaction experience with user models @xcite @xcite @xcite @xcite @xcite @xcite . those methods enable agents to simultaneously build and leverage user models in dialog policy learning. however, the problem of learning high-quality user models by itself can be challenging. our algorithms support user modeling, while further enabling agents to adaptively learn from both hindsight and real conversations. in comparison to many other rl applications, goal-oriented dialog systems have very sparse feedback from the "real world" (human users), where one frequently cannot tell dialogs being successful or not until reaching the very end. positive feedback is even rarer, when dialog policies are of poor qualities. hindsight experience replay (her) @xcite methods have been developed to convert unsuccessful trials into successful ones through goal manipulation. the "policy learning with hindsight" idea has been applied to various domains, including dialog @xcite . our methods support the capability of learning from hindsight experience, while further enabling user modeling and learning from simulated users. within the dialog policy learning context, reward shaping is another way of providing the dialog agents with extra feedback, where a dense reward function can be manually designed @xcite , or learned @xcite . researchers also developed efficient exploration strategies to speed up the policy learning process of dialog agents, e.g., @xcite . those methods are orthogonal to ours, and can potentially be combined to further improve the dialog learning efficiency. in comparison to all methods mentioned in this section, lhua is the first that enables dialog policy learning from real, simulated, and hindsight experiences simultaneously, and its performance is further enhanced through a meta-policy for switching between interactions with real and simulated users. ## background in this section, we briefly introduce the two building blocks of this research, namely markov decision process (mdp)-based dialog management, and deep q-network (dqn). ## algorithms in this section, we first introduce learning with hindsight, and user modeling (lhu), and then present lhu with adaptation (lhua), where algorithms lhu and lhua point to the main contribution of this research. lhu, for the first time, enables a dialog agent to learn dialog policies from three dialog sources, namely real users, simulated users, and hindsight dialog experience. more specifically, a real user refers to the human who converses with the dialog agent, and a simulated user refers to a learned user model that captures real users' interactive behaviors with our dialog agent. in this way, a simulated user is used for generating "human-like" dialog experience for speeding up the process of dialog policy learning. the last dialog source of "hindsight dialog experience" is used for creating many successful dialog samples using both successful and unsuccessful dialog samples, where the source samples are from both real and simulated users. different from "simulated users" that generate dialog samples of mixed qualities, hindsight experience produces only successful (though not real) dialog samples, which is particularly useful for dialog policy learning at the early phase due to the very few successful samples. among the three dialog sources, hindsight experience is "always on", and synthesizes dialog samples throughout the learning process. the "real" and "simulated" dialog sources bring in the selection problem: at a particular time, from which source should the agent obtain dialog experience for policy learning? the "adaptation" capability of lhua aims at enabling the dialog agent to learn to, before starting a dialog, select which user (real or simulated) to interact with. ## experiment experiments have been conducted in a dialog simulation platform, called tc-bot @xcite . 1 tc-bot provides a realistic simulation platform for goal-oriented dialog system research. we use its movie-ticket booking domain that consists of 29 slots of two types, where one type is on search constraints (e.g., number of people, and date), and the other is on system-informable properties that are needed for database queries (e.g., critic rating, and start time). the dialog agent has 11 dialog actions, representing the system intent (e.g., confirm question, confirm answer, and thanks). a dialog is considered successful only if movie tickets are booked successfully, and the provided information satisfies all the user's constraints. by the end of a dialog, the agent receives a bonus (positive reward) of 2 * l if successful, or a penalty (negative reward) of -l for failure, where l is the maximum number of turns allowed in each dialog. we set @xmath0. the agent receives a unit cost in each dialog turn to encourage shorter conversations. ## conclusions and future work in this work, we develop an algorithm called lhua (learning with hindsight, user modeling, and adaptation) for sample-efficient dialog policy learning. lhua enables dialog agents to adaptively learn with hindsight from both simulated and real users. simulation and hindsight provide the dialog agent with more experience and more (positive) reinforcements respectively. experimental results suggest that lhua outperforms competitive baselines (including success rate and learning speed) from the literature, including its nosimulation, no-adaptation, and no-hindsight counterparts. this is the first work that enables a dialog agent to adaptively learn from real, simulated, and hindsight experiences all at the same time. in the future, we plan to evaluate our algorithm using other dialog simulation platform, e.g., py-dial @xcite . another direction is to combine other efficient exploration strategies, including learning directed exploration policies with different trade-offs between exploration and exploitation @xcite . we will also focus on generating more synthetic dialog experience of different quality @xcite , to further improve the dialog learning efficiency.