Efficient Dialogue Complementary Policy Learning via Deep Q -network Policy and Episodic Memory Policy (2021)

Abstract: Deep reinforcement learning has shown great potential in training dialogue policies. However, its favorable performance comes at the cost of many rounds of interaction. Most of the existing dialogue policy methods rely on a single learning system, while the human brain has two specialized learning and memory systems, supporting to find good solutions without requiring copious examples. Inspired by the human brain, this paper proposes a novel complementary policy learning (CPL) framework, which exploits the complementary advantages of the episodic memory (EM) policy and the deep Q-network (DQN) policy to achieve fast and effective dialogue policy learning. In order to coordinate between the two policies, we proposed a confidence controller to control the complementary time according to their relative efficacy at different stages. Furthermore, memory connectivity and time pruning are proposed to guarantee the flexible and adaptive generalization of the EM policy in dialog tasks. Experimental results on three dialogue datasets show that our method significantly outperforms existing methods relying on a single learning system.

## introduction dialogue policy, one of the most critical modules of task-oriented dialogue systems, aims to determine system responses based on current states @xcite . one of the earliest methods is the rule-based policy @xcite . although this method often has acceptable performance, handcrafting rules are expensive and non-extensible. recently, deep reinforcement learning (rl) has become a mainstream method for training dialogue policies @xcite @xcite . since deep rl-based methods are learning in an online fashion, a large amount of interaction with * corresponding author real users is required, which is generally infeasible in practical application @xcite @xcite . intuitively, the reward-based learning mechanism in deep rl (drl) coincides with the dopamine-centered regulation in the human brain @xcite . human brains have two differentially specialized learning and memory systems for collaboration, allowing them to find good solutions without requiring copious examples @xcite @xcite . however, most of the drlbased dialogue policies @xcite @xcite @xcite rely on a single learning system, which neglects the human brain's memory structure. consequently, we imitate the human brain to construct an efficient complementary policy learning (cpl) model with learning and memory systems for cooperation. inspired by cognitive neuroscience studies @xcite @xcite , some researches evidence that episodic memory (em) plays a vital role in decision tasks. thus, they incorporate the em into rl to accelerate learning @xcite @xcite . despite the effectiveness of these methods on video game tasks, there is little research validating the practical usage of em in dialogue tasks. in this paper, we investigate the roles of the em policy and the dqn policy (a classic representative of the drl-based dialogue policies) in dialogue policy tasks. we observed that the em policy is similar to the human brain's memory system, which efficiently leans with little data and bridges interdependency between actions and results from past experience. it is of limited usefulness in novel situations, since it generalizes poorly. the dqn policy is analogous to the human brain's learning system. it effectively extracts and generalizes potential in-formation from a large amount of experience to drive decisions and calibrate strategies stored in the em. its good generalization comes at a cost of learning inefficiency and the demand for massive data. these two policies complement each other. nevertheless, directly combining the dqn policy with the vanilla em policy is difficult to maintain effectiveness consistently in the dialog policy task. thus we have the following considerations: for question (1), we propose a confidence controller that allows the two policies to form a seamless hybridization by controlling the complementary timing according to their relative efficacy at different stages. once the cpl is enabled, the em policy provides diversified guidances for the dqn policy: an extra memory objective (emo), an example memory action (ema), and an extra intrinsic reward (eir). for question (2), we define memory connectivity that allows the flexibility and generalization of the em policy by associating past familiar memories. then, time pruning prunes the outdated memories. in summary, our main contributions are two-fold: (1) we present a novel cpl framework, which gets rid of collecting any demonstrations and does not rely on any experts. preferably, it exploits the complementary superiorities of the em policy and the dqn policy through the confidence controller. to the best of our knowledge, this is the first work to learn a dialogue policy, which integrates the learning and memory systems seamlessly and avoids being stuck on a single system. (2) we experimentally demonstrate that the effectiveness of our framework, and em can be a crucial building block of effective dialogue policy learning. our model is the first step in that direction, as far as we know. ## related work the research on the learning efficiency of dialogue policies is not new. @xcite showed that pre-filling the replay buffer with few successful dialogue experiences at the beginning can accelerate learning. prioritized experience replay improves the sample efficiency by increasing the replay probability of experiences with higher temporal difference errors @xcite . @xcite proposed a world model to simulate users and integrated planning into policy learning. a lot of progress has been made in improving the effectiveness of dialogue policies by combining supervised learning (sl) @xcite . @xcite and @xcite proposed to use sl to initialize the policy network and then fine-tune it within the rl process. chen et al. (2017a,b), liu et al. (2018), and zhao et al. (2021) incorporated a teacher to guide policy learning. nevertheless, these methods require extra effort to hire or design teacher models. @xcite proposed an efficient policy learning from demonstrations. however, these methods require the collection of human demonstrations, and their performance depends on the quality of the demonstrations. parallelly, another solution is to increase the density of meaningful rewards @xcite @xcite . episodic memory has been used outside of dialogue research to improve data efficiency @xcite . @xcite proposed table-based model-free episodic control to learn past good experiences in a one-shot learning fashion. @xcite proposed neural episodic control, which uses differentiable neural dictionaries to store and lookup beneficial memories for decision. however, these table-based methods lack good generalization capabilities. @xcite proposes a em integrated into rl agent. but its computing time increases with the history length. based on this, @xcite proposed episodic memory deep q-network in the video game domains with high-dimensional. however, these researches focused on the video game fields, how effectively use the em in the dialogue domain, and whether it is feasible are less explored. ## proposed framework the cpl framework is described in figure 1 , which mainly includes three modules: (1) the episodic memory policy quickly latches familiar experiences from the past to provide auxiliaries for the dqn policy. it includes two operations. writing effectively retains memories while minimizing the retention of obsoleting memories. lookup with memory connectivity and time pruning selectively associates relevant memories while casting aside irrelevant or obsolete memories; (2) the dqn policy effectively extracts and generalizes potential information from a large amount of experience to drive decisions and calibrate strategies stored in the em; (3) the confidence controller choose an appropriate time to perform complementary policy learning according to their relative efficacy at different stages. ## performance evaluation we conduct sufficient experiments on three public task-oriented datasets in both simulation and human evaluation: movie-ticket booking, restaurant reservation, and taxi ordering foot_3 . ## conclusion in this paper, we propose a novel complementary policy learning (cpl) framework that realized dialogue policy learning in a more effective and faster manner through direct use of its own experience without any extra cost. this framework exploits the complementary advantages of the em policy and the dqn policy. additionally, we proposed a confidence controller to coordinate between the two policies according to their relative efficacy at different stages. further proposed memory connectivity and time pruning ensure the flexible and adaptive generalization of the em policy in dialogue tasks. the results show that the cpl significantly outperforms baselines in three domains, and an episodic memory component is a crucial building block of effective dialogue policy learning. to the best of our knowledge, this is the first work to learn a dialogue policy, which integrates the learning and memory systems seamlessly and avoids being stuck on a single system. in the future, we plan to expand our method to multi-domain tasks, e.g., multiwoz @xcite and evaluating it using other dialogue platforms, e.g., pydial @xcite , convlab @xcite .