Dynamic Dialogue Policy for Continual Reinforcement Learning (2022)

Abstract: Continual learning is one of the key components of human learning and a necessary requirement of artificial intelligence. As dialogue can potentially span infinitely many topics and tasks, a task-oriented dialogue system must have the capability to continually learn, dynamically adapting to new challenges while preserving the knowledge it already acquired. Despite the importance, continual reinforcement learning of the dialogue policy has remained largely unaddressed. The lack of a framework with training protocols, baseline models and suitable metrics, has so far hindered research in this direction. In this work we fill precisely this gap, enabling research in dialogue policy optimisation to go from static to dynamic learning. We provide a continual learning algorithm, baseline architectures and metrics for assessing continual learning models. Moreover, we propose the dynamic dialogue policy transformer (DDPT), a novel dynamic architecture that can integrate new knowledge seamlessly, is capable of handling large state spaces and obtains significant zero-shot performance when being exposed to unseen domains, without any growth in network parameter size. We validate the strengths of DDPT in simulation with two user simulators as well as with humans.

## introduction task-oriented dialogue systems are characterised by an underlying task or a goal that needs to be achieved during the conversation, such as managing a schedule or finding and booking a restaurant. modular dialogue systems have a tracking component that maintains information about the dialogue in a belief state, and a planning component that models the underlying policy, i.e., the selection of actions @xcite @xcite . the spectrum of what a task-oriented dialogue system can understand and talk about is defined by an ontology. the ontology defines domains such as restaurants or hotels, slots within a domain such as the area or price, and values that a slot can take, such as the area being west and the price being expensive. as dialogue systems become more popular and powerful, they should not be restricted by a static ontology. instead, they should be dynamic and grow as the ontology grows, allowing them to comprehend new information and talk about new topics -just like humans do. in the literature, this is referred to as continual learning @xcite @xcite . a learner is typically exposed to a sequence of tasks that have to be learned in a sequential order. when faced with a new task, the learner should leverage its past knowledge (forward transfer) and be flexible enough to rapidly learn how to solve the new task (maintain plasticity). on the other hand, we must ensure that the learner does not forget how to solve previous tasks while learning the new one (prevent catastrophic forgetting). rather, a learner should actually improve its behaviour on previous tasks after learning a new task, if possible (backward transfer). despite progress in continual learning @xcite @xcite @xcite , there is -to the best of our knowledge -no work that addresses continual reinforcement learning (continual rl) of the dialogue policy, even though the policy constitutes a key component of dialogue systems. research in this direction is hindered by the lack of a framework that provides suitable models, evaluation metrics and training protocols. in modular task-oriented dialogue systems the input to the dialogue policy can be modelled in many different ways @xcite @xcite @xcite ). an appropriate choice of state representation is key to the success of any form of rl @xcite . in continual rl for the dialogue policy, this choice is even more essential. different dialogue domains typically share structure and behaviour that should be reflected in the state and action representations. the architecture needs to exploit such common structure, to the benefit of any algorithm applied to the model. in this work, we therefore centre our attention on this architecture. we contribute 1 our architecture can seamlessly integrate new information, has significant zero-shot performance and can cope with large state spaces that naturally arise from a growing number of domains while maintaining a fixed number of network parameters. ## dynamic dialogue policy transformer our goal is to build a model that can talk about a potentially very large number of domains and is able to deal with new domains and domain extensions seamlessly without requiring any architectural changes. in particular, the number of model parameters should remain fixed. this is challenging since new domains require understanding of previously unseen information and the ability to talk about new topics. our approach is inspired by the way an employee would explain and act upon a novel task: 1) describe the information that can be used and the actions that can be taken in natural language, 2) restrict the focus to the information that is important for solving the task at hand, 3) when an action needs to be taken, this action is based on the information that was attended to (e.g. for the action to request the area, one would put attention on the information whether the area is already given). we propose an architecture that uses the transformer encoder with information embeddings (section 4.1 and figure 2(c )) to fulfill 1) and 2) and the transformer decoder that leverages the domain gate (section 4.2, 4.3 and figure 3 ) to fulfill 3), which we call dynamic dialogue policy transformer (ddpt). ## conclusion in this work we provided an algorithm, baseline models and evaluation metrics to enable continual rl for dialogue policy optimisation. moreover, we proposed a dynamic dialogue policy model called ddpt that builds on information descriptions, a pretrained language model and the transformer encoder-decoder architecture. it integrates new information seamlessly as long as it is descriptive, and obtains significant zero-shot performance on unseen domains while being robust to forgetting. the strengths of ddpt were validated in simulation with two simulators as well as humans. this opens the door for building evolving dialogue systems, that continually expand their knowledge and improve their behaviour throughout their lifetime. the critic function is then optimized to minimize the gap between its prediction and the vtrace-target: the actor is optimized using the following offpolicy policy gradient: