L earning D ialog P olicies from W eak D emonstrations (2020)

Abstract: Deep reinforcement learning is a promising approach to training a dialog manager, but current methods struggle with the large state and action spaces of multi-domain dialog systems. Building upon Deep Q-learning from Demonstrations (DQfD), an algorithm that scores highly in difficult Atari games, we leverage dialog data to guide the agent to successfully respond to a user’s requests. We make progressively fewer assumptions about the data needed, using labeled, reduced-labeled, and even unlabeled data to train expert demonstrators. We introduce Reinforced Fine-tune Learning, an extension to DQfD, enabling us to overcome the domain gap between the datasets and the environment. Experiments in a challenging multi-domain dialog system framework validate our approaches, and get high success rates even when trained on out-of-domain data.

## introduction the dialog manager (dm) is the brain of a taskoriented dialog system. given the information it has received or gleaned from a user, it decides how to respond. typically, this module is composed of an extensive set of hand-crafted rules covering the decision tree of a dialog @xcite . to circumvent the high development cost of writing and maintaining these rules there have been efforts to automatically learn a dialog manager using reinforcement learning (rl; @xcite . rl solves problems of optimal control -where past predictions affect future states -making it well-suited to dialog management, in which a misstep by the agent can throw the whole dialog off course. but using rl to train a dialog manager is not straightforward, and is often hindered by large dialog state spaces and sparse rewards @xcite . an established way to deal with these difficulties is to guide the dialog manager with expert demonstrations during rl training @xcite , a high-level illustration of which is shown in figure 1 . this approach, however, requires a rule-based oracle to provide a suitable system response given a dialog state, and does not exploit the knowledge contained in the growing number of dialog datasets @xcite . in this paper, we address two key-questions that arise when training rl dialog agents with expert demonstrations: (i) can we move away from rulebased experts and use weaker, cheaper demonstrations to guide the rl dialog manager? (ii) can we exploit information gathered during rl training to improve the demonstrator and bridge the domain gap between dialog data and the rl environment? to answer the first question, we explore three methods based on deep q-learning from demonstrations (dqfd; @xcite ) that use trained experts derived from progressively weaker data. our first and strongest expert is a full label expert (fle) trained on a labeled, in-domain dataset to predict the next system response. second, we train a reduced label expert (rle) to predict the type of the next system response, but not its exact nature. finally our third expert is a no label expert (nle) that does not rely on any annotation at all, but is instead trained on unlabeled user utterance and agent response sentences. we show that all three experts can be used to successfully train rl agents, and two of them even allow us to train without expensive and often hard to come-by fully annotated in-domain dialog datasets. we address our second key question -how to improve the experts during rl training -by presenting reinforced fine-tune learning (rofl), a fine-tuning algorithm inspired by dataset aggregation (dagger; @xcite . rofl bridges the domain gap between dialog data and the rl environment by using the dialog transitions generated during training to update the expert's weights, adapting the previously learned knowledge to the learning environment. our experiments show that rofl training improves demonstrations gathered from the employed experts, giving a boost in rl performance and hastening convergence. ## related work our work is closely related to research in using expert demonstrations to guide reinforcement learning dialog managers. lipton et al. (2018) "spike" the deep q-network (dqn; @xcite replay buffer with a few successful demonstrations from a rule-based dialog manager. gordon-hall et al. (2020) extend this approach and apply deep q-learning from demonstrations (dqfd) to dialog, prefilling a portion of the buffer with expert transitions and encouraging the agent to imitate them by adding an auxiliary term to the dqn loss. demonstrations are not the only way to incor-porate external expertise into the dialog manager. one alternative is to use supervised learning to train a neural network policy on an in-domain dialog dataset, and then fine-tune it with policygradient rl on a user-simulator @xcite @xcite . @xcite fine-tune their rl policy on human rather than simulated users. another, parallel, approach to rl-based dms aims to increase the frequency of meaningful rewards. takanobu et al. (2019) use inverse rl to learn a dense reward based on a dialog corpus, while lu et al. (2019) decompose the task into subgoals that can be regularly assessed. weak demonstrations have been used outside of dialog system research to tackle rl environments with large state spaces and sparse rewards. aytar et al. (2018) train an expert to imitate youtube videos of people playing challenging atari games and exceed human-level performance. salimans and chen (2018 ) beat their score on montezuma's revenge using only a single human demonstration, resetting the environment to different states from the expert trajectory. however we believe our work is the first to explore the use of weak demonstrations for dqfd in a dialog environment. rofl, our proposed fine-tuning method, is inspired by dagger @xcite , an iterative imitation learning algorithm that incorporates feedback from an expert to improve the performance of a policy. dagger requires an on-line expert that can be queried at any time, and which bounds the policy's performance. if the expert is suboptimal the policy will be too. @xcite lift this restriction, allowing the policy to explore the search space around expert trajectories, but their method (lols) does not incorporate rl policy updates as we do. ## background training a dialog manager -or agent -with reinforcement learning involves exposing it to an environment that assigns a reward to each of its actions. this environment consists of a database that the dm can query, and a user-simulator that mimics a human user trying to achieve a set of goals by talking to the agent. the more user goals the agent satisfies, the higher its reward. given the current state s t of the dialog, the agent chooses the next system action a t according to a policy π, a @xmath0, and receives a reward r t . the ex-pected total reward of taking an action a in state s with respect to π is estimated by the q-function: the term l aux depends on the expert used to provide demonstrations. for each of our three experts we will define a different auxiliary loss. ## method it has been shown that dqfd successfully trains a dialog manager when its demonstrations come from either a rule-based, or strong pre-trained expert @xcite . to avoid writing rules, and to exploit the knowledge contained in external datasets, we expand on previous work and adapt dqfd for use with three progressively weaker and cheaper experts. furthermore, we introduce our rofl algorithm, describing how we fine-tune the expert during rl training. full label expert we define a full label expert (fle) as a classifier trained on a human-tohuman in-domain dialog dataset to predict, given the conversation state, the next action. for such an expert, the action space of the dataset corresponds to the actions in the rl environment and, as a result, we can use the original dqfd large margin classification term as an auxiliary loss: this fle approach is similar to the data-driven expert introduced by @xcite . reduced label expert a full label expert is trained on fully-annotated in-domain data, but this is lacking for many domains, and is expensive to collect and label from scratch @xcite . however, although existing dialog datasets often differ in annotation, many share high-level system labels: inform and request. inform actions denote that the system provides information; request actions that the system asks for it. a system utterance from a hotel-booking dataset, e.g. "the le grand hotel costs $48 per night, how many nights do you want to stay?", could be labelled: [hotel-inform-price, hotel-request-duration], while a sentence from a taxi-booking dataset, e.g. "please let me know the dropoff location.", could be annotated: taxi-request-dropoff. although the domain and type of information are different, all actions a in either dataset can be broadly partitioned into sets a reduced ⊂ a according to whether they inform, request, or do both. we introduce a reduced label expert (rle) to take advantage of this common annotation format across diverse datasets. the rle is a multilabel classifier that predicts the high-level annotation set a reduced -or reduced label -of the next system action given the list s n l of the last few utterances in the dialog. the rle is trained on a dialog dataset stripped down to inform, request, and other (for all other actions) annotations. its architecture is outlined in figure 2 . the previous user utterances are passed through a recurrent encoder, for example an rnn. the final hidden state of the encoder is then passed through a multi-label classifier which uses the sigmoid function to score each reduced label. once trained, we use the rle to guide the dialog manager during dqfd training. first we divide all environment actions into reduced label sets. for example, the inform set would consist of the environment actions that pertain to providing information to the user. unlike the fle, the rle does not predict exact actions, so we uniformly sample an environment action from the predicted reduced label set a e ∼ a reduced to use as an expert demonstration when prefilling the replay buffer. for example, if the rle predicts request the expert might take the action request-hotel-price. in order to use the expert in network updates, we reformulate the ℓ term in the dqfd's auxiliary loss to account for the expert's reduced label prediction: the agent is penalized by a positive constant term c if the action predicted by its current policy π θ is not in the set of actions licensed by the rle. no label expert while the rle enables the use of data not annotated for the target dialog environment, it still requires labeled dialog data. this raises the question: can we employ an expert that does not rely on annotations at all? to address this challenge, we propose a no label expert (nle) that uses an unannotated dialog dataset consisting of pairs of sentences (s u , s a ), representing user utterances and the corresponding agent responses. the goal of the nle is to predict whether, for a given pair of sentences, s a is an appropriate response to s u . in this regard, it resembles models used to predict textual inference @xcite . the nle architecture is outlined in figure 3 . the previous user utterance and a verbalized system response -generated by an nlg component -are consecutively passed through a sentence embedder. their encodings are then concatenated and passed through a network which scores how appropriate the response is given the utterance. the nle is trained on unannotated human-tohuman dialog datasets which are formatted into pairs of user utterances and agent responses. we treat these as positive instances, making the tacit assumption that in the data the agent's reply is always relevant given a user utterance. as a result, the data lacks negative examples of irrelevant agent responses. this can be mitigated by artificially creating negative pairs (s u , s a ) from the original data by pairing each user utterance s u with random agent sentences s a , drawn uniformly from all agent responses that were not observed for the original s u . given such a dataset of positive and negative user-agent interactions, we train an nle that learns to output 1 if a system response corresponds to the last user utterance, and 0 if it does not. once trained, we use this nle to guide the dqfd dialog manager. when prefilling the replay buffer with expert demonstrations, we calculate the set a no label of all actions a whose verbalization s a leads to an nle output that exceeds a threshold ρ when taken as a response to the last user utterance s u . we then use a random action from this set a e ∼ a no label as the expert demonstration and place it in the replay buffer. we use a similar ℓ term in the auxiliary loss to the reduced label expert, which penalizes the agent if the action a predicted by its current policy is not in the set of actions licensed by the expert, i.e., if @xmath1: ## experimental setup we evaluate our weak experts in convlab @xcite , a multi-domain dialog framework based on the multiwoz dataset @xcite . in convlab, the dialog manager's task is to help a user plan and book a trip around a city, a problem that spans multiple domains ranging from recommending attractions for sightseeing, to booking transportation (taxi and train) and hotel accommodation. convlab supports rl training with an environment that includes an agenda-based user-simulator @xcite and a database. the agent has a binary dialog state that encodes the task-relevant information that the environment has provided so far. this state has 392 elements yielding a state space of size 2 392 . in each state there are 300 actions that the dm can choose between, corresponding to different system responses when verbalized by the natural language generation (nlg) module. these actions are composite and can consist of several individual informs and requests. for example, [attraction-inform-name, attraction-request-area] is one action. we train our dms on the exact dialog-acts produced by the user-simulator, avoiding error propagation from a natural language understanding (nlu) module. we use convlab's default template-based nlg module to verbalize system actions when using the rle and nle. first, we experiment with experts trained on the in-domain multiwoz dataset 1 . for the fle we train on the full annotations; for the rle we reduce the annotations to minimal inform, request, other labels; and for the nle we only use the unannotated text. we also experiment with experts trained on out-of-domain (ood) data. to this end, we combine two datasets: microsoft e2e @xcite -10,087 dialogs composed of movie, restaurant and taxi booking domains -and maluuba frames @xcite which is made up of 1,369 dialogs from the flight and hotel booking domains. while three of these domains are also in multiwoz, the specifics of the conversations are different. our full label expert is a feedforward neural network (ffn) with one 150 dimensional hidden layer, relu activation function and 0.1 dropout which takes the current dialog state as input. the reduced label expert uses the last utterance in the conversation as context, which is embedded with 300 dimensional pre-trained glove embeddings @xcite , then passed through a uni-directional 128 dimensional hidden layer gru @xcite from which the last hidden state is used to make a multi-label prediction. finally, our no label expert uses pre-trained bert base-uncased @xcite to embed and concatenate user and agent utterances into 1536dimensional input vectors, and employs a feedforward neural network with selu activations @xcite to predict whether the agent's response is an appropriate answer to the last user utterance. note that the rle and nle both take natural language as input yet use different word embeddings. we conducted preliminary experiments to evaluate the efficacy of bert and glove embeddings for the respective expert training tasks. while we found that the nle greatly benefited from bert over glove, the rle performance did not differ between embeddings. since glove vectors yield a significant runtime advantage over the course of rl training, we used glove 1 we use multiwoz2.0 with convlab user annotations for the rle, while employing slower bert embeddings for the nle due to the significantly better performance. for rl training of our dqfd agents, we use a prioritized replay buffer @xcite with a maximum buffer size of 100,000 transitions. we follow the dqfd setup of @xcite and apply l2 regularization with a weight of 10 -5 and drop the n-step term from the original dqfd loss. all rl networks have a 100 dimensional hidden layer, a dueling network structure, and use the double dqn loss @xcite . all our networks are trained with the radam optimizer @xcite with a learning rate of 0.01. for a complete list of hyperparameters used for our experiments refer to the attached supplemental material. we slightly alter the rofl algorithm presented in 4 to account for the fact that convlab only rewards the agent based on whether it successfully completed the task at the end of a dialog (intermediate steps are uniformly assigned a -1 step penalty). rather than immediately adding transitions to the fine-tune dataset d, we wait until the end of a conversation and check if its total reward exceeds a threshold th. if it does, we assume that all transitions in that conversation are perfect, and add them to d. for our experiments we empirically determine th, and set it to 70. we train all our rl-based dialog managers for 3 sessions of 2,500,000 steps, and anneal the exploration parameter over the first 500,000 to a final value of 0.01. results and training graphs in the following section are the average of these 3 sessions. each session takes under 10 hours on one nvidia geforce rtx 2080 gpu. we compare our approach to supervised and reinforcement learning baselines. ## results in-domain weak expert dqfd we train indomain reduced and no label experts on the mul-tiwoz dataset. the rle scores 77 f1 on the reduced label test set, while the nle manages 71 f1 of predicting whether an agent response belongs to a user utterance on the unannotated test set. as shown in figure 4 shows graphs of the average success rates of dqn, ppo, and our proposed dqfd agents over the course of training. dqn struggles to find successful dialog strategies, although its success rate slowly inclines and seems to gain some traction towards the end of the maximum training steps. to begin with ppo learns rapidly, faster than rle and nle, but its success rate plateaus in the 60% range; it seems to learn to end dialogues too early. both re and fle start with performance advantages, due to their high quality expert demonstrations. over time, re even approaches the success rate of its rule-based expert demonstrator. the fle consistently outperforms approaches with weaker demonstrations, quickly exceeding the success rate of the underlying ffn after an early dip when the agent's exploration parameter is relatively high. the nle comfortably outperforms the reduced label expert throughout training, with the rle only overtaking it at the end. we believe that this strong relative performance makes sense if we consider that, during pre-training, the nle acts according to a more fine-grained action set than the rle. while the rle partitions the actions according to their reduced label, these sets are broad and contain many irrelevant responses, whereas the nle acts randomly according to a smaller, potentially higher-quality, set of actions which have high correspondence scores. finally, the graphs in figure 4 indicate that none of the agents fully converge after the training step limit, although re and fle plateau. it is possible that after significantly more steps even dqn would converge to the ceiling performance of the rule dm -but all our methods are considerably more sample efficient. ## conclusions and future work in this paper, we have shown that weak demonstrations can be leveraged to learn an accurate dialog manager with deep q-learning from demonstrations in a challenging multi-domain environment. we established that expert demonstrators can be trained on labeled, reduced-labeled, and unlabeled data and still guide the rl agent by means of their respective auxiliary losses. evaluation has shown that all experts exceeded the performance of reinforcement and supervised learning baselines, and in some cases even approached the results of a hand-crafted rule-based dialog manager. furthermore, we introduced reinforced finetune learning (rofl) a dagger-inspired extension to dqfd which allows a pre-trained expert to adapt to an rl environment on-the-fly, bridging the domain-gap. our experiments show that rofl training is beneficial across different sources of demonstration data, boosting both the rate of convergence and final system performance. it even enables an expert trained on unannotated out-ofdomain data to guide an rl dialog manager in a challenging environment. in future, we want to continue to investigate the possibility of using even weaker demonstrations. since our no label expert is trained on unannotated data, it would be interesting to leverage large and noisy conversational datasets drawn from message boards or movie subtitles, and to see how rofl training fares with such a significant domain gap between the data and the rl environment.