Multi-Action Dialog Policy Learning with Interactive Human Teaching (2020)

Abstract: We present a framework for improving task-oriented dialog systems through online interactive teaching with human trainers. A dialog policy trained with imitation learning on a limited corpus may not generalize well to novel dialog flows often uncovered in live interactions. This issue is magnified in multi-action dialog policies which have a more expressive action space. In our approach, a pre-trained dialog policy model interacts with human trainers, and at each turn the trainers choose the best output among N-best multi-action outputs. We present a novel multi-domain, multi-action dialog policy architecture trained on MultiWOZ, and show that small amounts of online supervision can lead to significant improvement in model performance. We also present transfer learning results which show that interactive learning in one domain improves policy model performance in related domains.

## introduction task-oriented dialog systems help users to complete tasks by interacting with the user through a multi-turn natural dialogue @xcite . the dialog manager module plays a key role of maintaining state across the conversation and selecting actions in each turn to drive the dialog to successful completion. within the dialog manager, the dialog policy module chooses the system's actions in each state @xcite , and it is typically constructed in one of the following ways: (1) handcrafted with rules defined by a conversation designer @xcite , (2) trained with imitation learning on dialog samples collected from human-human interactions @xcite @xcite , or (3) trained with reinforcement learning with a user simulator @xcite . in practice, each approach has its unique advantages and disadvantages, making it difficult to build an optimal dialog policy with a single approach. systems crafted from large numbers of rules @xcite are time-intensive to build and often lead to rigid dialog flows. supervised learning over humanhuman dialog samples is widely studied. however, human-human dialogs collected in a wizard-of-oz setup @xcite cannot cover all dialog states occurring in human-machine interactions, such as dialog states occurring due to system errors. models trained on human-human data alone do not generalize well to human-machine dialogs and face compounding errors when a deviation in a single turn takes the dialog to a new state which the model might have never seen during training @xcite . in contrast, dialog systems trained with reinforcement learning, either with user simulators or by receiving feedback from user interactions, have shown improved robustness in diverse dialogue scenarios @xcite . however, the reward signal used in rl provides distant and weak supervision, resulting in large amounts of samples required for the model to learn the credit assignment between actions and outcomes @xcite . a number of works attempt to combine the best of both worlds through hybrid approaches @xcite . most prior work on dialog policy modeling assumes only one policy action per turn @xcite @xcite , which limits interaction quality and increases dialog length, leading to more errors. generating multiple dialog acts in a single turn can increase the system's expressive power, and this can be formulated as a multi-label classification or a sequence generation problem @xcite . however, having more than one act in a single turn exponentially increases the space of possible outputs. a limited corpus is unlikely to cover a large number of combinations of output acts, and models trained with supervised learning alone will be restricted to a small subspace of the complete output action space. in this paper, we propose "policy learning with interactive action selection" (plias), a generic framework for learning dialog policies which combines pre-training on human-human dialog samples and interactive learning with human-machine interactions. the interactive learning step is designed to maximize supervision quality while minimizing annotation time and cost. we employ the plias framework on dialog action sequence policy (dasp), a novel multi-domain, multi-action dialog policy architecture. experiments on multi-woz @xcite show that plias significantly improves model performance. ## policy learning with interactive action selection (plias) figure 1 shows the 3-step approach of plias: (1) pre-train a dialog policy model on an annotated human-human dialog corpus, (2) generate humanmachine interactions where a human interacts with the model and picks the best output from n-best policy outputs, (3) fine-tune the policy model on the interactive learning dialog sessions from step 2. in this section, we describe plias in context of interactively improving the dasp model. ## experiments we present experiments on multiwoz @xcite , restricted to dialogs in two domains, restaurant and hotel, including dialogs that span both of them, which we refer to as multi. for all the experiments, we use a rule-based belief tracker to track the slot updates across each turn, and a template-based nlg module @xcite . the dasp model requires a nlu slot tagger to delexicalize the user inputs. to isolate the impact of plias from the effectiveness of the slot tagger, we devised two modes in our interactive learning step: trained-slot-tagger (tst) and ground-truth-slot-tagger (gtst). in tst, we trained a seq2seq slot tagger @xcite on user utterances in multiwoz corpus, and integrated it in the action selection step to tag the human trainer's input utterances. in gtst, we switched the trainer's input from free-form text to a search over templated user utterances extracted from multiwoz (fig 3 ), which skips the need for slot tagging and enables us to collect interactive learning samples with gold nlu labels. we pre-trained a single multi-domain model on the entire train split of multiwoz (4000 dialogs), then ran interactive action selection of 300 dialog sessions for each pair of restaurant, hotel, multi and tst, gtst. to measure the effectiveness of plias, we evaluate all three models dasp-pt, dasp-bt and dasp-ft. in the interactive evaluation mode, action selection is disabled and the system responds with the top action sequence prediction. the trainer gives a 1-5 rating for each turn based on the quality of the system's chosen output. we collected 100 sessions of interactive evaluation for each combination of dasp model, domain, and slot-tagger mode. we report two scores for each experiment: (1) task success rate (tsr), which aggregates the overall rate of task completion of the model in human-machine interactions, and (2) avg. turn-wise human rating, which aggregates the subjective per-turn feedback score given by the human trainers. we also present a transfer learning experiment to evaluate the effectiveness of interactive policy learning to generalize knowledge to related domains. in this experiment, we trained new dasp-ft and dasp-bt models (in gtst mode) on the interactive learning samples restricted to restaurant domain, and performed interactive evaluations of these models on tasks from all three domainsrestaurant, hotel and multi. ## conclusion and future work in this paper, we presented plias, a generic framework for policy learning through interactive action selection. we described the framework in context of a novel multi-domain multi-action policy model,