Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition (2020)

Abstract: Many studies have applied reinforcement learning to train a dialog policy and show great promise these years. One common approach is to employ a user simulator to obtain a large number of simulated user experiences for reinforcement learning algorithms. However, modeling a realistic user simulator is challenging. A rule-based simulator requires heavy domain expertise for complex tasks, and a data-driven simulator requires considerable data and it is even unclear how to evaluate a simulator. To avoid explicitly building a user simulator beforehand, we propose Multi-Agent Dialog Policy Learning, which regards both the system and the user as the dialog agents. Two agents interact with each other and are jointly learned simultaneously. The method uses the actor-critic framework to facilitate pretraining and improve scalability. We also propose Hybrid Value Network for the role-aware reward decomposition to integrate role-specific domain knowledge of each agent in the task-oriented dialog. Results show that our method can successfully build a system policy and a user policy simultaneously, and two agents can achieve a high task success rate through conversational interaction.

## introduction dialog policy, which decides the next action that the dialog agent should take, plays a vital role in a task-oriented dialog system. more recently, dialog policy learning has been widely formulated as a reinforcement learning (rl) problem @xcite @xcite @xcite , which models users as the interactive environment. since rl requires much interaction for training, it is too time-consuming and costly to interact with real users directly. the most common way is first * corresponding author to develop a dialog agent with a user simulator that mimics human behaviors in an offline scenario. designing a reliable user simulator, however, is not trivial and often challenging as it is equivalent to building a good dialog agent. with the growing needs for the dialog system to handle more complex tasks, it will be much challenging and laborious to build a fully rule-based user simulator, which requires heavy domain expertise. datadriven user simulators have been proposed in recent studies @xcite , but they require a considerable quantity of manually labeled data, most of which regard the simulator as a stationary environment. furthermore, there is no standard automatic metric for evaluating these user simulators, as it is unclear to define how closely the simulator resembles real user behaviors. in this paper, we propose multi-agent dialog policy learning (madpl), where the user is regarded as another dialog agent rather than a user simulator. the conversation between the user and the system is modeled as a cooperative interactive process where the system agent and the user agent are trained simultaneously. two dialog agents interact with each other and collaborate to achieve the goal so that they require no explicit domain expertise, which helps develop a dialog system without the need of a well-built user simulator. different from existing methods @xcite , our approach is based on actor-critic framework @xcite in order to facilitate pretraining and bootstrap the rl training. following the paradigm of centralized training with decentralized execution (ctde) @xcite in multi-agent rl (marl), the actor selects its action conditioned only on its local stateaction history, while the critic is trained with the actions of all agents. it should be noted that the roles of two agents are different though they interact with each other figure 1 : the user has his/her own goal to be accomplished and the system is provided with an interface to access an external database. both agents can only obtain information from the other side via communication. in a cooperative setting. as shown in fig. 1 , only the user agent knows the user goal, while only the system agent can access the backend database. the user agent should express the requirements completely in an organized way, and the system should respond with useful information accurately and immediately. so it is inappropriate to apply simple self-play rl @xcite that views two agents as the same agent in this task. to address this issue, the system and the user are viewed as two asymmetric agents in madpl. we introduce hybrid value network (hvn) for roleaware reward decomposition. it decomposes the reward into two parts: one is the role-specific reward that focuses on its local target, and the other is the global reward that represents the shared goal. to evaluate the proposed approach, we conduct our experiments on a multi-domain, multiintent task-oriented dialog corpus, multiwoz @xcite . the corpus involves high dimensional state and action spaces, multiple decision making in one turn, which makes it more difficult to get a good system policy as well as a good user policy. the experiments demonstrate that madpl can successfully build a system policy as well as a user policy with the aid of hvn, and two agents can achieve high task success rate in complex tasks by interacting with each other as well as with benchmark policies. to summarize, our contributions are in three folds: ## multi-agent dialog policy learning we first formally describe the task, and then present the overview of our proposed model. specifically, given a user goal @xmath0,r) composed of the user constraints c (e.g. a japanese restaurant in the center of the city) and requests r (e.g. inquiry for address, phone number of a hotel), and given an external database db containing all candidate entities and corresponding information, the user agent and system agent interact with each other in a dialog session to fulfill the user goal. there can be multiple domains in g, and two agents have to accomplish all the subtasks in each domain. both agents can partially observe the environment, i.e. only the user agent knows g, while only the system agent can access db, and the only way to know each other's information is through conversational interaction. different from ordinary multiagent task setting, two agents in dialog are executed asynchronously. in a single dialog turn, the user agent posts an inquiry first, then the system agent returns a response, and the two communicate alternately. therefore, each dialog session τ can be seen as a trajectory of state-action pairs here we present a novel algorithm, multi-agent dialog policy learning (madpl), as shown in fig. 2 , which can be naturally formulated as a marl problem. two agents interact through dialog acts following @xcite . we choose the actor-critic framework in order to learn an explicitly stochastic dialog policy (actor) for high scalability along with an estimated value function (critic) to bootstrap rl training. besides, this can facilitate imitation learning to pretrain the dialog policy using human-human dialogs. since two agents cooperate to reach success, yet their roles are asymmetric in the dialog, we propose hybrid value network (hvn) to decompose the task reward into different parts for better policy learning. note that our approach is fully data-driven without building a user simulator beforehand, and does not need any other human supervision during training. in the subsequent subsections, we will first explain the state and action used in two dialog policies. then we describe how we decompose the reward and the proposed hvn. at last, we present model optimization. ## human evaluation for human evaluation, we hire amazon mechanical turkers to conduct pairwise comparison between madpl and baselines. since all the policies work at the dialog act level, we generate the texts from dialog acts using hand-crafted templates to make the dialog readable. each turker is asked to read a user goal first, then we show 2 dialog sessions around this user goal, one from madpl and the other from another baseline. we randomly sample 100 goals for each baseline. for each goal, 5 turkers are asked to judge which dialog is better (win, draw or lose) according to different subjective assessments independently: (i) system quality, (ii) user quality, and (iii) task success. the system quality metric evaluates whether the system policy provides the user with the required information efficiently, and the user quality metric evaluates whether the user policy expresses the constraints completely in an organized way. note that we do not evaluate the quality of language generation here. the proportion of the pairwise annotations in which at least 3 of 5 annotators assign the same label to a task is 78.7%/77.3%/83.3% for system quality/user quality/task success, respectively. this indicates that annotators have moderate agreements. the human judgements align well with the results of automatic evaluation, which also indicates the reliability of the metrics used in task-oriented dialog. ## conclusion we present a multi-agent dialog policy algorithm, madpl, that trains the user policy and the sys-tem policy simultaneously. it uses the actor-critic framework to facilitate pretraining and bootstrap rl training in multi-domain task-oriented dialog. we also introduce role-aware reward decomposition to integrate the task knowledge into the algorithm. madpl enables the developers to set up a dialog system rapidly from scratch. it only requires the annotation of dialog acts in the corpus for pretraining and does not need to build a user simulator explicitly beforehand. extensive experiments foot_0 demonstrate the effectiveness, reasonableness and scalability of madpl. as future work, we will apply madpl in the more complex dialogs and verify the role-aware reward decomposition in other dialog scenarios.