A Collaborative Multi-agent Reinforcement Learning Framework for Dialog Action Decomposition (2021)

Abstract: Most reinforcement learning methods for dialog policy learning train a centralized agent that selects a predefined joint action concatenating domain name, intent type, and slot name. The centralized dialog agent suffers from a great many user-agent interaction requirements due to the large action space. Besides, designing the concatenated actions is laborious to engineers and maybe struggled with edge cases. To solve these problems, we model the dialog policy learning problem with a novel multi-agent framework, in which each part of the action is led by a different agent. The framework reduces labor costs for action templates and decreases the size of the action space for each agent. Furthermore, we relieve the non-stationary problem caused by the changing dynamics of the environment as evolving of agents’ policies by introducing a joint optimization process that makes agents can exchange their policy information. Concurrently, an independent experience replay buffer mechanism is integrated to reduce the dependence between gradients of samples to improve training efficiency. The effectiveness of the proposed framework is demonstrated in a multi-domain environment with both user simulator evaluation and human evaluation.

## introduction dialog policy optimization is one of the most critical tasks of task-oriented dialog modeling. recently, it has shown great potentials for using reinforcement learning (rl) based methods to formulate dialog policy learning @xcite . however, most of these methods learn a centralized agent based on the joint action space that covers predefined atomic action @xcite , which is the concatenation of domain name, intent type, and slot name, e.g. 'restaurant-inform-address', or both atomic actions and the top-k most frequent atomic action combinations @xcite . the elaborate concatenated actions may achieve acceptable performance in simple cases, however, continuously suffer from being laborious to engineers and struggled with edge cases in multi-domain or complex scenes. another drawback of the centralized agent is its exponential growth in the observation and actions spaces with the growing number of domains @xcite . to alleviate the problem of large user-agent interaction requirements caused by the large action space, a hierarchical reinforcement learning framework was proposed to learn the dialog policy that operates at different temporal scales @xcite . it has achieved promising results, however, is still up against some challenges. firstly, their setting requires a rule-based critic to provide the intrinsic reward for the low-level agent. however, creating such a critic is not easy, especially in intricate scenarios. the man-made critic, somewhat inadvertently, may bias the convergent optimal. moreover, the action space composed of intent and slot for the low-level agent can be still large, especially when there are a lot of intent types and slot names. drawing the structural features of dialog actions, we address the above problems with a proposed collaborative multi-agent reinforcement learning framework, where the concatenated dialog action space is decomposed into subspaces corresponding to the domain, intent type, and slot name. furthermore, each subspace is assigned to different agents, which cooperate to make the final joint action without any human knowledge. the agents concatenate together and pass the output to the next agent. to relieve the non-stationary problem @xcite caused by unexpected changes in the dynamics of the environment as evolving of the agents' policies and to reduce the dependence of the gradients due to the non-independent data, we propose a new approach which allows joint optimization based on independent experience replay buffers for all agents, termed as joie. our experiments show that such a multi-agent framework reduces the state-action space size significantly and make exploration more efficient. furthermore, joie leads to a better performance benefit from the proposed optimization mechanism. to the best of our knowledge, this is the first work that strives to develop a multi-agent rl-based dialog action decomposition framework. our main contributions are three-fold: ## related work many studies have been dedicated to optimizing dialog policy with reinforcement learning, most of which learn a centralized agent that maps the observation to a joint action @xcite @xcite @xcite @xcite @xcite . for more efficient exploration, @xcite factor the centralized spaces into hierarchical reinforcement learning paradigms. meanwhile, cooperative multi-agent reinforcement learning methods have started moving from tabular methods to deep learning methods and are widely applied especially on computer games @xcite @xcite . towards multi-agent taskoriented dialog policy, a lot of progress is being made in modeling the interaction as a stochastic collaborative game, where dialog agent and the user simulator are jointly optimized with their objectives @xcite @xcite . building a user simulator in this way is more flexible. however, different from existing frameworks, our multi-agent framework is devoted to decompose concatenated actions in order to reduce the large action space size to improve the performance of dialog agents. different from the previous methods that learn a centralized agent or that adopt hierarchical rl paradigms, we cast the policy learning as a multiagent rl framework, as shown in figure 1 . it integrates three agents specified to be responsible for the domain a d , intent type a i , and slot name a s , respectively. they share reward r and make decisions cooperatively based on the state s from the user. consequently, a concatenation a a from the three agents is passed to the user. ## experiments comparison is on multiwoz @xcite with a public available agenda-based user simulator @xcite . the detail of the user simulator and implementation is in appendix b, c. we first evaluate 2-agent based models that factor the centralized spaces into two subspaces of the domain and joint intent-slot on 3 different domains sizes of 2, 4, and 7 on multiwoz. then we compare 3-agent based models that decompose the action spaces into three subspaces of the domain, intent, and slot. the dataset contains 7 domains, 13 intents, and 28 slots totally. details of the dataset are provided in appendix a.