Learning Goal-oriented Dialogue Policy with opposite Agent Awareness (2020)

Abstract: Most existing approaches for goal-oriented dialogue policy learning used reinforcement learning, which focuses on the target agent policy and simply treats the opposite agent policy as part of the environment. While in real-world scenarios, the behavior of an opposite agent often exhibits certain patterns or underlies hidden policies, which can be inferred and utilized by the target agent to facilitate its own decision making. This strategy is common in human mental simulation by first imaging a specific action and the probable results before really acting it. We therefore propose an opposite behavior aware framework for policy learning in goal-oriented dialogues. We estimate the opposite agent’s policy from its behavior and use this estimation to improve the target agent by regarding it as part of the target policy. We evaluate our model on both cooperative and competitive dialogue tasks, showing superior performance over state-of-the-art baselines.

## introduction in goal-oriented dialogue systems, dialogue policy plays a crucial role by deciding the next action to take conditioned on the dialogue state. this problem is often formulated using reinforcement learning (rl) in which the user serves as the environment @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . however, different from symbolic-based and simulation-based rl tasks, such as chess @xcite and video games @xcite ., * corresponding author. therefore, there are some efforts using simulation methods to provide an affordable training environment. one principle direction for mitigating this problem is to leverage human conversation data to build a user simulator, and then to learn the dialogue policy by making simulated interactions with the simulator @xcite @xcite . however, there always exist discrepancies between simulated users and real users due to the inductive biases of the simulation model, which can lead to a sub-optimal dialogue policy @xcite . another direction is to learn the dynamics of dialogue environment during interacting with real user, and concurrently use the learned dynamics for rl planning @xcite @xcite . most of these works are based on deep dyna-q (ddq) framework @xcite , where a world model is introduced to learn the dynamics (which is much like a simulated user) from real experiences. the target agent's policy is trained using both real experiences via direct rl and simulated experiences via a world-model. in the above methods, both the simulated user and world model facilitate target policy learning by providing more simulated experiences and remain a black box for the target agent. that is, the target agent's knowledge about the simulated agents is still passively obtained through interaction and implicitly learned by the policy model updating as indirect try-and-error with real user. however, we argue that from the angle of a target agent, actively exploring the world with proper estimation would not only make user simulation more efficient but also improve the target agent's performance. in agreement with the findings from cognitive science, humans often maintain models of other people they interact with to capture their goals @xcite . and humans manage to use their mental process to simulate others' behavior @xcite . therefore, to carefully treat and model the behaviors of other agents would be full of potential. for example, in competitive tasks such as chess, the player often sees a number of moves ahead by considering the possible reaction of the other player. in goal-oriented dialogues for a hotel booking task, the agent can reduce interaction numbers and improve user experience by modeling users as business travellers with strict time limit or backpackers seeking adventure. in this paper, we propose a new dialogue policy learning method with opposite agent awareness (oppa), where the agent maintains explicit modeling of the opposite agent or user for facilitating its own policy learning. different from ddq, the estimated user model is not utilized as a simulator to produce simulated experiences, but as an auxiliary component of the target agent's policy to guide the next action. figure 1(c ) shows the framework of our model. specifically, whenever the system needs to output an action, it foresees a candidate action ât and consequently estimates the user's response behavior a o t+1 . on top of this estimation, as well as the dialogue context, it makes better decisions with a dynamic estimation of the user's strategy. to further regulate the behavior of the system agent, we mitigate the difference between the real system action a t and the sampled action ât with decay for better robustness and consistency. without any constraint on the type of agents (either competitive or cooperative), the proposed oppa method can be applied to both cooperative and non-cooperative goal-oriented dialogues. to summarize, our contributions are three-fold: ## model in this section, we introduce our proposed oppa model. there are two agents in our framework, one is the system agent we want to optimize, and the other is the user agent. we refer to these two agents as target and opposite agents in the following sections. note that the proposed model works at dialog act level, and it can also work at natural language level when equipped with natural language understanding (nlu) and natural language generation (nlg) modules. ## experiment depending on the task, dialogues can be divided into cooperative and competitive ones. in a cooperative task, the aim can be reducing unnecessary interactions by inferring the opposite person's intention. while in competitive tasks, the aim is usually to maximize their own interests by considering the opposite agents' possible reactions. to test our method's wide suitability, we evaluated it on both cooperative and competitive tasks. ## conclusion in this work, we present an opposite agent-aware dialogue policy model which actively estimates the opposite agent instead of doing passive learning from experiences. we have shown that it is possible to harvest a reliable model of the opposite agent through more efficient dialogue interactions. by incorporating the estimated model output as part of dialogue state, the target agent shows significant improvement on both cooperative and competitive goal-oriented tasks. as future work, we will explore multi-party dialogue modeling in which multi-agent learning techniques can be applied.