Learning Efficient Dialogue Policy from Demonstrations through Shaping (2020)

Abstract: Training a task-oriented dialogue agent with reinforcement learning is prohibitively expensive since it requires a large volume of interactions with users. Human demonstrations can be used to accelerate learning progress. However, how to effectively leverage demonstrations to learn dialogue policy remains less explored. In this paper, we present Sˆ2Agent that efficiently learns dialogue policy from demonstrations through policy shaping and reward shaping. We use an imitation model to distill knowledge from demonstrations, based on which policy shaping estimates feedback on how the agent should act in policy space. Reward shaping is then incorporated to bonus state-actions similar to demonstrations explicitly in value space encouraging better exploration. The effectiveness of the proposed Sˆ2Agentt is demonstrated in three dialogue domains and a challenging domain adaptation task with both user simulator evaluation and human evaluation.

## introduction with the flourishment of conversational assistants in daily life (like google assistant, amazon alexa, apple siri, and microsoft cortana), task-oriented dialogues that are able to serve users on certain tasks have increasingly attracted research efforts. dialogue policy optimization is one of the most critical tasks of dialogue modeling. one of the most straightforward approaches is the rule-based method, which contains a set of expert-defined rules for dialogue modeling. though rule-based dialogue systems have a reasonable performance in some scenarios, handcrafting such kinds of rules is time-consuming and not scalable. recently, dialogue policy learning is formulated as a reinforcement learning (rl) problem and tackled with deep rl models @xcite @xcite . it has shown great potentials of using the rl-based method for building robust dialogue systems automatically. however, due to its interactive nature, rl-based agents demand of an environment to operate in. as illustrated in figure 1 , rl-based dialogue agents need to interact with human users and update its policy in an online fashion requiring that the agents have a good online performance from the start of training. in addition, one of the biggest challenges of rl approaches is reward sparsity issue, which leads to exploration in large action space inefficient. as a consequence, training rl-based agents expects a prohibitively large number of interactions to achieve acceptable performance, which may incur a significant amount of expense @xcite @xcite . several attempts are made to improve learning efficiency and tackle reward sparsity issues. different types of heuristics has been proposed in the form of intrinsic rewards to guide exploration more efficiently @xcite @xcite @xcite . when building a dialogue system, it is typically affordable to recruit experts to gather some demonstrations about the expected agent behaviors. we therefore aim to address the aforementioned challenges from a different perspective and assume having access to human-provided demonstrations. in this paper, we investigate how to efficiently leverage these demonstrations to alleviate reward sparsity and improve policy learning quality. previous work @xcite used a simple technique termed as replay buffer spiking (rbs) to pre-fill experience replay buffer with human demonstrations, which yields good performance, especially in the beginning. @xcite proposed deep q-learning from demonstrations (dqfd) that combines temporal difference updates with a supervised classification loss of actions in demonstrations to improve learning efficiency in gaming domains. however, whether it is feasible and how to effectively leverage human demonstration in dialogue scenarios are less explored. hence, in this paper, we propose a new strategy of leveraging human demonstrations to learn dialogue policy efficiently. our dialogue agent, termed as s 2 agent 1 , learns dialogue policy from demonstrations trough policy shaping and reward shaping. policy shaping @xcite is an approach to incorporating human feedback to advise how policy should behave like experts. it estimates feedback of a state-action pair from human demonstrations and then utilizes the feedback to reconcile the policy from any rl-based agents. this method speeds up learning progress in gaming domains but has not yet been studied in dialogue. however, directly applying policy shaping to dialogue faces several challenges. the original policy shaping uses a tabular analogous method to estimate feedback. this method limits its feasibility for complex problems like dialogue that has large state action representations. to deal with this issue, we propose to use deep neural networks, which represent state-action space with function approximation and distill knowledge from human demonstrations, to estimate feedback. in addition, policy shaping calibrates agents' behavior in policy space, and it is inherently not designed to tackle reward sparsity issues. considering this, we further introduce reward shaping to bonus these state-action pairs that are similar to demonstrations. it can be viewed as a shaping mechanism explicitly in value space to guide policy exploration towards actions which human experts likely conduct. our contributions in this work are two-fold: policy shaping policy shaping is an algorithm that enables introducing prior knowledge into policy learning. @xcite formulates human feedback on the actions from an agent policy as policy feedback and proposes advise algorithm to estimate humans bayes feedback policy and combine it with the policy from the agent. it shows significant improvement in two gaming environment. @xcite uses policy shaping to bias the search procedure towards semantic parses that are more compatible with the text and achieve excellent performance. reward shaping reward shaping leverages prior knowledge to provides a learning agent with an extra intermediate reward f in addition to environmental reward r, making the system learn from a composite signal r + f @xcite . however, it is not guaranteed that with reward shaping, an mdp can still have an optimal policy that is policy model imitation model user policy shaping human demonstrations reward shaping supervised learning 𝜋 𝑎 𝑎|𝑠 𝜋 𝑒 𝑎|𝑠 g(𝜋 𝑎 , 𝜋 𝑒 ) (𝑠, 𝑎, 𝑟, 𝑠 ′ ) 𝑟 = 𝑟 + 𝐹 𝑑 figure 1: illustration of the s foot_0 agent for dialogue policy learning. ## approach our s 2 agent is illustrated in figure 1 , consisting of four modules. 1) dialogue policy model which selects the best next action based on the current dialogue state.; 2) imitation model is formulated as a classification task that takes dialogue states as input and predicts associated dialogue action, aiming to distill behaviors from human demonstrations.; 3) policy shaping provides feedback on how policy should behave like demonstrations. it then reconciles a final action based on actions from the policy model and imitation model attempting to generate more reliable exploration trajectories; 4) followed by a reward shaping module that encourages demonstration similar state-actions by providing extra intrinsic reward signals. ## experiments and results we evaluate the proposed s 2 agent with a user simulator on several public task-oriented datasets, including movie ticket booking, restaurant reservation, and taxi reservation. additionally, to asses the generalization capability of shaping mechanism, we conduct domain adaptation experiments. finally, human evaluation results are reported. ## conclusion in this paper, we present a new strategy for learning dialogue policy with human demonstrations. compared with previous work, our proposed s 2 agent is capable of learning in a more efficient manner. by using policy shaping and reward shaping, s 2 agent can leverage knowledge distilled from the demonstrations to calibrate actions from underlying rl agents for better trajectories, and obtains extra rewards for these state-actions similar to demonstrations alleviating reward sparsity for better exploration. the results of simulation and human evaluation show that our proposed agent is efficient and effective in both single domain and a challenging domain adaptation setting.