Guided Dialogue Policy Learning without Adversarial Learning in the Loop (2020)

Abstract: Reinforcement learning methods have emerged as a popular choice for training an efficient and effective dialogue policy. However, these methods suffer from sparse and unstable reward signals returned by a user simulator only when a dialogue finishes. Besides, the reward signal is manually designed by human experts, which requires domain knowledge. Recently, a number of adversarial learning methods have been proposed to learn the reward function together with the dialogue policy. However, to alternatively update the dialogue policy and the reward model on the fly, we are limited to policy-gradient-based algorithms, such as REINFORCE and PPO. Moreover, the alternating training of a dialogue agent and the reward model can easily get stuck in local optima or result in mode collapse. To overcome the listed issues, we propose to decompose the adversarial training into two steps. First, we train the discriminator with an auxiliary dialogue generator and then incorporate a derived reward model into a common reinforcement learning method to guide the dialogue policy learning. This approach is applicable to both on-policy and off-policy reinforcement learning methods. Based on our extensive experimentation, we can conclude the proposed method: (1) achieves a remarkable task success rate using both on-policy and off-policy reinforcement learning methods; and (2) has potential to transfer knowledge from existing domains to a new domain.

## introduction task-oriented dialogue systems (tdss), such as siri, google assistant, and amazon alexa, aim to offer users assistance with completing tasks. tdss need dialogue policies to select appropriate actions at each dialogue step according to the current context of the conversation @xcite . the development of rl in robotics and other domains has brought a new view on learn-ing dialogue policies @xcite @xcite : it allows us to train with far more data than can be feasibly collected from actual users. the aim of tdss is to maximize positive user feedback. tdss based on rl are amenable to training with user simulators instead of real humans @xcite . user simulators rely on a reward function that scores system actions given dialogue context @xcite @xcite . the most straightforward way to design a dialogue reward function is to score the agent based on the dialogue status in a rule-based fashion: if the dialogue ends successfully, a large positive reward will be returned; if the dialogue fails, the reward will be a large negative value; if the dialogue is still ongoing, a small negative value will be returned to encourage shorter sessions @xcite . however, the rule-based solution is inflexible as it assigns the same negative reward to all the system actions before the dialogue ends. the sparse reward makes the qualities of different actions indistinguishable. additionally, the rule-based approaches only return a meaningful reward when dialogue finishes, which can delay the penalty for low-quality actions and a high reward for high-quality ones during the conversation itself. @xcite address the difficulties listed above by employing adversarial training for policy learning by jointly training two systems: (1) a policy model that decides which action to take at each turn, and (2) a discriminator that marks if a dialogue was successful or not. feedback from the discriminator is used as a reward to push the policy model to complete a task indistinguishably from humans. improving upon this solution, @xcite propose to replace the discriminator with a reward function that acts at the dialogue action level and returns the reward for the given action relying on the dialogue state, system action, and next dialogue state as its input. however, the described methods are limited to policy gradient-based algorithms, such as re-inforce @xcite and proximal policy optimization (ppo) @xcite , to alternatively update the dialogue policy and the reward model on the fly, while off-policy methods are not able to benefit from self-learned reward functions. furthermore, the alternative training of the dialogue agent and the reward model can easily get stuck in local optima or result in mode collapse. to alleviate the problems mentioned above, in this work we propose a new approach for training dialogue policy by decomposing the adversarial learning method into two sequential steps. first, we learn the reward function using an auxiliary dialogue state generator where the loss from the discriminator can be backpropagated to the generator directly. second, the trained discriminator as the dialogue reward model will be incorporated into the rl process to guide dialogue policy learning and will not be updated, while the state generator is discarded. therefore, we can utilize any rl algorithm to update the dialogue policy, including both on-policy and off-policy methods. additionally, since the reward function is pre-trained in an offline manner, we can first infer common information contained in high-quality human-generated dialogues by distinguishing human-generated dialogues from machine-generated ones, and then make full use of the learned information to guide the dialogue policy learning in a new domain in the style of transfer learning. to summarize, our contributions are: • a reward learning method that is applicable to off-policy rl methods in dialogue training. the reward function is a discriminator aiming to assign a high value to real human dialogues and a low value to dialogues generated by the current dialogue policy. in contrast, the dialogue policy attempts to achieve higher reward from the discriminator given the generated dialogue. following this solution, @xcite replaces the discriminator with a reward function a reward function that acts at the dialogue action level, which takes as input the dialogue state, system action, and next dialogue state and returns the reward for the given dialogue action. the key distinction of our work compared to previous efforts is being able to train dialogue agents with both: (1) off-policy methods in adversarial learning settings; (2) the on-policy based approaches while avoiding potential training issues, such as mode collapse and local optimum. we propose to train (1) reward model and (2) dialogue policy consecutively, rather than alternatively as suggested in @xcite . to train the reward model, we introduce an auxiliary generator that is used to explore potential dialogue situations. the advantage of our setup is the transfer from seqgan @xcite to a vanilla gan @xcite . in seq-gan setup, the policy gradient method is essential to deliver the update signal from the discriminator to the dialogue agent. in contrast, in the vanilla gan, the discriminator can directly backpropogate the update signal to the generator. once we restore a high-quality reward model, we update the dialogue agent using common rl methods, including both on-policy and off-policy. ## learning reward functions in this section, we introduce our method to learn reward functions with an auxiliary generator. ## conclusion in this work, we have proposed a guided dialogue policy training method without using adversarial training in the loop. first, we trained the reward model with an auxiliary generator. then the trained reward model was incorporated into a common reinforcement learning method to guide training of a high-quality dialogue agent. by conducting extensive experimentation, we demonstrated that the proposed methods achieve remarkable performance, in terms of task success, as well as the potential to transfer knowledge from previously utilized task domains to new ones.