Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation (2020)

Abstract: Dialogue policy optimization often obtains feedback until task completion in task-oriented dialogue systems. This is insufficient for training intermediate dialogue turns since supervision signals (or rewards) are only provided at the end of dialogues. To address this issue, reward learning has been introduced to learn from state-action pairs of an optimal policy to provide turn-by-turn rewards. This approach requires complete state-action annotations of human-to-human dialogues (i.e., expert demonstrations), which is labor intensive. To overcome this limitation, we propose a novel reward learning approach for semi-supervised policy learning. The proposed approach learns a dynamics model as the reward function which models dialogue progress (i.e., state-action sequences) based on expert demonstrations, either with or without annotations. The dynamics model computes rewards by predicting whether the dialogue progress is consistent with expert demonstrations. We further propose to learn action embeddings for a better generalization of the reward function. The proposed approach outperforms competitive policy learning baselines on MultiWOZ, a benchmark multi-domain dataset.

## introduction task-oriented dialogue systems complete tasks for users, such as making a restaurant reservation or finding attractions to visit, in multi-turn dialogues @xcite @xcite . dialogue policy is a critical component in both the conventional pipeline approach @xcite and recent end-to-end approaches @xcite . it decides the next action that a dialogue system should take at each turn. considering its nature of sequential decision making, dialogue policy is usually learned via reinforcement learning @xcite ., * rui zhang is the corresponding author. 2015; @xcite . specifically, dialogue policy is learned by maximizing accumulated rewards over interactions with an environment (i.e., actual users or a user simulator). handcrafted rewards are commonly used for policy learning in earlier work @xcite , which assigns a small negative penalty at each turn and a large positive/negative reward when the task is successful/failed. however, such reward setting does not provide sufficient supervision signals in each turn other than the last turn, which causes the sparse reward issues and may result in poorly learned policies @xcite . to address this problem, reward function learning that relies on expert demonstrations has been introduced @xcite . specifically, state-action sequences generated by an optimal policy (i.e., expert demonstrations) are collected, and a reward function is learned to give high rewards to state-action pairs that better resemble the behaviors of the optimal policy. in this way, turn-by-turn rewards estimated by the reward function can be provided to learn dialogue policy. obtaining expert demonstrations is critical to reward function learning. since it is impractical to assume that an optimal policy is always available, a common and reasonable approach is to treat the decision makings in human-human dialogues as optimal behaviors. to accommodate the learning of reward function, human-human dialogues need to be annotated in the form of state-action pairs from textual utterances. to address the above issues, we propose to learn dialogue policies in a semi-supervised setting where the system action of expert demonstrations only need to be partially annotated. we propose to use an implicitly trained stochastic dynamics model as the reward function to replace the conventional reward function that is restricted to stateaction pairs. dynamics models describe sequential progress using a combination of stochastic and deterministic states in a latent space, which promotes an effective tracking and forecasting @xcite @xcite . in our scenario, we train the dynamics model to describe dialogue progress of expert demonstrations. the main rationale is that the reward function should give high rewards to actions that lead to dialogue progress similar to those in expert demonstrations. this is because dialogue progress at the early stage highly influences subsequent progress, and the latter directly determines whether the task can be completed. since the learning of dynamics model maps observations to latent states and further reason over the latent states, we are no longer restricted to fully annotated dialogues. using dynamics model as reward function also promotes a more stable policy learning. learning the dynamics model in the text space is, however, prone to compounding errors due to complexities and diversities of languages. we tackle this challenge by learning the dynamics model in an action embedding space that encodes the effect of system utterances on dialogue progress. we achieve action embedding learning by incorporating an embedding function into a generative models framework for semi-supervised learning @xcite . we observe that system utterances with comparable effects on dialogue progress will lead to similar state transitions @xcite . therefore, we formulate the generative model to describe the state transition process. using the generative model, we enrich the expert dialogues (either fully or partially annotated) with action embedding to learn the dynamics model. moreover, we also consider the scenarios where both state and action annotations are absent in most expert dialogues, referred to as unlabeled dialogues. to expand the proposed approach to such scenarios, we further propose to model dialogue progress using action sequences and reformulate the generative model accordingly. our contributions are summarized as follows: ## preliminaries for task-oriented dialogues, a dialogue policy π(a|s) decides an action @xmath0, where a and s are the predefined sets of all actions and states, respectively. reinforcement learning is commonly applied to dialogue policy learning, where the dialogue policy model is trained to maximize accumulative rewards through interactions with environments (i.e., users): ## experiments to show the effectiveness of the proposed model (denoted as act-vrnn), we experiment on a multi-domain dialogue environment under semisupervised setting (sec. 4.1). we compare against state-of-the-art approaches, and their variants enhanced by semi-supervised learning techniques (sec. 4.2). we analyze the effectiveness of action learning and reward estimation of act-vrnn under different supervision ratios (sec. 4.3). ## related work reward learning aims to provide more effective and sufficient supervision signals for dialogue policy. early studies focus on learning reward function utilizing external evaluations, e.g., user experience feedbacks @xcite , objective ratings @xcite , or a combination of multiple evaluations @xcite . these approaches often assume a human-in-the-loop setting where interactions with real users are available during training, which is expensive and difficult to scale. as more largescale high-quality dialogue corpus become available (e.g., multiwoz @xcite ), recent years have seen a growing interest in learning reward function from expert demonstrations. most recent approaches apply inverse reinforcement learning techniques for dialogue policy learning @xcite . these all require a complete state-action annotation for expert demonstrations. we aim to overcome this limitation in this study. semi-supervised learning aims to utilize unlabeled data to boost model performance, and is studied in computer vision @xcite , item ranking @xcite , and multi-label classification @xcite @xcite . many studies apply semi-supervised vae @xcite for different classification tasks, e.g., sentiment analysis @xcite ), text matching @xcite . while these work focus on prediction accuracies, we aim to enrich expert demonstrations via semi-supervised learning. ## conclusions we study the problem of semi-supervised policy learning and propose act-vrnn to provide more effective and stable rewards estimations. we formulate a generative model to jointly infer action labels and learn action embeddings. we design a novel reward function to first model dialogue progress, and estimate action rewards by determining whether the action leads to similar progress as expert dialogues. the experimental results confirm that act-vrnn achieves better task completion compared with the state-of-the-art in two settings that consider partially labeled or unlabeled dialogues. for future work, we will explore the scenarios that annotations are absent for all expert dialogues.