W ea S u L : Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue (2021)

Abstract: An intelligent dialogue system in a multi-turn setting should not only generate the responses which are of good quality, but it should also generate the responses which can lead to long-term success of the dialogue. Although, the current approaches improved the response quality, but they over-look the training signals present in the dialogue data. We can leverage these signals to generate the weakly supervised training data for learning dialog policy and reward estimator, and make the policy take actions (generates responses) which can foresee the future direction for a successful (rewarding) conversation. We simulate the dialogue between an agent and a user (modelled similar to an agent with supervised learning objective) to interact with each other. The agent uses dynamic blocking to generate ranked diverse responses and exploration-exploitation to select among the Top-K responses. Each simulated state-action pair is evaluated (works as a weak annotation) with three quality modules: Semantic Relevant, Semantic Coherence and Consistent Flow. Empirical studies with two benchmarks indicate that our model can significantly out-perform the response quality and lead to a successful conversation on both automatic evaluation and human judgment.

## introduction dialog policy for multi-turn dialogue decides the next best action to take on the environment so as to complete the conversation based on various success criteria. reinforcement learning can help to learn such a policy where the environment can be users (human or model) and the policy takes action on the environment from which it gets a reward signal @xcite @xcite @xcite @xcite . 1 work done prior to joining amazon learning a dialogue policy using reinforcement learning can be challenging with humans users, since it requires a large set of samples with a reward to train. since there are a lot of previous works on neural response generation @xcite @xcite @xcite we can model the users also, using any of these encoder-decoder architectures. this helps to simulate the conversations between the simulated user and the agent (policy model) replying to each other @xcite @xcite . reward signal for policy learning can be as simple as the small constant negative reward at each turn and a large reward at the end (if the goal completes) to encourage shorter conversations @xcite . however, reward estimation for dialogue is challenging, the small constant negative reward at each turn may lead to ending the conversation prematurely. instead of handcrafting the reward at the end based on success or failure, it is more useful if we can evaluate reward at every turn to guide the policy to dynamically change actions as per the need for the user and end the conversation naturally. with the growing complexity of the system across different topics, it is required to build a more sophisticated reward function to avoid manual intervention for accounting different factors towards conversation success. in this work, we proposed a novel model for contextual response generation in multi-turn dialogue. the model includes the turn-level reward estimator, which combines the weak supervision signals obtained from three basic modules 1) semantic coherence, 2) consistent flow, 3) semantic relevance. these modules are learned jointly with the response generation model with the counterfactual examples obtained from negative sampling. leveraging the weak supervision signals obtained from these models, we further update the reward estimator and dialog policy jointly in an alternative way, thus improving each other. our proposed approach integrates semantic understanding of utterances using encoder-decoder systems with the power of reinforcement learning (rl) to optimize long-term success. we test the proposed approach with two benchmarks: daily-dialog @xcite and personachat @xcite . experimental results demonstrate on both datasets indicate that our model can significantly outperform state-of-the-art generation models in terms of both automatic evaluation and human judgment. ## related work open-domain dialogue in a multi-turn setting has been widely explored with different encoderdecoder architectures @xcite @xcite @xcite @xcite @xcite . the basic encoder-decoder architectures like seq-to-seq models have been widely extended and modified to generate the generic responses, context modelling and grounding by persona/emotion/knowledge @xcite @xcite @xcite @xcite . the dialogue literature widely applies reinforcement learning, including the recent ones based on deep architectures @xcite @xcite @xcite . but these taskoriented rl dialogue systems often model the dialogue with limited parameters and assumptions specific to the dataset, targeted for that task. the dataset includes hand-built templates with state, action and reward signals designed by humans for each new domain making this setting difficult for extending these to open domain dialogue systems. our goal in this work is to integrate the stateof-the-art encoder-decoder architectures like in @xcite ; @xcite ; @xcite and reinforcement learning paradigms to efficiently learn the dialogue policy optimized for long-term success in the multi-turn dialogue scenarios. we are recently inspired by the works in @xcite ; @xcite to jointly learn the reward function and dialogue policy, and reduce the effort and cost for manual labelling the conversations for building the reward model. specifically, we leverage the weak supervision inspired from @xcite to generate the labelled dataset to facilitate this joint learning and building reward estimation model. ## approach we represent dialog sessions @xmath0, τ 2 , τ 3 , .......τ n } where each dialog session τ represents the trajectory of state-action pairs as {s u 0 , a u 0 , s 0 , a 0 , s u 1 , a u 1 , s 1 , a 1 , .....}. the user in our case is a simulator which utters a response a u given the state s u denoted as µ(a u , e u |s u ) where e u denotes the binary signal indicating the end of a dialog session, in that case the response a u is empty. the dialog policy π θ (a|s) decides the action a according to the current state s after the agent interacts with the user simulator µ. at each time, the state given to the either dialog party is updated after recording the action uttered by the other party. the reward estimator f evaluates the quality of response/action uttered by the dialog policy π. the dialog policy π is based on the bert @xcite encoder-decoder model and the reward function f is the mlp model parameterized by θ and ω respectively. we have modeled the user simulator exactly in the same way as the agent but trained only using supervised learning objective. in the subsequent section, we will introduce the components action, state, policy, quality modules and reward estimator. further, sections explain the setup we have used for weakly supervised learning and, finally, the experimental results. ## experiments we conduct experiments on dailydialog @xcite , personachat @xcite and used gutenberg dialogue dataset @xcite as a pre-training corpus. we compare our model performance with baselines on various aspects of response quality. ## conclusions we proposed a weak supervision framework for policy and reward estimation for long-term success of the dialogue by simulating the conversation between a virtual agent and user. empirical studies on two benchmarks proves the effectiveness of our approach.