Human-centric dialog training via offline reinforcement learning (2020)

Abstract: How can we train a dialog model to produce better conversations by learning from human feedback, without the risk of humans teaching it harmful chat behaviors? We start by hosting models online, and gather human feedback from real-time, open-ended conversations, which we then use to train and improve the models using offline reinforcement learning (RL). We identify implicit conversational cues including language similarity, elicitation of laughter, sentiment, and more, which indicate positive human feedback, and embed these in multiple reward functions. A well-known challenge is that learning an RL policy in an offline setting usually fails due to the lack of ability to explore and the tendency to make over-optimistic estimates of future reward. These problems become even harder when using RL for language models, which can easily have a 20,000 action vocabulary and many possible reward functions. We solve the challenge by developing a novel class of offline RL algorithms. These algorithms use KL-control to penalize divergence from a pre-trained prior language model, and use a new strategy to make the algorithm pessimistic, instead of optimistic, in the face of uncertainty. We test the resulting dialog model with ratings from 80 users in an open-domain setting and find it achieves significant improvements over existing deep offline RL approaches. The novel offline RL method is viable for improving any existing generative dialog model using a static dataset of human feedback.

## introduction training open-domain dialog models is inherently difficult, since for each utterance there are many acceptable responses, yet no perfect response. while supervised learning from conversational corpora allows models to learn grammatical structure and even topic coherence, these models do not gener-alize, since the training objectives mostly lead the models to memorize responses within the corpus. humans are the ultimate authority in evaluating what makes one conversational reply better than another. to learn from real conversations with humans, we created an interactive, online platform which hosted a diverse set of neural network dialog models that users could chat with in real time. however, when learning from human interactions in the wild it is crucial to be able to learn offline and test the policy before deploying it, lest it learn inappropriate behaviors (e.g. @xcite ). thus, we need to train and test models offline, to ensure safe model outputs. in order to safely learn to optimize human feedback we pursued an offline reinforcement learning approach to training dialog models (see figure 1 ). offline rl is challenging; most deep rl algorithms fail to learn from data that is not heavily correlated with the current policy @xcite . even models based on off-policy algorithms like q-learning fail to learn in the offline rl setting, as the model is not able to explore. if the offline dataset is not sufficient to cover the input-response space, offline rl models suffer from extrapolation error, learning arbitrarily bad estimates of the value of responses not contained in the data. we solve these problems by developing a new method for offline rl. the method starts by leveraging a pre-trained language model to constrain offline rl updates. while training with rl, we penalize divergence from this prior model using forms of kl-control. this combats extrapolation error, and ensures that the rl model learns a policy that stays close to the distribution of realistic language, while learning to maximize positive human responses using the offline data. further, we use dropout to obtain uncertainty estimates of the target q-values, and to obtain a lower bound to alleviate over-optimistic bias in estimating future reward. we show that this new method is able to learn successfully from many different reward functions, even in a very large space with 20,000 tokens. both linguistic theory (e.g. grice's maxims @xcite ) and empirical experiments correlating human judgement with language features suggest that there are many criteria that could be used to evaluate a conversational agent @xcite . we develop a set of reward functions for our dialog agents to optimize, which are designed to approximate implicit human preferences expressed during conversational responses. we show that the new method is better able to optimize these rewards using the offline data, and when tested with a new set of 80 human conversation partners, leads to more positive responses and higher quality ratings than a state-of-the-art offline deep rl method. novel contributions of this paper are: ## way off-policy rl we adapt typical rl notation to the problem of generating a conversation. here, we consider human interaction to represent the rl environment. the conversation history is the state s t of the environment at timestep t, and is composed of a series of utterances, which are composed of vocabulary tokens. the action a t that the rl model must take at each timestep is to select the most appropriate token according to its policy π(a t |s t ). once it has constructed an utterance, the response of a human to that utterance is used to compute a reward signal r t to train the model. the agent's goal is to maximize reward over a conversation trajectory τ , with a discount factor of γ applied to future rewards. q-learning methods learn an action-value estimate of the total expected discounted future reward, q π (a t , s t ) = e π [ t @xmath0, through iterative updates based on the bellman equation: q θπ (s t , a t ) = r t + γe s t+1 ∼p(•|st,at) [max in deep q-learning @xcite , a qnetwork approximates q θπ (s t , a t ) and drives the policy π. a second target q-network approximates the expected reward from the next state, q θ t (s t+1 , a t+1 ) @xcite . here, we used pre-trained language models to initialize our qand target qnetworks. ## measuring implicit conversation cues our goal is to improve a dialog model's ability to engage in natural conversation with a human by learning from the implicit signals in the human's response. requiring a human to manually rate good interactions is unnatural and cumbersome, and we hypothesize it cannot scale as effectively as recognizing and learning from informative cues within the user's text responses. the golden question is which goals should be used to train a good chit-chat dialog model. understanding when a human is satisfied with the conversation is an unsolved problem. as a first step, we designed several intrinsic conversation rewards, taking inspiration from prior work in dialog, as well as the psychology of human conversation. we noted that psychologists have identified the importance of emotion in creating a sense of understanding @xcite , laughter as important to building solidarity @xcite , paraphrasing and style matching as helping to facilitate good conversation @xcite , and asking questions as an important active listening skill @xcite . further, prior work has found that eliciting longer conversations can be a signal of engagement @xcite , and that reducing repetition and increasing specificity on the part of the model can improve conversation quality @xcite . we compute a large collection (30 in total) of bot rewards (rewards based on bot behavior e.g. asking questions), user rewards (rewards based on eliciting positive user behavior e.g. laughter), and interac-tion rewards (rewards based on similarity between the user's input and bot's response e.g. similarity to the user's response in sentence embedding space). to determine which of these rewards objectively relate to user satisfaction, we examine the reward score for those responses that received positive, negative, and neutral manual feedback using the upvote/downvote buttons provided in the interface. we found that only some of the rewards mapped accurately to user ratings (see figure 2b ), and these are the ones we optimize with our rl models. for more details about the reward functions, please see the appendix. notably, conversation length and specificity score were not found to be higher in upvoted bot responses. note that four of the rewards (starting with the bot prefix) can be optimized by the model itself, but the remaining four rewards include eliciting positive responses from a human user or measuring user-bot response similarity (e.g. using word overlap or similarity in universal sentence encoder (use) embeddings @xcite ). ## discussion in this work, we present novel techniques that enable successful offline reinforcement learning on any base language model from real human conversations. this allows the dialog systems practitioner to train models that learn language structure from vast, readily-available corpora, then fine-tune for specific desirable behaviors post-hoc through rl rewards. we observe that the new offline rl method successfully optimizes both generated bot rewards and elicited human responses. we show that it presents a better option than using regularization in training a specific bot behavior. further, rl currently remains the only option for maximizing user feedback over the course of a conversation. compared to prior work in offline rl, the novel wop offline rl algorithm achieves higher performance in traditional rl tasks, elicits more positive feedback in conversations with novel humans at test time, and earns overall higher human ratings. a limitation of our study is that the question of what to optimize with rl to improve overall qualitative ratings remains open. we have shown that manual ratings are too sparse to optimize effectively, and instead suggest using implicit rewards. however, our reward set proved insufficient to achieve higher human quality ratings, at least with the limited offline training data we were able to collect. it is unlikely the rewards proposed here fully cover what it means to have a high quality openended conversation. future work should investigate more rewards for training an open-domain dialog model such as long term conversation rewards that may need to be computed over many conversation turns. our work computes conversational rewards based on dialog data and annotations from online task workers in the united states. considering the broader impacts of our work, a representative and diverse set of conversations and annotations should be collected before real world systems are trained and deployed using our algorithms. we have shown that the proposed techniques can be useful for shaping dialog model behavior towards a desired objective. for many practical applications, we may have specific requirements for the language generated by a model-for example, that it is appropriate, positive, and polite-even if this leads to a lower perception of conversation quality for some users. we have shown that the way off-policy algorithm provides a more effective way to teach a language model specific behaviors from offline data than previously proposed rl or regularization techniques.