CHAI : A CH atbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning (2022)

Abstract: Conventionally, generation of natural language for dialogue agents may be viewed as a statistical learning problem: determine the patterns in human-provided data and generate appropriate responses with similar statistical properties. However, dialogue can also be regarded as a goal directed process, where speakers attempt to accomplish a specific task. Reinforcement learning (RL) algorithms are designed specifically for solving such goal-directed problems, but the most direct way to apply RL, through trial-and-error learning in human conversations, is costly. In this paper, we study how offline reinforcement learning can instead be used to train dialogue agents entirely using static datasets collected from human speakers. Our experiments show that recently developed offline RL methods can be combined with language models to yield realistic dialogue agents that better accomplish task goals.

## introduction constructing fluent and intelligent dialogue agents could pave the way for intuitive interfaces and automation of human-interactive tasks. however, this requires dialogue agents that both generate fluent, natural responses and effectively pursue the goals of the given dialogue task. a predominant approach to training dialogue agents is through supervised learning, where an agent is tasked with imitating language provided by humans. while this can provide for fluent responses, it becomes difficult to ensure that such agents systematically pursue the goals of the dialogue interaction. if we instead view dialogue as a control problem, frameworks such as reinforcement learning (rl) could allow agents to automatically optimize dialogue with respect to a task goal through a trial-and-error process and improve over human behavior. however, implementing an rl system in practice, where an agent learns online from interacting with real humans, can be prohibitively expen-sive and time-consuming. this is in stark contrast to supervised learning approaches, where we can cheaply construct datasets for training imitation agents by simply logging conversations. therefore, existing rl approaches for dialogue often rely on interacting with a learned model of a human @xcite , from which experience can be generated inexpensively. however, naïve training in this manner can result in the dialogue agent exploiting the model, which can degenerate into non-intelligible language. to mitigate this, algorithms must typically enforce strong priors to keep generated language similar to those seen in the dataset @xcite , or adopt dialogue management and template-based approaches which directly re-use language seen in the dataset @xcite . issues such as model exploitation and distribution shift when training on static datasets are a primary concern of offline rl @xcite , and can provide a formalized approach to tackling these problems. while offline rl is motivated by scaling rl to large datasets, annotated datasets for dialogue are still small compared to the large amount of raw text datasets available today. therefore, we propose an offline, model-free approach to dialogue generation that leverages language models. because the size of unlabeled language datasets dwarfs that of curated datasets for dialogue, using a pre-trained language model as a central component of our method allows it to learn aspects of language fluency from unlabeled datasets, while learning higher-level strategies for goal-directed dialogue via rl on a smaller annotated datasets. this combined approach enables us to utilize the large amounts of existing language data that standard rl methods cannot. the main contribution of this work is chai (chatbot ai), an algorithm for learning taskoriented dialogue that utilizes a language model in conjunction with offline rl. we show that this leads the policy to generate goal-oriented dialogue that is both realistic and functional, and does not require training against a simulated model of human language. we evaluate our method on a negotiation task, which requires the model to both reason about strategic aspects of conversation along with generating fluent language. we show that chai consistently bargains for better prices and with higher rates of successful negotiation than prior rl approaches to goal-oriented dialogue. ## related work recent developments in deep learning have led to end-to-end approaches to dialogue using supervised learning, such as sequence-to-sequence models @xcite , hierarchical models @xcite , attention @xcite , and transformer-based models @xcite @xcite . however, supervised learning only allows an agent to imitate behaviors, requires optimal data, and does not allow agents to exceed human performance. supervised learning for dialogue generation also has well-known issues such as outputting commonplace responses (e.g., i do not know) regardless of the inputs @xcite . therefore, additional training of the dialogue agent is required for performing goal-oriented tasks. task-oriented dialogue has been formulated as a sequential decision making problem in a markov decision process (mdp) since the 1990s @xcite @xcite @xcite @xcite @xcite @xcite . dialogue is converted into abstract states and actions from which an agent is trained using reinforcement learning (rl) @xcite @xcite @xcite @xcite @xcite @xcite @xcite . these methods differ in how the abstract states/actions are designed and whether the simulated environment for training the policy is hand created, learned as a fixed model, or is an agent itself. for instance, @xcite ; @xcite learn a fixed transition model from human conversations @xcite learn negotiation agents where each agent is the user simulator for the other agent. these methods also differ in how the decision making policy is trained, e.g., online @xcite or offpolicy/offline @xcite using actor-critic @xcite , policy gradient @xcite , or fitted q-iteration @xcite . regardless of the rl method used, since policies are trained on abstract states and actions, these methods lack the ability to generate natural language (i.e., response is created via templates depending on an abstract action). to overcome these limitations, recent work has trained policies directly on text, using a recurrent neural network to output language tokens, and using self-play for policy training while interacting with another learned agent @xcite @xcite . to further improve the generated language quality, hierarchical methods decouple the strategic high-level dialogue decisions from generation @xcite @xcite . these model-based approaches require accurate estimation of the environment/human (e.g., the trained self-play agent needs to mimic complex human behavior), which is beyond current capability of model-based reinforcement learning algorithms. similar to our proposal, @xcite use offline rl based on kl-control for text generation in open-domain dialogue. our work differs in that our model is able to utilize large amounts of unsupervised data through the use of pre-trained language models, and that our work focuses on task-oriented (as opposed to opendomain) dialogue tasks. goal-oriented tasks have clearly defined objectives that can be quantified, allowing us to provide an objective comparison between our method and prior approaches. ## preliminaries in this section, we describe our evaluation task and cover the necessary background and notation. ## offline reinforcement learning with language models offline rl potentially allows reinforcement learning methods to leverage large datasets for policy learning. however, it still requires datasets to be annotated with rewards, and for conversations to come from the task at hand. because of this, annotated dialogue datasets, such as the craigslistbargain dataset presented in section 3.1, are many orders of magnitude smaller than unlabeled datasets collected for unsupervised and language modeling tasks. in order to utilize these large unlabeled datasets, we propose an algorithm that combines offline rl with fine-tuned language models. our approach begins with training a language model, such as gpt-2 @xcite , and fine-tuning it on our task-specific dialogue corpus (sec. 3.1). we use lm(u|s 1:t ) to denote a distribution over utterances u produced by the language model given the dialogue history, denoted s 1:t . we then train a critic or q-function as described in sec. 4.1, which is responsible for scoring good and bad responses and is used to select responses from a pool of candidates generated from the language model. our approach can be viewed as using a q-function to steer a language model (which has no concept of a task) towards producing language that accomplishes some task-specific goal. ## experiments our experimental evaluation aims to compare our proposed goal-directed offline rl dialogue method to both prior dialogue management approaches and language modeling baselines. we conduct two studies: an objective evaluation against other dialogue agents to measure each method's performance in negotiation, and a subjective human study to measure the overall end-to-end performance of the system in a similar manner to prior work @xcite . qualitative results showing actual dialogue generated from our method can be found in appendix a.4. we consider 4 baseline approaches. the first is the current state-of-the-art approach for the craigslistbargain task proposed by @xcite (referred to as the retrieval-based baseline). this is a hierarchical approach to dialogue generation that separately handles language generation and dialogue management. this method parses utterances into coarse "dialogue acts," which represent high-level categorizations of the utterance such as greetings, offers, or counter-offers. an rl agent is then trained against a learned model of the environment to select "dialogue acts", and a retrieval-based generator is then used to convert "dialogue acts" back into text. in contrast, our method directly generates text, and does not require any manually designed categorizations of natural dialogue into dialogue acts. since our method utilizes a modern language model, we also include a pure language modeling baseline, which consists of the same gpt-2 language model @xcite finetuned on the craigslistbargain dataset using the same method as done in chai. this baseline allows us to determine whether any improvement from our method is due to the language model, or to the use of offline rl. finally, we also evaluate the end-to-end approaches described by @xcite , which include a dialogue agent trained via supervised learning, and an rl agent optimized for the task objective. ## discussion and future work we presented a system for goal-directed dialogue based on combining offline rl with finetuned language models. chai learns with rl, but does so from offline datasets of human dialogue. the language model allows chai to benefit from largescale unsupervised pre-training, and the offline rl component enables chai to select responses that are more likely to lead to a successful task outcome. quantitatively, chai achieves higher acceptance rate at higher revenue than prior dialogue manage-ment systems designed for this task. goal-oriented dialogue agents have many potentially useful applications such as building personal assistants, improving accessibility to technology for the disabled or the elderly, and simply saving time by automating menial tasks. of course, as with any natural language generation technology, this kind of method can be used both beneficially and maliciously, for example by users who aim to create intentionally deceptive and realistic agents. while chai provides a proof-of-concept that offline rl can successfully learn complex humaninteractive tasks such as dialogue, it also has limitations. the goal of rl is to maximize reward, which can lead to unintended responses -for example, without additional objective terms, there is no reason for chai to be truthful. similar issues affect language models more broadly, though we anticipate that it would be easier to address such issues in rl by employing better reward design. although reward design can itself be a difficult problem, it does provide a more direct lever for influencing the agent's behavior than what is available in standard language models, which must be directed either through the choice of training data or other indirect mechanisms. in chai, we only investigated a single task due to architectural constraints, and the exact same architecture presented in this paper (e.g. a price prediction head) may not be directly transferable to other domains. however, using a value-based selection mechanism is more generally applicable to any goal-oriented task. an exciting direction for future work is to extend offline rl to address a wider range of humaninteractive tasks, particularly tasks with longerrange dependencies and delayed rewards, where complex task goals can lead to the emergence of dialogue that is ultimately more useful to human users.