Context-Aware Language Modeling for Goal-Oriented Dialogue Systems (2022)

Abstract: Goal-oriented dialogue systems face a trade-off between fluent language generation and task-specific control. While supervised learning with large language models is capable of producing realistic text, how to steer such responses towards completing a specific task without sacrificing language quality remains an open question. In this work, we formulate goal-oriented dialogue as a partially observed Markov decision process, interpreting the language model as a representation of both the dynamics and the policy. This view allows us to extend techniques from learning-based control, such as task relabeling, to derive a simple and effective method to finetune language models in a goal-aware way, leading to significantly improved task performance. We additionally introduce a number of training strategies that serve to better focus the model on the task at hand. We evaluate our method, Context-Aware Language Models (CALM), on a practical flight-booking task using AirDialogue. Empirically, CALM outperforms the state-of-the-art method by 7% in terms of task success, matching human-level task performance.

## introduction dialogue systems have typically approached the problem of generating realistic dialogue from the perspective of supervised learning @xcite @xcite @xcite @xcite . however, dialogue can also be viewed as a sequential decision making process, which is well-suited to planning and reinforcement learning (rl) algorithms. a challenge with the classical rl approach to dialogue is the requirement for active interaction with humans @xcite . training such a system with active human-in-the-loop interaction quickly becomes expensive and cumbersome, making it desirable to develop techniques for goal-directed training of dialogue systems that can effectively leverage offline data. while many dialogue generation techniques based on rl and learned control have been proposed @xcite @xcite @xcite @xcite , most such systems take a pipelined approach, where an abstract representation of states and actions is designed by hand and then combined with rl to train a "dialogue management" system, rather than generating dialogue end-to-end. these pipelined approaches rely on a manually designed decomposition of the dialogue task, which may be domain-specific and, more importantly, may not enjoy all of the benefits of tightly integrating lowlevel text generation with the overall goals of the task. in this work, we instead ask: how can we scalably and effectively introduce the mechanisms of goal-directed decision making into end-to-end language models, to directly steer language generation toward completing specific dialogue tasks rather than simply generating probable responses? to this end, rather than utilizing a pipelined approach, we aim to directly finetune language models in a task-aware manner such that they can maximize a given utility function. we observe that large language models can already be formulated within a markov decision processes (mdp) as capturing both the dynamics and policy for a decision-making task, where dialogue history serves as state, and the agent's utterances serve as actions. we could utilize this observation by finetuning the models directly with online rl, but the need for human-inthe-loop training makes this difficult. offline rl methods @xcite @xcite provide an alternative approach, but typically require value function estimation, which is not straightforward to perform with a language model. instead, we propose a conditional imitation learning strategy coupled with a novel task relabeling approach that can finetune language models from offline data, such that the model still represents the joint distribution over dialogues, but tilts this distribution toward dialogues with a high reward. this amounts to a task-aware finetuning strategy that integrates task information into the model. the main contribution of our work is calm (context-aware language modeling), a framework for end-to-end goal-directed dialogue generation. calm unifies the traditional language modeling objective with task-specific supervision, where a language model is interpreted as a joint representation of dynamics and policies in an mdp, and the finetuning process utilizes a conditional imitation learning objective with a novel task relabeling strategy that teaches the model how to generate high-utility dialogues (see figures 1 and 2 ). because calm interprets the language model as both a dynamics model and a policy, it can be used as either a model-free method, where the dynamics are discarded and the policy component is used to greedily generate responses, or as a model-based method, where the dynamics component can be used to plan at test-time. we empirically evaluate calm on airdialogue @xcite , the largest dataset for goal-oriented dialogue based-on a flight-booking task. calm improves the task success by 10% over the previous state-of-the-art method @xcite following the evaluation protocol proposed by @xcite , achieving the first-ever human-level performance on this dataset. ## related work our goal is to enable end-to-end training of goaldirected dialogue agents. in these settings, an agent aims to complete a particular task with its utterances @xcite . goal-directed agents have been explored in contexts such as personal assistants @xcite @xcite , recommendation systems @xcite , education @xcite , and negotiation @xcite . while there are multiple approaches to constructing dialogue agents, in this work we frame the problem of generating dialogue as a sequential decision making problem within a (partially observed) markov decision process (mdp) @xcite . prior works that utilize such an mdp formulation typically aim to train a dialogue management system @xcite , in which the agent reasons about higher-level abstractions of the state of the conversation, and language generation is performed using a downstream procedure. dialogue management systems have been trained using techniques such as online reinforcement learning via policy gradients @xcite , off-policy reinforcement learning @xcite or actor-critic methods @xcite . our method differs from dialogue management systems in that calm is an end-to-end system optimized for successful task completion, and performs both high-level decision making and language generation. recent advancements in language models, such as recurrent neural networks @xcite @xcite @xcite and attention-based architectures @xcite @xcite , have spurred increasing interest in such endto-end dialogue systems @xcite @xcite . modelbased approaches, in which a learned agent is substituted for a human, allow learning to be done entirely within simulation without human intervention @xcite @xcite @xcite . in con- bad task context good task context reward 0 hello. how can i help you? could you help me in booking a flight ticket from aus to ewr? sure, flight 1001 meets your needs, shall i book it? reward 1 re-label context to maximize reward 2) context aware fine-tuning 1) dialogue task relabeling imperfect dialogue dataset ~hello. how can i help you? could you help me in booking a flight ticket from aus to ewr? language modeling loss task specific auxiliary loss re-labeled dialogue dataset ~sure, flight 1001 meets your needs, shall i book it? 20% 30% 50% figure 2 : a visual outline of calm. we apply task relabeling to our static offline dataset, by swapping out the task context -in this case a flight table -such that the attached dialogue becomes an optimal example of task completion. when fine-tuning on this relabeled dataset, we then apply a task specific auxiliary loss on top of the standard language modeling objective; this helps the model learn to use the task context. once trained, calm can consistently solve goal-directed dialogue tasks. ## preliminaries in this section, we review our notation and problem formulation for casting dialogue within a sequential decision making framework. pomdp formulation. we formulate dialogue generation as a partially observable markov decision process (pomdp) @xcite , with a state that consists of known and unknown context information about the task. let c @xmath0, and let c @xmath1. for instance, in a flight booking task, a table of available flights might correspond to c o , while the particular flight that the human wants to book, which is unknown to the agent, corresponds to c h . note that the reward, which requires booking the right flight, depends on both hidden and observed contexts. we can define such an environment as a pomdp @xmath2, a, o, t , z, µ 0 , r, γ). we denote a conversation τ as τ := {a 0 , e 0 , ..., a t }, where t denotes the number of turns in a conversation and a t and e t represent utterances (strings of tokens) from the dialogue agent (a t ) and the human (e t ) at the t-th turn, respectively. we additionally use τ <t to denote conversation history up to the t-th turn. we can represent the underlying pomdp state s @xmath3:= {c h , c o , τ <t } = {c h , c o , a 0 , e 0 , ..., a t-1 , e t-1 }. however, we only observe the last two elements of the state tuple, such that our observation o @xmath4, τ <t }. an action a @xmath5. given our definition of the state, the full conversation in a dialogue can be conveniently represented by the last observation and action, {o t , a t }. an agent π : @xmath6. a transition function t (•|s t , a t ), represents a distribution over the human's utterances, returning s t+1 as the state at turn t + 1. we only consider the sparse reward setting with r @xmath7, a t ) ∈ {0, 1} denoting task completion, and r @xmath8, ∀@xmath9. our final reward is therefore dependent on both the context and the dialogue: r(s t , a t ) = r(τ, c h , c o ), where the context {c o , c h } is randomly sampled for each dialogue from some initial distribution µ 0 . goal-oriented dialogue. goal-oriented dialogue systems aim to maximize the expected reward of the above pomdp e {co,c h }∼µ 0 ,π,t [ t @xmath10, a t )], (1) where {c h , c o } is sampled from distribution µ 0 . onpolicy rl algorithms optimize this objective via environment interaction, which is represented by a real human. however, because human-in-the-loop training is expensive, we pursue an offline learning approach where we are given a fixed dataset and there is no further interaction with the human in the learning process. this dataset is composed of n trajectories with t } and its corresponding final reward for task completion r (i) . our goal is to learn the policy π(a|o) which improves the dialog agent's ability in achieving the highest task reward defined in equation 1. language models. while conventionally a language model is seen simply as a sequence model over tokens of the form t @xmath11:t ), when the sequence x 1:t corresponds to a dialogue trajectory τ , we can also interpret a language model as learning the distribution over τ . this distribution can be factored into the product of the policy π(a t |τ <t ) and the dynamics t (τ <t+1 |τ <t , a t ), and so we can say that a language model also represents the policy and the dynamics. therefore, the maximum likelihood objective for training or finetuning a language model on a dialogue dataset d off consisting of dialogue trajectories τ can be written as a naïve approach to train dialogue systems is to jointly parameterize both π and t as one language model, and optimize equation 2 on pre-collected conversations d off . this method corresponds to behavioral cloning (bc) @xcite . ## context-aware language modeling in this section, we present our method for goaloriented dialogue systems, context-aware language modeling (calm). calm interprets a language model as a combination of a policy and a dynamics model in the pomdp formulation of a dialogue task, as described in section 3. under this interpretation, naïve supervised finetuning on the dialogue dataset can be viewed as behavioral cloning (bc) @xcite . however, bc only imitates data and does not necessarily produce a good policy in terms of completing tasks. we propose to improve the policy by utilizing a task relabeling strategy (described in section 4.1), analogous to prior task relabeling approaches @xcite @xcite @xcite @xcite . this relabeling procedure augments the data with examples of near-optimal utterances, making the language model more task-aware. however, we find several shortcomings with this approach alone and propose the following improvements. first, an expressive language model is liable to ignore the task context, which we address by proposing an auxiliary loss (section 4.2) that forces the model to utilize this information. second, learning from structured task information is difficult and can result in models that fail to capture complex task structure, so we propose a task pre-training procedure to improve the learnability (section 4.3). finally, to further improve performance we use a model-based planning procedure (section 4.4) on top of the proposed method that samples multiple dialogues in parallel and selects the most promising candidates. ## calm for airdialogue in this section, we instantiate our proposed method, calm, for the airdialogue flight booking task @xcite . we first give an overview of the task, and then describe how to do relabeling and context conditioning on this specific task. ## experiments in this section, we empirically evaluate the performance of calm on airdialogue @xcite . we first show that calm outperforms the sota on the airdialogue dataset by around 7% in the standard simulated evaluation protocol proposed by @xcite , which prior work denotes as "self-play" (see appendix e), and this matches human-level performance as reported by @xcite . beyond this, we also perform a comprehensive set of ablation studies to validate the necessity of each component of calm. experiment setup and baselines. we compare calm on airdialogue with two baselines. the first is airconcierge, the previous sota on air-dialogue, which explicitly parses and executes sql queries from the dialogue @xcite . the other is a standard language model (denoted as lm(gpt2-small)) trained on a dataset filtered for successful task examples, without any of our context-aware language modeling techniques (see appendix section a for more details on dataset filtering). calm uses the fine-tuned gpt2-small model @xcite as the backbone of the policy and dynamics model. after learning the dynamics model, both calm and the lm(gpt2small) can employ two different planning strategies: (1) a simple greedy decoding of the next utterance (equivalent to beam search with beam-width one) and (2) the rollout planning as described in section 4.4. for airconcierge, we only evaluate greedy decoding, as this method cannot be easily adapted for producing full rollouts. rollout planning requires a method for predicting the reward of a given dialogue, and we describe our specific reward predictor for airdialogue in appendix section e.1. results for task success. in terms of task success, calm outperforms the prior sota (airconcierge) by approximately 7%, achieving 88% task success when using greedy decoding from the language model (see removing any single component from calm drops performance by at least 10%. ## conclusion we proposed an end-to-end framework, calm, for goal-oriented dialogue systems. formulating end-to-end dialogue generation as a markov decision process, calm employs task relabeling and context-aware finetuning to steer supervised learning of language models towards specific goals, improving task performance drastically while preserving language quality. we show that this improves performance on airdialogue over the previous state of the art, and matches previously reported human performance under the standard simulated evaluation protocol. calm optimizes for task-specific measures of success, and while such measures might be comparatively simple for domains such as airdialogue, in general specifying the right success measure or reward function may present challenges. furthermore, as with all methods based on end-to-end language models, calm is susceptible to internal biases and inconsistencies in the language model itself. there is for example no constraint that ensures that calm produces truthful answers, or that it avoids harmful or socially unacceptable outputs. a practical deployable dialogue system would likely require additional measures to account for such issues, analogously to how learning-based methods for self-driving vehicles might require some additional safety mechanisms to ensure constraints, and indeed further research on reward specification, ensuring truthful outputs, and other constraint strategies for dialogue systems that combine language models and reward maximization is a promising and important direction. the context-conditioned supervised learning strategy used by calm provides for reward maximization, but is in general not optimal for arbitrary reinforcement learning problems: in general rl settings, learning a value function with dynamic programming in general can attain significantly better returns than imitating high-performing trajectories, by recombining good parts of multiple different trajectories (which might individually be suboptimal) @xcite . the simple supervised learning strategy works well in the domain we tested, but extending calm to use value-based reinforcement learning methods is a promising direction for future work. indeed, the improvement obtained from planning on top of the calm model likely indicates that the supervised learning approach we employ has room for improvement. additionally, the auxiliary objectives and relabeling strategies we employ require some amount of domain-specific design, and more general strategies could be developed in future. addressing these limitations in future work and developing more advanced methods that combine end-to-end language generation via large language models with concepts from reinforcement learning and planning is a promising research direction for making dialogue systems more capable, while also making language models more task aware. we hope that calm will serve as an indication for the potential of such methods.