KRLS : Improving End-to-End Response Generation in Task Oriented Dialog with Reinforced Keywords Learning (2023)

Abstract: In task-oriented dialogs (TOD), reinforcement learning (RL) algorithms train a model to directly optimize response for task-related metrics. However, RL often needs to perform exploration, which can be time-consuming due to the slow auto-regressive sequence generation process. We investigate an approach to create a more efficient RL-based algorithm to improve TOD performance in an offline setting. First, we use a faster generation procedure that samples from independent next-word distributions after training the language model (LM) with supervised learning. We then introduce a fine-grained reward function to help the model focus on learning key information in a dialog, by measuring the importance and semantic closeness of each generated token. Experiments on the MultiWoZ dataset show our new training algorithm, Keywords Reinforcement Learning with Next-word Sampling (KRLS), achieves state-of-the-art performance on the end-to-end response generation task, with a 15% training time reduction compared to a standard RL algorithm using auto-regressive generation.

## introduction task-oriented dialog systems help users complete pre-defined tasks such as booking a hotel or reserving a table in a restaurant. with advances in large-scale pre-trained generative models @xcite @xcite , many recent approaches @xcite @xcite @xcite handle tod as a holistic task of end-to-end (e2e) generation, as opposed to the traditional modular approach. in this e2e setting, the dialog history is often used as input, and reinforcement learning (rl) algorithms can train a model to generate a response that directly optimizes for task-related metrics, such as the task success rate (budzianowski in this work, we aim to create a more efficient rl procedure for tod, which does not need a user simulator nor use auto-regressive generation during exploration. we propose a new training algorithm, keywords reinforcement learning with next-word sampling (krls), which combines a faster sequence generation procedure and a fine-grained per-token reward in an offline setting @xcite . first, we treat each turn in a dialog as a separate episode and consider an offline rl task to generate a response only at the end of each episode (see figure 1 ). since this procedure only explores/generates system responses at the last turn, no interactive environment (e.g., a user simulator) is needed. during this rl process, krls generates new sequences by directly sampling from independent next-word reward reward decoder [value_car] [pad] booked car type : gold [value_car] booked car type : finetuned decoder [value_car] [pad] booked car type : gold plane booked plane type : sampled reward reward reward nll loss nll loss nll loss nll loss nll loss figure 2 : overview of the krls algorithm. during traditional supervised training, the language model learns/imitates the gold response. during offline rl training, the fine-tuned model generates sequences by sampling from next-word distributions conditioned on the gold response and receives a per-token reward. ## background to introduce rl in nlp tasks, we begin by formulating the response generation process as an mdp. given a supervised dataset @xmath0, x i )} where (i) is the context and x (i) is a response, the probability of generating x (i) can be modeled as: t -1 } in an episode, where a (i) @xmath1. the reward function r would represent the "utility" of each action contributing towards the overall performance, such as task success in tod. typically, this is modeled by using r(s, a) = 0 for non-terminal states, and r(s t , a) for terminal states which can be computed by combining scores such as task success and bleu @xcite . the transition function p : s × @xmath2, . . . , c m , a 0 , . . . , a t ). finally, γ ∈ [0, 1) is the discount factor. ## approach in the mulitiwoz dataset @xcite @xcite , we observe that key information, such as restaurant "phone number" and "address", needs to be generated correctly in a response to achieve a high task success/inform rate. however, traditional sl uses a negative log-likelihood loss, which asks the ## related work end-to-end dialog systems such as @xcite ; @xcite ; @xcite ; @xcite have shown promising results in tod benchmarks such as multiwoz. however, as the standard sl objective does not directly account for tod metrics such as task success rate, much recent work seeks to incorporate rl techniques to improve tod performance. in this section, we discuss related applications of rl in tod, as well as other non-rlbased approaches that have similarities in concept. rl for text generation ranzato et al. ( 2016 2022 ) applies rl to text generation tasks by treating each word as an action and then uses auto-regressive generation to explore high-reward sequences. this results in a large action space for exploration, and these work focuses on methods to stabilize the training process. in principle, these approaches can be modified for tod tasks, but they would still generally use a user simulator and/or the slow auto-regressive generation step. rl for policy planning many direct applications of rl in tod focus on optimizing dialog policy planning @xcite @xcite . @xcite ; @xcite jointly optimize both a user system and a dialog system to improve a model's tod task performance and/or domain adaptation ability, but could be resource intensive as additional userside training is needed. alternatively, @xcite ; @xcite uses rl to optimize system action generation in a latent space, but tends to result in the model generating very short responses (i.e., a low bleu score of 10.8 in multiwoz). offline rl in tod many offline rl applications in tod consider an actor-critic type algorithm @xcite , which involves using a critic to choose better responses among several generated candidates. these approaches tend to be vulnerable to errors made by the critic model (especially for ood actions @xcite ), and is resource intensive as multiple auto-regressive generations are needed per episode. alternatively, pang and he (2021) proposes the gold algorithm, which uses policygradient based method in an off-policy setting, by learning solely from the gold demonstrations without any generation/exploration. krls additionally performs sequence generations and utilizes gold demonstrations in computing the reward function (also see section 5.1 for an empirical comparison). other notable related techniques qian et al. (2021) utilizes a student-teacher architecture and maml @xcite to improve domain adaptation ability of the student model. specifically, the teacher model provides weights to scale the nll loss of each gold token when training the student model. in this aspect, this is similar to gold, performing a "weighted learning" on the gold demonstrations. krls aims to directly improve tod performance and achieves this by utilizing rl to perform a "weighted learning" on generated tokens. ## conclusion in this work, we explore an approach to utilize rl to improve a model's tod performance, but also to avoid using a user-simulator or the slow autoregressive generation process. we propose the keywords reinforcement with next-word sampling (krls) training algorithm, which combines offline rl with a fast sequence generation scheme that directly samples from next-word distributions after supervised training, and a fine-grained per-token reward function that measures the importance and semantic closeness of each generated token. we then evaluate krls on the multiwoz dataset and show that a) it can help improve e2e response generation performance, reaching new state-of-the-art in the inform rate, success rate, and combined score; b) it can be trained 15% faster than using a standard rl algorithm that performs auto-regressive generation during training/exploration.