Long-term Control for Dialogue Generation: Methods and Evaluation (2022)

Abstract: Current approaches for controlling dialogue response generation are primarily focused on high-level attributes like style, sentiment, or topic. In this work, we focus on constrained long-term dialogue generation, which involves more fine-grained control and requires a given set of control words to appear in generated responses. This setting requires a model to not only consider the generation of these control words in the immediate context, but also produce utterances that will encourage the generation of the words at some time in the (possibly distant) future. We define the problem of constrained long-term control for dialogue generation, identify gaps in current methods for evaluation, and propose new metrics that better measure long-term control. We also propose a retrieval-augmented method that improves performance of long-term controlled generation via logit modification techniques. We show through experiments on three task-oriented dialogue datasets that our metrics better assess dialogue control relative to current alternatives and that our method outperforms state-of-the-art constrained generation baselines.

## introduction despite recent advances in dialogue systems @xcite , controlling dialogue generation remains a significant challenge. response generation in dialogue can be controlled towards different topics and styles @xcite or towards a set of hard constraints (i.e., lexical control words need to appear in the generated text) @xcite . we focus on the hard constraint setting, also known as constrained generation, as this provides a more fine-grained method of controlling dialogues. for example, consider a customer service use case (figure 1 ), in which an agent speaks to a 1 our code is available at https://github.com/a sappresearch/constrained-dialogue-genera tion customer about an issue. the goal is to generate a given set of control words in the responses of one of the speakers (agent or customer). naive constrained generation approaches @xcite use methods like beam search and stochastic search to force the generation of these control words for short-term control, where control words need to appear in a single utterance or phrase. because they do not consider the future, these approaches may generate the words all at once in a single response or not generate them at natural places in the conversation (figure 1 , left). the above example highlights the challenges of applying existing constrained generation methods to long-term dialogue generation. first, since another speaker is involved in the dialogue, the model does not have full control of the generated text. instead, the model can only control the dialogue indirectly. second, dialogues can be long and thus, controlling utterances several time steps into the future is non-trivial. in this work, we propose the problem of long-term dialogue control, where the goal is to generate a set of control words over many utterances in a dialogue, which requires appropriately timing the generation of control words (figure 1 , right). to the best of our knowledge, we are the first work to constrain long-term dialogue generation through lexical control words. we begin by highlighting challenges with evaluation for this problem. successful long-term control of dialogue can be difficult to measure. we describe current evaluation metrics for constrained text generation and show that these metrics can be gamed by generating all or many control words early in the conversation. to resolve this and measure how natural the control is, we propose a new set of metrics: long-term success rate, which measures the percentage of control words in simulated roll-outs of the conversation, and precision, recall, and f1-score, which compare control words in generated responses to those in reference responses from a historical dataset. the second set of metrics specifically help to capture whether the control words are generated at the right time. next, we propose a novel method to explicitly address long-term control. prior methods are unable to handle this task as the number of possible future sequences is exponential. to alleviate this issue, we retrieve similar conversations from training and condition on them during generation. we first identify similar neighbors using a knn-based approach and then guide the language model towards generating similar responses, inspired by plug-andplay methods @xcite @xcite . the motivation for this is that retrieved conversations guide the model to generate the control words at more natural points in the conversation. we conduct experiments on multiple taskoriented dialogue datasets and show that our method outperforms several constrained text generation baselines on automated evaluation metrics as well as human evaluation. specifically, we are able to generate 30-40% more control words on long-term success rate compared with baselines, while preserving fluency (scores @xmath0.3 out of 5), as measured by human evaluation. ## problem definition we first define the problem of long-term constrained dialogue generation. a conversation @xmath1, u 1 , s 2 , u 2 , ..., s t , u t } is defined as a list of utterances generated by two speakers: the system s that we are trying to control and the user u, which we don't have explicit control over. t denotes the total number of turns in the conversation. given the current dialogue context of a conversation @xmath2, u 1 , ..., s t , u t } up until timestep t and a set of control words @xmath3, w 2 , ..., w m }, our goal is to generate the remaining responses of the conversation r t+1:@xmath4, ..., s t } such that the control words w appear in the future generated responses. we consider a scenario in which someone provides a set of control words to be included in the conversation without assumptions on their order. this means methods need to handle control words given in any order. we additionally assume access to a historical dataset of conversations @xmath5, @xmath6, ..., n ] and a fine-tuned language model m on this dataset. we can leverage these inputs in order to control future responses r t+1:t . we focus on the plugand-play setting @xcite , in which approaches simply guide the given language model m towards generating the control words without any additional re-training. ## proposed metrics for evaluation directly evaluating the generated responses in terms of prior evaluation methods can lead to misleading results. previous works on constrained text generation @xcite have used metrics like perplexity to measure fluency and success rate to measure the percentage of control words generated. however, these metrics are more relevant for short-term generation, as they can be gamed in settings where the control words would be naturally distributed across the full conversation. as shown in the left-hand side of figure 1 , when several words are forced into the first response, the conversation may move away from the desired future and control word generation could be inappropriately timed. to better evaluate how well the model generates the right words at the right time, we propose the following new metrics. the first metric we propose is long-term success rate, which involves simulating conversations with a user language model and computing the percentage of generated control words in the system responses of these simulated roll-outs. prior work @xcite has used self-play for evaluation, but they do not propose roll-outs as a way to measure dialogue control. long-term success rate: our modified success rate metric is computed as the fraction of control words generated in a full simulated roll-out of the conversation. we compute this as: @xmath7, where n w is the number of control words that appear in all of the future system responses r t+1:t . one limitation of long-term success rate is that it doesn't measure the timing of control words in the conversation. so next, we want to evaluate whether the methods generate control words at appropriate points in the conversation. to measure this, we propose computing precision, recall, and f1-score for control words. this particular evaluation is not done in simulation. instead, we consider each true system response in the evaluation dataset in isolation and generate a response for each, given the conversation history up until that point. we compute the number of generated control words that are correctly predicted, when compared with the control words in the ground truth response in the same time step. for example, on the right side of figure 1 , when generating the second customer response (given the true conversation history up until then), we would count a "correct" prediction for p/r/f1 as a response that includes the word "shirt" (in any position in the response), as it is a control word that appears in the ground truth response in that time step. it is true that control words can also appear later in the conversation, but this setting is already evaluated by long-term success rate in simulated rollouts. after counting the number of correctly predicted control words for each response individually, we aggregate across all responses. precision: precision is calculated at the corpuslevel as the number of correctly predicted control words over the total number of predicted control words (@xmath8. recall: recall is similarly computed at the corpus-level as the number of correctly predicted control words over the total number of actual control words (@xmath9. f1-score: finally, f1-score combines precision and recall into one metric (f 1 = (2 * p * r) (p+r) ). these metrics penalize models that condense all control words into one response. instead, we want the models to naturally generate control words when they are relevant. these metrics evaluate whether control words are generated at the appropriate position in a conversation. to introduce some flexibility, an extension could be to compute a soft version of precision, recall, and f1-score that scores utterances based on whether control words appear within n utterances of the ground truth position. finally, we use human evaluation to evaluate how realistic and relevant the generated responses are. specifically, we evaluate each conversation on fluency, consistency of control word generation, relevance, coherence, and diversity. ## retrieval-based control we now present our proposed approach for constrained dialogue generation. inspired by work in retrieval-augmented generation @xcite , we retrieve similar pasts based on the current context x and use their futures to control dialogue response generation. the key insight here is that by looking at how people have used these control words in similar conversations in the past, we can bias the models towards more natural dialogues. in other words, we use futures of the past conversations to guide the current response generation. to better motivate the use of retrieval in our problem, consider the example conversation in figure 1 . the agent asks which item the customer wants to return, and there are many possible answers (e.g., "i want my pant refunded.", "i want to return gloves i bought yesterday."). keywordbased retrieval will surface a response about shirts, a control word, which encourages the model to generate a natural response with that word: "it's a nike shirt i bought a week ago." we present two variants of our retrieval-inspired futures of the past (fop) approach: 1) fopretrieval: we retrieve the desired future from historical data and simply use the retrieved utterance as the generated response and 2) fop-guided: we use the utterance from fop-retrieval as a reference sentence to guide the model towards similar responses. the simple variant of our approach, fopretrieval, is shown in figure 2 . it focuses on identifying what the model should say now that will lead to the control words in the future. the reason we need to determine what to say now is that control words in our problem are distributed across a long dialogue conversation. one possible approach to generate the current response is to run many rollouts of the conversation and select the response that leads to the highest number of control words. however, this brute force approach is computationally expensive and will not be effective for rich, diverse conversations. instead, we leverage historical conversation data to identify the most relevant futures given the current context and control words. the retrieved futures can guide the model towards what to say now that will lead to the desired future. the guided variant, shown in figure 3 , involves guiding the language model towards generating a response similar to the retrieved utterance. our proposed approaches address some of the challenges of long-term control for dialogue generation. first, another speaker can change the course of the conversation, which is why we retrieve a new set of similar past contexts at each time step to re-align with the current context. second, to control responses many steps into the future, we retrieve historical conversations with the desired future (high percentage of control words) and gently nudge the conversation in that direction, thus controlling not only the current utterance but also the future of the conversation. ## conclusion in this paper, we propose the problem of constrained dialogue generation, which involves controlling dialogue responses such that a set of control words appear at some point in the future of the conversation. we propose a new set of metrics as well as a novel method that leverages retrieval of relevant conversations to control future generated responses. we show on three datasets that our method outperforms several constrained text generation baselines on quantitative metrics as well as human evaluation. as far as we are aware, this is the first work to address the problem of long-term control for dialogue generation.