Prompting for a conversation: How to control a dialog model? (2022)

Abstract: Dialog modelling faces a difficult trade-off. Models are trained on a large amount of text, yet their responses need to be limited to a desired scope and style of a dialog agent. Because the datasets used to achieve the former contain language that is not compatible with the latter, pre-trained dialog models are fine-tuned on smaller curated datasets. However, the fine-tuning process robs them of the ability to produce diverse responses, eventually reducing them to dull conversation partners. In this paper we investigate if prompting can help with mitigating the above trade-off. Specifically, we experiment with conditioning the prompt on the query, rather than training a single prompt for all queries. By following the intuition that freezing the pre-trained language model will conserve its expressivity, we find that compared to fine-tuning, prompting can achieve a higher BLEU score and substantially improve the diversity and novelty of the responses.

## introduction prompting large language models (llm) has recently demonstrated an impressive performance on a number of natural language processing (nlp) tasks such as machine translation @xcite , summarisation @xcite or question answering @xcite . prompts are tokens which are appended or prepended to the input of a language model. they are employed to induce the model into generating useful information, while keeping the model weights frozen. soft prompts, continuous trainable vectors prepended to the model input, have in particular proven useful for a number of tasks @xcite . while requiring fine-tuning of only a relatively small number of parameters, they excel in a few-shot setting. furthermore, as the underlying llm's grow in parameter size, * work done while at apple. they become competitive even in the full data setting @xcite . simultaneously, prompts remove the burden of storing the full copy of a fine-tuned llm for every task, which becomes increasingly useful as the llm size grows. crucially for us, prompting preserves the llm parameters which should help retain their general language abilities for a downstream task. dialog modelling is the task of generating a response given the previous dialog turn, the query @xcite . dialog models are typically trained using the maximum-likelihood estimation (mle) objective. however, mle-trained models have a high propensity to provide dull responses, such as "i don't know" @xcite @xcite . while state-of-the-art models, such as dialogpt @xcite , can overcome this issue by training large models on massive amounts of data, a trade-off emerges. on the one hand, these models are expressive by virtue of the large datasets they are trained on. on the other hand, the same scale of training data and model parameters is responsible for the lack of control over the content of their responses. since dialog models are not useful without the ability to control their responses, in this paper, we turn to prompting as a possible method of exerting such control. instead of fine-tuning the entire model, we keep the weights of the model intact, tuning only the prompts (and word embeddings) in an effort to preserve the models' expressivity. furthermore, we develop a dynamic-prompt, foot_0 which conditions the prompt on the query in an effort to dynamically induce the response by having a different prompt for every turn of the conversation, see fig. 1 . the intuition is to separate the task of language generation, which llm's are very good at, from the task of selecting appropriate responses, which llm's struggle to learn from the limited examples they are fine-tuned on. in our experiments with dailydialog dataset, we find that the dynamic prompt outperforms both fine-tuning and the soft-prompting of dialogpt in terms of a bleu4 score. the best dynamic prompt model achieves 0.12 bleu4 compared to 0.11 and 0.08 of fine-tuning and soft-prompting respectively. furthermore, the dynamic prompt finds the best trade-off between the bleu score and novelty as well as the diversity of responses, maintaining above 0.90 novelty and diversity as the bleu4 score increases, while the other models can not. finally, we find that dynamic-prompting on gpt-2 achieves the best bleu4 results, improving over prompted dialogpt by an additional 19% and casting doubt about the utility of dialog-specific pre-training when it comes to prompting. ## prompting for dialog generation fine-tuning a pre-trained model on a small dialog dataset degrades the novelty of the responses @xcite . prompting opens up a possibility of exerting control over the model behaviour, without touching the majority of the weights that might be useful for generating novel responses @xcite . however, typical prompting methods are restrained in using a single prompt for a single task. in this section, we motivate the dynamic prompt as a natural next step in generalising the prompting paradigm for dialog modelling. prompting. auto-regressive models with billions of parameters trained on a language modelling objective, such as openai's gpt-3 @xcite , have demonstrated a strong few-shot performance without the need to update the model parameters. instead of fine-tuning the model with target task examples, a manually designed prompt, for example in the form of a natural language sentence describing the task, is fed to the model to solicit the desired response. for instance, to induce a translation the model might be told to: translate english to french. the description is followed up with pairs of examples of english sentences and their french translations. to derive a new translation the model is fed an english sentence alone and it is left to infer the french translation. however, not all human-designed prompts elicit the desired response. in practice ensembles of many prompts have gone some way in improving the performance, but still necessitate humans in the loop to design the said prompts @xcite . ## method notation. let a dialogue be denoted as this probability can be parametrized by an autoregressive language model (in our case dialogpt). fine-tuning. for our experimental baseline, we simply fine-tune pre-trained dialogpt on the dai-lydialog dataset, see § 4. ## experimental setup datasets. we conduct our experiments on the dailydialog dataset consisting of 13, 118 dialogues, split into 11, 118/1, 000/1, 000 training/validation/test sets @xcite . we only focus on single-turn dialog modelling and process the dataset so that our pairs of queries and responses correspond to every two steps in a conversation. inspired by @xcite , we train our models on 10%, 20%, 30%, 50%, 70%, and 100% of the dailydialog dataset to observe how the number of training samples affects the performance. we keep the validation and test sets full-sized for all experiments. for full experimental details, see app. b. metrics. we follow @xcite and evaluate the models using bleu4 score. since we do not want the model to simply repeat the responses it has memorised from the training data, i.e. the dull response issue, we additionally introduce two new metrics: novelty and diversity. we define novelty as the proportion of model outputs on the test set that are not found in the training set. given a test set, diversity is defined as the number of unique model outputs divided by the total number of outputs. a good dialog model should be able to achieve a high bleu4 score, while maintaining a high level of novelty and diversity in its responses. ## results our main results are contained in fig. 3 . first, we compare the performance of prompting vs fine-tuning on the dialogpt model. we find that under the full data setting the dynamic prompt model outperforms the soft prompt model and even the fine-tuned model on all three metrics. with a bleu4 of 0.12 dynamic prompt is 9% better than fine-tuning (0.11) and 15% better than a soft prompt (0.08). the difference is even more dramatic when we compare the corresponding diversity and novelty scores. dynamic prompt achieves a novelty of 0.93 and diversity of 0.96, an improvement of 9% (0.85) and 10% (0.87) respectively over the soft prompt, and an even more pronounced improvement of 26% (0.74) and 22% (0.79) respectively over fine-tuning the model. next, we observe that soft-prompting is competitive in the low data setting and outperforms fine-tuning. however, with more than 30% of the training data available, soft-prompting stops improving altogether and begins to under-perform the other models. dynamic prompt on the other hand maintains the best bleu4 score no matter the amount of training data. now we turn to the question of novelty and diversity degradation. we observe that the dynamic prompt does not suffer from this issue nearly as much as the other models. in contrast, for the soft-prompted and fine-tuned models, the better the bleu4 score gets the lower the novelty drops. while soft prompt already mitigates this drop-off, degrading slower than dialogpt, dynamic prompt does not suffer from this effect and maintains above 90% novelty throughout. similarly, for diversity, the dynamic prompt maintains around 0.95 score for any percentage of training data, while the soft prompt drops down over time and fine-tuning hovers around 0.80. finally, we can turn to the comparison of dialogpt and gpt-2. against our expectations, dialogpt under-performs gpt-2 on bleu4 despite the former having been pre-trained on dialog-specific data. the performance improvement is most notable in the case of the dynamic prompt, where gpt-2 (0.14) achieves 19% higher performance than dialogpt (0.12). on the other hand, fine-tuning either model leads to nearly identical performance. you can find our discussion of this phenomenon in app. a, all our results in app. c and example outputs in app. d. ## related work our work builds on two strains of thought. first, we deal with the problem of dull responses in dialog modelling. related work includes the use of reinforcement learning @xcite , latent variables @xcite and decoding techniques to mitigate this issue @xcite . second, we build on the idea of continuous prompting, which was developed concurrently by @xcite and @xcite . there are many variations of the prompting paradigm. for instance, we fine-tune the embeddings along with the prompts, but @xcite tune the prompt with the full model. @xcite on the other hand tune only the model, while keeping the prompt fixed. ## conclusion we find dynamic-prompting bests fine-tuning by generating more novel and diverse responses with a higher bleu score while training only a small portion of the dialogpt/gpt-2 parameters. thus proving itself as a useful method for mitigating dialog modelling issue of dull responses. prompting the general purpose gpt-2 achieves much higher performance than prompting the specialist dialogpt model, suggesting that for pre-training data, diversity is more valuable for dialog modelling than dialog-specific information. a appendix a: discussion @xcite designed their soft prompt based on the observation that context can control the llm without the need to change its parameters. they demonstrate that the context might be hard to find among the existing word embeddings and instead learn it via back-propagation. we go a step further and demonstrate that when it comes to dialog generation it is harder to learn a single prompt than to learn a function that conditions the prompt on the query. we believe that this is because there is no single context, represented by words or embeddings, that can capture a complex task such as a conversation. this becomes intuitive if we separate the task of language generation from the task of making a conversation. the conversation is an interaction of the query with the state of the conversation agent. without the ability to respond to the query the agent/language model is simply saying whatever is the most probable response. a prompt can induce some general modification over this response, but there is no process by which the agent can react to the query. in our dynamic prompt setting, the agent, represented by the controller module, is allowed to learn how to respond to each query. the separation of thought from language production is of course not a new idea. under the language of thought hypothesis, mentalese is the language of thought @xcite . while similar to natural language in its compositional structure, mentalese is separate from language itself. from this perspective, building models that separate the task of learning a language from that of learning how to use language makes perfect sense. following this intuition, it also makes sense to pre-train the language model on general language, rather than a specific task. we believe that in our setting, the superior language ability of gpt-2 allows for a better controller in the dynamic prompt setting.