D ial G uide: Aligning Dialogue Model Behavior with Developer Guidelines (2023)

Abstract: Dialogue models are able to generate coherent and fluent responses, but they can still be challenging to control and may produce non-engaging, unsafe results. This unpredictability diminishes user trust and can hinder the use of the models in the real world. To address this, we introduce DialGuide, a novel framework for controlling dialogue model behavior using natural language rules, or guidelines. These guidelines provide information about the context they are applicable to and what should be included in the response, allowing the models to generate responses that are more closely aligned with the developer’s expectations and intent. We evaluate DialGuide on three tasks in open-domain dialogue response generation: guideline selection, response generation, and response entailment verification. Our dataset contains 10,737 positive and 15,467 negative dialogue context-response-guideline triplets across two domains - chit-chat and safety. We provide baseline models for the tasks and benchmark their performance. We also demonstrate that DialGuide is effective in the dialogue safety domain, producing safe and engaging responses that follow developer guidelines.

## introduction current open-domain dialogue models such as di-alogpt @xcite , blenderbot @xcite , and plato @xcite have shown the ability to generate fluent and interesting responses. however, they are generally difficult to control and require large datasets to re-purpose them for a new task or domain. on the other hand, deployed conversational systems generally rely on handcrafted rules and templates @xcite @xcite @xcite . such systems allow for more control over responses and produce interesting, high quality responses, yet they are rigid and have poor coverage due to the difficulty of writing responses for every situation. we propose a new framework, dialguide, to control dialogue response generation using natural language rules, which we call guidelines. a guideline consists of an "if x" condition part specifying the context it is relevant to, and a "then y" action part that specifies what the response should contain. figure 1 presents an overview of our framework. we use a retrieve-then-infer process to retrieve guidelines relevant to the context and then use one of them to either generate or verify a response candidate. using guidelines in our proposed framework offers several benefits. guidelines allow developers to drive system actions towards predefined agendas, enhance response engagement, and address common issues in system outputs such as the generation of toxic responses. guidelines can be added, removed, or edited during deployment, and updating them does not require retraining the model. guidelines are more flexible, as they are not as rigid as regex-based rules, but are more abstract. our framework merges language models' instruction understanding with developers' intuitive guidelines expressed in natural language. it is important to note that the model's ability to generate responses is not limited solely to the guidelines present. in the absense of relevant guidelines, our models are trained to generate responses directly without conditioning on guidelines. in the dialguide framework * , we benchmark three tasks: 1) guideline selection, where a model needs to retrieve context-relevant guidelines, 2) response generation, where a model generates a response that follows a selected guideline, and 3) response entailment verification, where the model determines whether a response follows or violates a guideline. we augment conversations from existing dialogue datasets -blended skills talk @xcite and prosocialdialog @xcite by collecting annotations of 1) relevant/irrelevant guidelines to the conversation context and 2) responses following/violating the guideline. to test the models' semantic understanding, we also create adversarial train and test sets. we establish benchmark performance on these tasks and show that models tuned on our data can generate better controlled and coherent responses. although the dataset is medium-sized, few-shot based models enable generalization to new guidelines and contexts. we also demonstrate our framework's effectiveness in the dialogue safety domain, generating safe and engaging responses. ## related work controlling dialogue systems has been a focus of research to generate engaging responses @xcite , prevent toxic content and biases @xcite , steer the conversation towards specific keywords or topics @xcite , and ground responses in background knowledge such as persona @xcite , emotions @xcite , or documents @xcite . many approaches train models on discrete labels or control codes, but this can be inflexible and requires retraining to incorporate new * code and data will be available labels. while neural dialogue models are the mainstream in research, chatbots in deployment often still rely on handcrafted rules @xcite and templates @xcite due to the ease of update and ability to generate high quality, controllable responses. there has also been progress in using natural language prompts and instructions to control models @xcite @xcite , but our work extends this by providing fine-grained semantic control through guidelines over open domain response generation. ## experiments and results in this section we discuss the experimental setup and results for the three tasks in dialguide setup. ## qualitative analysis in in overall, the model outputs show a range of quality, with some following the gold guideline more closely than others. although dialguide-tuned has the best performance in both results and qualitative analysis and forms a performance upper-bound using the gold guidelines, the retrieval baselines also show good performance and are more practical, as systems need to retrieve relevant guidelines at test time. the multistep baseline is also useful in scenarios where no good guideline is available, as the model can first generate a guideline on how it is going to respond and then generate the response. ## conclusion dialguide framework and dataset provide a solution for controlling dialogue model behavior using natural language rules, or guidelines. through the three tasks of guideline selection, response generation, and response entailment verification, di-alguide aims to enable better control of dialogue models and improve their trustworthiness and realworld use. we evaluate dialguide on two domains, chit-chat and safety, and provide baseline models and benchmark performance for these tasks. mod-els trained on dialguide data generate coherent, diverse, and safe responses that generalize well to new guidelines and contexts.