CONSCENDI : A Contrastive and Scenario-Guided Distillation Approach to Guardrail Models for Virtual Assistants (2024)

Abstract: A wave of new task-based virtual assistants has been fueled by increasingly powerful large language models (LLMs), such as GPT-4 (OpenAI, 2023). A major challenge in deploying LLM-based virtual conversational assistants in real world settings is ensuring they operate within what is admissible for the task. To overcome this challenge, the designers of these virtual assistants rely on an independent guardrail system that verifies the virtual assistant’s output aligns with the constraints required for the task. However, relying on commonly used, prompt-based guardrails can be difficult to engineer correctly and comprehensively. To address these challenges, we propose CONSCENDI. We use CONSCENDI to exhaustively generate training data with two key LLM-powered components: scenario-augmented generation and contrastive training examples. When generating conversational data, we generate a set of rule-breaking scenarios, which enumerate a diverse set of high-level ways a rule can be violated. This scenario-guided approach produces a diverse training set and provides chatbot designers greater control. To generate contrastive examples, we prompt the LLM to alter conversations with violations into acceptable conversations to enable fine-grained distinctions. We then use this data, generated by CONSCENDI, to train a smaller model. We find that CONSCENDI results in guardrail models that improve over baselines in multiple dialogue domains.

## introduction the emergence of transformer-based @xcite large language models (llms), such as @xcite and @xcite , have enabled highly-capable conversational agents. with this increase in natural language * work completed while an intern at curai health. for correspondence, please contact alsunyu@gmail.com sophistication, agent designers must ensure both responsible usage and adherence to task-specific constraints. guardrail models have been designed to ensure these rules are enforced @xcite . most of these systems primarily focus on preventing the generation of harmful text @xcite @xcite . customized domain-specific guardrails often consist of manually-engineered llm prompts. yet constructing a prompt sufficiently robust to all rulebreaking behavior is challenging through instructions and in-context examples alone. for example, a rule prohibiting an agent from stating political opinions can guard against generating controversial text. yet defining the intricacies of this rule is challenging -are widely accepted statements acceptable, but more sectarian statements out-of-bounds? issues with higher cost and latency for higher-performance llms compound concerns around this approach. using a smaller llm as a guardrail alleviates cost and latency concerns, but does not address the likely reduction in performance. one solution is to fine-tune smaller llms on the output of larger llms, enabling these smaller models to have better performance @xcite . however, naively generating data from a large model such as gpt-4 could produce datasets that do not encapsulate all possibilities that break desired behavior. therefore, we propose a multi-stage data generation pipeline to ensure gpt-4 produces a broad, domain-specific dataset. we begin by prompting an llm to generate a variety of scenarios that illustrate different ways a dialog agent might break each given rule. scenarios can be added or removed from this set given the engineer's preferences, providing a granular level of control. next, we use gpt-4 to simulate a conversation between a user and a dialog agent that violates the rule according to the provided scenario. this scenario-guided data generation method results in a more diverse figure 1 : example guardrail task. in this example, a virtual assistant in the restaurant domain provides information about an ongoing promotion to the user, thereby breaking rule 2. the guardrail model uses the last turn of the conversation (non-grayed text; in our models, we use the last two turns) to classify the last two turns as a rule violation (which rule) or no violation. furthermore, we employ a contrastive approach to generate non-violating conversations that are alterations of a conversation with violations @xcite . in addition to directly generating non-violating conversations, contrastive example generation takes further advantage of gpt-4's generation capabilities and provides a richer dataset for model training. the combined dataset is used to fine-tune models to serve as guardrail models. we show these distilled models can often serve as better guardrail models than prompt-based llms, providing a crucial tool for user-facing text generation tools. our paper makes the following contributions: ## guardrails for virtual assistants a virtual assistant typically consists of an agent model a and a guardrail model g. a's role is to have a conversation with the users of the virtual assistant. g's goal is to ensure that a converses with the user within its rules for the task. this paper presents an approach to building a reliable guardrail model, conscendi. con-scendi takes as input a set of rules and utilizes a large, highly-capable llm such as gpt-4 to automatically generate an expansive labeled training dataset (section 2.2). this dataset is then used in the distillation of a smaller model (section 2.2.2). figure 2 : conscendi. we finetune a gpt-3 model by 'distilling' gpt-4 through a focused data generation paradigm. we use gpt-4 to generate rule-specific scenarios (see section 2.2). we generate three types of conversations for each scenario: 1. violations: conversations that violate the rule, 2. contrastive nonviolations: conversations that are identical to our generated violations but replace the rule-violating turn with a non-violating turn, and 3. nonviolations: conversations that don't violate any of our rules. these newly generated conversations are few-shot generated using example conversations from @xcite . ## datasets we demonstrate the efficacy of our approach to virtual assistants in three domains: flights, restaurants, and buses. these are drawn from the schema guided dialogue (sgd) dataset's 20 schemas @xcite . the sgd dataset contains con ## experiments baselines: we compare conscendi (distilled ✓contrastive ✓scenarios) with the following baselines: metrics: we use accuracy as the main metric for evaluation. specifically, we count each guardrail prediction as correct only if it classifies the conversation with the correct rule. this strict metric is indicated by the critical issues that an incorrect classification might produce. we apply the same metric to our ood examples. we additionally report cost in (u.s. dollars and latency (in seconds) for some experiments). costs were also calculated using the openai pricing page (see appendix a.2). ## discussion leveraging a distilled gpt-3 model combines the efficiency of a smaller model with the accuracy of a more powerful one. in all cases, fine-tuned gpt-3 models outperform vanilla gpt-3 models in terms of accuracy. even compared to larger models, such as gpt-4, our distilled approach not only provides benefits in terms of latency and cost but also delivers improvements in terms of accuracy. this is the case for both scenarios seen during model training (id examples), and unseen scenarios (ood examples) that have been held out. we find that a major factor in its ability to generalize is the inclusion of contrastive examples. as broadly shown in previous work @xcite , we find that these examples allow gpt-3 to specifically better model the fine-grained differences that can occur between conversations with and without violations. we also note that the ability of gpt-4 to produce these contrastive examples illustrates its generative power. ## related work language models are increasingly used to power task-oriented dialogue systems, like chatgpt (openai, 2022) and google's bard @xcite . they are used as personal assistants and customer support in different domains @xcite . with this increase in language model ability, there has been an increased focus on ensuring that generated text does not contain harmful content @xcite @xcite or is better aligned with user preferences @xcite . previous works have used reinforcement learning from human feedback (rlhf) to minimize harmful content from large language models @xcite . @xcite advocates for fine-tuning models with human feedback without reinforcement learning. our approach of using language models to scale oversight and help supervise other language models is also similar to the approach in @xcite . they focus on general harmlessness/harmfulness rules, while our approach is a more general approach that allows chatbot designers decide what type of rules they want to enforce downstream. knowledge distillation has shown to be an effective way to compress the knowledge of larger models/ensembles of models into single, smaller models @xcite . previous work has shown the ability of large language models to transfer reasoning capabilities to smaller language models for specific tasks @xcite @xcite . unlike previous work, we train our student model on generated examples from the teacher model. this is unlike previous work that trains student models on the inference or reasoning capabilities of a teacher model. this allows us to harness the generation abilities of larger models while minimizing latency and hardware costs. ## conclusion we propose conscendi, a distillation approach for guardrail models. these verification models are crucial for enabling large language model-based tools to be deployed with confidence. in addition to potential applications in harm reduction, they also allow conversational agent designers to include application-specific rules not accounted for in the original model training. we propose a distillation pipeline that enables data generation across a broad variety of cases. by first generating rule-breaking scenarios, the resulting conversations will cover a broader set of possibilities than doing so without this step. second, by transforming these rule-breaking conversations into non-rule-breaking conversations, we provide the model with a set of contrastive examples that better teach it how to differentiate between the cases. our results demonstrate that gpt-4 generated training data allows fine-tuned smaller models (gpt-3) to surpass baselines in various metrics like accuracy, speed, and cost. there are several future directions for distilling guardrail models. while we design separable violations, this might not be possible in practice. approaches that can handle multi-label violations will likely be helpful in those settings. further, designing evaluation strategies for generated conversational data will be important in ensuring that output will be similar to real-world data.