N e M o Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails (2023)

Abstract: NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems. Guardrails (or rails for short) are a specific way of controlling the output of an LLM, such as not talking about topics considered harmful, following a predefined dialogue path, using a particular language style, and more. There are several mechanisms that allow LLM providers and developers to add guardrails that are embedded into a specific model at training, e.g. using model alignment. Using a runtime inspired from dialogue management, NeMo Guardrails provides a different approach by allowing developers to add programmable rails to LLM applications - these are user-defined, independent of the underlying LLM, and interpretable. Our initial results show that the proposed approach can be used with several LLM providers to develop controllable and safe LLM applications using programmable rails.

## introduction steerability and trustworthiness are key factors for deploying large language models (llms) in production. enabling these models to stay on track for multiple turns of a conversation is essential for developing task-oriented dialogue systems. this seems like a serious challenge as llms can be easily led into veering off-topic @xcite . at the same time, llms also tend to generate responses that are factually incorrect or completely fabricated (hallucinations) @xcite @xcite . in addition, they are vulnerable to prompt injection (or jailbreak) attacks, where malicious actors manipulate inputs to trick the model into producing harmful outputs @xcite @xcite . building trustworthy and controllable conversational systems is of vital importance for deploy-* equal contribution 1 https://github.com/nvidia/nemo-guardrails ing llms in customer facing situations. nemo guardrails is an open-source toolkit for easily adding programmable rails to llm-based applications. guardrails (or rails) provide a mechanism for controlling the output of an llm to respect some human-imposed constraints, e.g. not engaging in harmful topics, following a predefined dialogue path, adding specific responses to some user requests, using a particular language style, extracting structured data. to implement the various types of rails, several techniques can be used, including model alignment at training, prompt engineering and chain-of-thought (cot), and adding a dialogue manager. while model alignment provides general rails embedded in the llm at training and prompt tuning can offer user-specific rails embedded in a customized model, nemo guardrails allows users to define custom programmable rails at runtime as shown in fig. 1 . this mechanism is independent of alignment strategies and supplements embedded rails, works with different llms, and provides interpretable rails defined using a custom modeling language, colang. to implement user-defined programmable rails for llms, our toolkit uses a programmable runtime engine that acts like a proxy between the user and the llm. this approach is complementary to model alignment and it defines the rules the llm should follow in the interaction with the users. thus, the guardrails runtime has the role of a dialogue manager, being able to interpret and impose the rules defining the programmable rails. these rules are expressed using a modeling language called colang. more specifically, colang is used to define rules as dialogue flows that the llm should always follow (see fig. 2 ). using a prompting technique with in-context learning and a specific form of cot, we enable the llm to generate the next steps that guide the conversation. colang is then interpreted by the dialogue manager to apply the guardrails rules predefined by users or automatically generated by the llm to guide the behavior of the llm. while nemo guardrails can be used to add safety and steerability to any llm-based application, we consider that dialogue systems powered by an llm benefit the most from using colang and the guardrails runtime. the toolkit is licensed as apache 2.0, and we provide initial support for several llm providers, together with starter example applications and evaluation tools. ## sample guardrails applications adding rails to conversation applications is simple and straightforward using colang scripts. ## conclusions we present nemo guardrails, a toolkit that allows developers to build controllable and safe llmbased applications by implementing programmable rails. these rails are expressed using colang and can also be implemented as custom actions if they require a complex logic. using cot prompting and a dialogue manager that can interpret colang code, the guardrails runtime acts like a proxy between the application and the llm enforcing the user-defined rails.