G uard B ench: A Large-Scale Benchmark for Guardrail Models (2024)

Abstract: Generative AI systems powered by Large Language Models have become increasingly popular in recent years. Lately, due to the risk of providing users with unsafe information, the adoption of those systems in safety-critical domains has raised significant concerns. To respond to this situation, input-output filters, commonly called guardrail models, have been proposed to complement other measures, such as model alignment. Unfortunately, the lack of a standard benchmark for guardrail models poses significant evaluation issues and makes it hard to compare results across scientific publications. To fill this gap, we introduce GuardBench, a large-scale benchmark for guardrail models comprising 40 safety evaluation datasets. To facilitate the adoption of GuardBench, we release a Python library providing an automated evaluation pipeline built on top of it. With our benchmark, we also share the first large-scale prompt moderation datasets in German, French, Italian, and Spanish. To assess the current state-of-the-art, we conduct an extensive comparison of recent guardrail models and show that a general-purpose instruction-following model of comparable size achieves competitive results without the need for specific fine-tuning.

## introduction in the recent years, generative ai systems have become increasingly popular thanks to the advanced capabilities of large language models (llms) @xcite . those systems are in the process of being deployed in a range of high-risk and safetycritical domains such as healthcare @xcite , education @xcite , and finance @xcite . as ai systems advance and are more extensively integrated into various application domain, it is crucial to ensure that their usage is secure, responsible, and compliant with the applicable ai safety regulatory framework. particular attention has been paid to chatbot systems based on llms, as they can potentially engage in unsafe conversations or provide users with information that may harm their well-being. despite significant efforts in aligning llms to human values @xcite , users can still misuse them to produce hate speech, spam, and harmful content, including racist, sexist, and other damaging associations that might be present in their training data @xcite . to alleviate this situation, explicit safeguards, such as input-output filters, are becoming fundamental requirements for safely deploying systems based on llms, complementing other measures such as model alignment. very recently, researchers have proposed the adoption of the so-called guardrail models to moderate user prompts and llm-generated responses @xcite @xcite . given the importance of those models, their evaluation plays a crucial role in the generative ai landscape. despite the availability of a few datasets for assessing guardrail models capabilities, such as the openai moderation dataset @xcite and beavertails @xcite , we think there is still need for a large-scale benchmark that allows for a more systematic evaluation. we aim to fill this gap by providing the scientific community with a large-scale benchmark comprising several datasets for prompts and responses safety classification. to facilitate the adoption of our proposal, we release a python library that provides an automated evaluation pipeline built on top of the benchmark itself. moreover, we share the first large-scale multi-lingual prompt moderation datasets, thus overcoming english-only evaluation. finally, we conduct the first extensive comparison of recent guardrail models, aiming at shedding some light on the state-of-the-art and show a general-purpose instruction-following model of comparable size achieves competitive results without the need for specific fine-tuning. our contributions can be summarized as follows: in this section, we discuss previous work related to our benchmark. firstly, we discuss the moderation of user-generated content. secondly, we introduce the moderation of human-ai conversations. ## benchmark composition in this section, we introduce the benchmark we have built by collecting several datasets from previous works and extending them through data augmentation. to decide which datasets to include in our evaluation benchmark, we first conducted a literature review and consulted safetyprompts foot_1 @xcite . we considered over 100 datasets related to llm safety. to narrow down the initial list of datasets and identify those best suited for our evaluation purposes, we defined inclusion and exclusion criteria, which we present in section 3.1. as many of these datasets were not proposed to evaluate guardrail models, we repurposed them to our needs as they already contained safety information. we include 35 datasets from previous works in our benchmark, which can be broadly categorized as prompts (instructions, question, and statements) or conversations (singleturn and multi-turn), where the object to be moderated is the final utterance. due to the lack of non-english datasets @xcite , we augmented those available through automatic translation, providing the scientific community with the first prompts safety evaluation sets for guardrail models in german, french, italian, and spanish. we detail such process in section 3.3. finally, as described in section 3.4, we generate safe and unsafe responses to unsafe questions and instructions from previous works to obtain a novel large-scale conversational dataset for our evaluation. the final list of datasets comprised in our benchmark is presented in ## experimental setup in this section, we introduce the experimental setup adopted to answer the following research questions: rq1 what is the best model at moderating user prompts? rq2 what is the best model at moderating human-ai conversations? rq3 how does available models perform on languages other than english? rq4 how does content moderation policies affect models' effectiveness? to answer the research questions rq1 and rq2 we compare the effectiveness of several models at classifying prompts and conversation utterances as safe or unsafe. then, to answer rq3, we compare the models on our newly introduced multi-lingual prompt datasets, described in section 3.3. finally, we evaluate the importance of moderation policies by comparing the results of a general-purpose llm with different policies to answer rq4. in the following sections, we introduce the models we have compared (section 4.1) and discuss the evaluation metrics chosen to assess the models' effectiveness (section 4.2) before presenting the results in section 5. ## results and discussion in this section, we present the results of our comparative evaluation. first, we discuss the models' effectiveness in assessing user prompts and human-ai conversations safety in section 5.1 and section 5.2, respectively. then, in section 5.3, we show preliminary results on non-english prompts. finally, we evaluate the importance of content moderation policies in section 5.4. note that the results of mistral with refined policy are considered only in section 5.4. we refer the reader to ## conclusion and future work in this work, we proposed guardbench, a largescale benchmark for evaluating guardrail models. guardbench comprises 40 datasets for prompts and conversations safety evaluation. we included 35 datasets in english from previous works and five new datasets. specifically, we built a new dataset for conversation safety evaluation by generating 22k answers to unsafe prompts from previous works. moreover, we translated 31k english prompts to german, french, italian, and spanish, producing the first large-scale prompts safety datasets in those languages. to facilitate the adoption of guardbench by the research community, we released a python library offering a convenient evaluation pipeline. we also conducted the first large-scale evaluation of state-of-the-art guardrail models, showing that those models perform close to each other when identifying unsafe prompts, while we register more pronounced differences when used to moderate conversations. fi-nally, we showed general-purpose and instructionfollowing models can achieve competitive results when correctly prompted for safety moderation. in the future, we plan to extend guardbench with an enhanced evaluation procedure to provide more structured results over the different categories of unsafe content. safety classification of prompts and conversation utterances remains an open problem with considerable room for improvement. advancements in this area are of utmost importance to safely deploy large language models in highrisk and safety-critical domains, such as healthcare, education, and finance.