W alled E val: A Comprehensive Safety Evaluation Toolkit for Large Language Models (2024)

Abstract: WalledEval is a comprehensive AI safety testing toolkit designed to evaluate large language models (LLMs). It accommodates a diverse range of models, including both open-weight and API-based ones, and features over 35 safety benchmarks covering areas such as multilingual safety, exaggerated safety, and prompt injections. The framework supports both LLM and judge benchmarking, and incorporates custom mutators to test safety against various text-style mutations such as future tense and paraphrasing. Additionally, WalledEval introduces WalledGuard, a new, small and performant content moderation tool, and SGXSTest, a benchmark for assessing exaggerated safety in cultural contexts. We make WalledEval publicly available at https://github.com/walledai/walledeval with a demonstration video at https://youtu.be/50Zy97kj1MA.

## introduction llm technology has undoubtedly proven to be a valuable tool that simplifies various aspects of our lives. it can act as an email writing assistant, streamline information access, and help us write code blocks, saving us hours of work. starting with openai's chatgpt-3.5, we have seen the emergence of numerous llm variants, including both proprietary and closed-weight models, such as the chatgpt series models (chatgpts, @xcite ) and the claude series models @xcite ). alongside these closed variants, there has been a surge in open-weight models, including the popular series of mistrals @xcite , llamas @xcite and gemmas @xcite . as new models continue to emerge with enhanced knowledge and multitasking capabilities, it is crucial to assess their safety risks comprehensively. potential harms include training data leakage, biases in responses and decision-making (potentially leading to bias laundering), and unauthorized use, for example, for purposes such as terrorism and the generation of sexually explicit content @xcite . this increases the need for a one-stop center for safety evaluations of advanced ai systems; we thus introduce a pythonbased framework walledeval. the following are features of walledeval: beyond this, walledeval supports using generic llms as safety evaluators in the form of an llm-as-a-judge mode for both open-and closed-weight models. evaluating judges is just as important as evaluating the llms themselves, as a poorly performing judge may lead to erroneous safety measures @xcite . thus, wallede-val additionally facilitates the benchmarking of judges by comparing judge predictions against gold-standard labels. we also release hixstest, a manually curated small dataset in hindi consisting of 25 safe and unsafe prompts each, to further challenge judges 4 . as a framework, walledeval supports a range of off-the-shelf open-and closed-weight llms (e.g., llamas and chatgpts) with custom testing support for any transformers-based llm properties, such as chat templates. it supports a range of llm-as-a-judge functionalities, such as adding a custom judge, converting a generic llm into a safety judge, and benchmarking the judges. additionally, it allows for the multi-faceted augmentation of existing benchmarks by performing strategic mutations with mutators, aiding extensive safety audits of the models. ## framework design the walledeval framework consists of three main classes for creating core objects: a) dataset loader huggingfacedataset; b) llm loader hf_llm; and c) judge loader llmasajudge. this combination allows three types of testing: llm benchmarking (dataset -→ llm -→ judge -→ score), judge benchmarking (dataset -→ judge -→ score) and mcq benchmarking (dataset -→ template -→ llm -→ judge -→ score). getting the dataset ready. the first step is preparing the benchmark dataset. using functions in the huggingfacedataset class, the dataset object can be created in several ways: through a list of prompts, a csv/json file, or a huggingface dataset @xcite as shown in figure 2 . the list can contain either string prompts that one can directly feed into the llm or a list of dictionaries. the rest should contain the field "prompt" to be loaded correctly, while other fields specified will be ignored. getting the llm ready. now, the system under test -the llm object to be studied for safety -is created using hf_llm. here is a code snippet: note that device_map and model_kwargs are the standard huggingface arguments to load a model 5 . getting the judge ready. next, the judge used to evaluate llm responses is created. judges are designed to be binary text classifiers, determining if the text is safe. below are a few ways to create judge objects from different open-weight guardrails: 1 # llamaguard judge 2 @xmath0, @xmath1, 3 ** model_kwargs ) 4 5 # walledguard judge 6 @xmath2, 7 @xmath3 walledeval additionally supports llmbased judges, also referred to as llms-as-a-judge. ## evaluating llms and judges once the core objects are created, we can perform two tests: a) llm benchmarking, i.e., llm safety evaluations; and b) judge benchmarking, i.e., judge accuracy evaluations. ## experimental settings walledeval hosts over 35 datasets that test different safety behaviors of llms and facilitates the addition of custom datasets (figure 2 ). in this paper, we demonstrate its utility using harmful behavior datasets consisting of unsafe prompts, such as harmbench @xcite , 2024), advbench @xcite , and catqa (english) @xcite , as well as refusal behavior datasets with tricky safe and unsafe prompts, including xstest @xcite and sgxstest (ours). (details on datasets and prompting are relegated to appendix a.1. we perform experiments on several open-weight models, namely llamas (2023), mistrals (2023), qwens (2023), gemmas (2024), phi (2024), and aya models (2024), as well as the closed-weight models chatgpt-4 (2023), gemini 1.5 pro (2017), and claude 3 sonnet (2024). for llm harmful behavior benchmarking, we use llamaguard 2 8b as judge given it outperforms others ## mutations walledeval hosts mutators that perform textstyle transformations of a given prompt. in this demo, we show the effectiveness of nine such mutations: rephrasing, altering sentence structure, changing style, inserting meaningless characters, misspelling sensitive words, paraphrasing with fewer words, translating english to chinese @xcite , and converting between past and future tenses. for demonstration, we create a mutated version of the harmbench dataset, referred to as harmbench m , with 1,800 samples (nine mutations on 200 samples). similarly, we create a mutated version of xstest, referred to as xstest m , with 3,600 samples (eight mutations on 450 samples). we omit the rephrase mutation as the mutator was not able to preserve semantics on this dataset. ## experiments & discussions we showcase the results obtained by interacting with walledeval by performing various safety tests, such as standard benchmark testing, refusal tests (primarily english), and multilingual safety tests (in eight languages). harmful behavior tests. in ## related libraries existing evaluation frameworks for llm safety primarily focus on evaluating a specific component of llm safety. here, we detail a couple of opensource ai safety testing platforms. jailbreakeval @xcite hosts various safety judges from huggingface hub @xcite and api providers, such as openai moderation and perspective. it also supports substring judges as seen in @xcite . walledeval supports huggingface and string-based judges included in jailbreakeval. easyjailbreak @xcite provides support for various attack methods such as gcg @xcite , allowing one to use own dataset and mutate it to jailbreak an llm. however, it has limited support for evaluators and custom llms. walledeval currently implements only one-toone mutators, largely inspired by many implementations from easyjailbreak. to the best of our knowledge, walledeval is the first library to support customizable llms, datasets, and llms-as-a-judge, while also hosting a comprehensive set of safety evaluation benchmarks. this enables users to holistically compare both open and closed-weight llms and judges. ## limitations and future plans while walledeval aims to provide a comprehensive method for evaluating llms across a range of safety benchmarks, we acknowledge some limitations that will be addressed as feature enhancements in future work: currently, walledeval supports only nine mutators, which are primarily simple text-style transformations and are agnostic to the llm under test and the context of the conversation. moving forward, we plan to add more complex mutators, such as gcg @xcite and pair @xcite that adapt to the llm under test and trigger harmful behaviors. currently, huggingfacedataset loads only one split of a dataset at a time, which is highly inefficient as it limits the amount of data that can be loaded at once. therefore, we plan to add support for merging datasets and splits in huggingfacedataset to allow users to test various benchmarks more effectively and efficiently. ## conclusion in this paper, we propose walledeval, a tool for benchmarking llms and content moderators (judges) on a range of safety evaluation datasets, over 35 of which are hosted on the platform. we demonstrate the tool's utility in testing both harmful and refusal behavior. additionally, we introduce a new content moderator, walledguard -a significantly smaller yet high-performing guardrail -and a culturally tailored refusal dataset, sgxstest and hixstest.