AEGIS 2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails (2025)

Abstract: As Large Language Models (LLMs) and generative AI become increasingly widespread, concerns about content safety have grown in parallel. Currently, there is a clear lack of high-quality, human-annotated datasets that address the full spectrum of LLM-related safety risks and are usable for commercial applications. To bridge this gap, we propose a comprehensive and adaptable taxonomy for categorizing safety risks, structured into 12 top-level hazard categories with an extension to 9 fine-grained subcategories. This taxonomy is designed to meet the diverse requirements of downstream users, offering more granular and flexible tools for managing various risk types. Using a hybrid data generation pipeline that combines human annotations with a multi-LLM “jury” system to assess the safety of responses we obtain Aegis2.0, a carefully curated collection of 34,248 samples of human-LLM interactions, annotated according to our proposed taxonomy. To validate its effectiveness, we demonstrate that several lightweight models, trained using parameter-efficient techniques on Aegis2.0, achieve performance competitive with leading safety models fully fine-tuned on much larger, non-commercial datasets generated leveraging GPT-4. Additionally, we introduce a novel training blend that combines topic following data with safety data. This approach enhances the adaptability of guard models, enabling them to generalize to new risk categories defined during inference. We plan to open-source Aegis2.0 data and models to the research community to aid in safety guardrailing of LLMs.

## introduction systems designed to ensure safe interactions between humans and large language models (llms) * primary and equal contributors. more recent approaches to content moderation like meta's llama guard @xcite ) and google's shieldgemma @xcite look to improve flexibility in content moderation systems by leveraging the ability of llms to utilize their internal knowledge and zero-shot generalization capabilities to handle new safety risks. however, these models are trained on closed source datasets, limiting the possibility of iterating over them by the larger research community. the development of aegis2.0 addresses the need for a safety-focused dataset suitable for commercial applications, featuring a diverse collection of samples curated from a comprehensive taxonomy of harms. our scalable content safety risk taxonomy, consisting of 12 core categories and 9 fine-grained risks, captures critical safety concerns in human-llm interactions. aegis2.0 is an extension of the aegis1.0 dataset and family of models @xcite . this version improves upon the prior work, offering a more comprehensive dataset with added finegrained hazard categories and better balance of previously underrepresented categories, along with multiple data quality improvements like the availability of separated prompts and responses from previously combined conversation turns, the availability of separate labels for prompts and responses, and synthetic data augmentations with refusals and jury of llms. we also present significant modeling improvements and discussion on the prompts and modeling choices that worked best through detailed ablations. aegis2.0 is designed for flexibility and scalability, the taxonomy allows human annotators to provide free text input for unclassified risks, which are later standardized into fine-grained categories, enabling the discovery of new hazards and ensuring scalability without predefined constraints. the dataset includes a variety of prompts that cover critical risks, adversarial jailbreaks, and cultural contexts, with responses generated by unaligned llms. annotations are performed at the dialogue level, with prompt and response labels extracted using weak supervision from a jury of llms, aligned with human judgments. models fine-tuned on aegis2.0 demonstrate performance comparable to recent models like wildguard @xcite that have been trained on larger datasets that leverage powerful, non-commercial sources like gpt4. our key contributions are as follows: the aegis2.0 dataset is now available to download foot_0 and the llama3.1-aegisguard model is available both as lora adapter weights foot_1 and as an nvidia nim foot_2 for optimized inference. ## related work as llm safety is becoming an area of growing research and commercial interest, there are an increasing number of datasets available to benchmark llm safety for evaluation. however many of these are of small size and not intended to be used for training content moderation models @xcite @xcite ). an earlier available dataset suitable for training safety classifiers is toxicchat @xcite , but its use of vicuna @xcite responses limits commercial use due to licensing of the sharegpt 4 data used to train it. a more recent dataset for content moderation is wildguard-mix @xcite which covers wide-ranging safety risks, response refusals, and adversarial jailbreak data to have a total of 92k samples. however, 85% of its training split (wildguardtrain) is generated using gpt4, the use of which constrains commercial use. additionally, both toxicchat and wildguard-train do not include annotated categories of safety hazards, thus providing a binary label annotation only. we argue that prediction of categories from a diverse taxonomy is important in production use cases in a guardrails system @xcite , as the orchestration layer usually needs to generate a reason to relay to the user on why a request was blocked. finally, a topic modeling based @xcite analysis of the wild-guardmix dataset shows that important safety risk categories such as "sexual abuse in children" and "suicide and self-harm" are not well represented in it (more details in appendix a.7). these are extremely critical to moderate and have direct implications to mental health crisis and crimes against children. another recent dataset similar to ours is beavertails @xcite which includes human annotations over a safety taxonomy of 14 categories, broadly aligned with ours (see appendix a.1). however beavertails also poses issues for commercial use due to employing alpaca-7b (taori et al.) for response generation, which is trained using self-instruct @xcite ) style 4 https://sharegpt.com/ data generated from openai models. additionally, our dataset includes more sources of adversarial prompts @xcite @xcite which we expect to result in more robust content moderation models. meta's llama guard @xcite was one of the first content moderation llms, and the more recent llamaguard3-8b, based on the llama 3.1 family of models @xcite is the latest in a series of content moderation models by meta. however, the llama guard family of models are instruction-tuned on an unreleased internal dataset. our experimental results show that parameter-efficient fine-tuning on the aegis2.0 using llama3.1-8b-instruct as a base model surpasses llamaguard3-8b, a model that is instruction-tuned with the same starting backbone, providing evidence of the utility of aegis2.0 as a training blend. shieldgemma @xcite from google is also trained on a closed dataset and covers a safety risk taxonomy of only 4 categories. in addition, it is difficult to adjust shieldgemma to novel safety policies on the fly as it is optimized to handle one policy at a time with multiple inference calls needed to handle several risk categories. more recently, content moderation llms @xcite have been trained using the wildguardtrain and beavertails datasets which we compare to these in our experiments. ## content safety risk taxonomy to guide content moderation dataset annotation, we define an extensive and scalable content safety risk taxonomy comprising 12 core categories and 9 additional fine-grained risks, as outlined in our aim during annotation was to develop a scalable, tiered taxonomy that allows for revisiting policy definitions, minimizing errors, and discovering new risks. in addition to the core risk categories, we included a top-tier category, called other, which annotators selected when no predefined category applied under the given policy. for instances labeled as other, annotators were asked to choose from a set of potentially unsafe categories not yet included in the taxonomy or to provide a free-text description of the most relevant hazard, along with an explanation. as demonstrated by @xcite , free-text explanations significantly improve the identification of subtle unsafe content. this approach facilitates risk discovery, enhances scalability, and supports policy updates. we later integrate these free-text annotations into the fine-grained categories outlined in in addition to deciding whether samples are safe or violate specific risk categories, annotators were given the option to label ambiguous instances as needs caution, in order to prevent the unnecessary classification of uncertain content as unsafe. by incorporating this label, we enable the design of a system that can either adopt a defensive approach -blocking the request or response -or remain permissive while still being helpful, depending on how the needs caution designation is interpreted. ## creation of aegis2.0 dataset our content moderation dataset, aegis2.0 comprises of a diverse collection of benign and adversarial english prompts, covering both safe and harmful scenarios, alongside llm-generated responses. unlike other safety datasets @xcite , which rely on synthetic prompt generation, we source potentially harmful prompts from realworld interactions. to ensure prompt diversity, we selected prompts from anthropic/hh-rlhf dataset @xcite , do-anything-now dan @xcite , ai-assisted red-teaming (aart) @xcite , and do-not-answer @xcite for each of the selected prompts, we generated responses using mistral-7b-v0.1 @xcite . the coherence of synthetically generated responses was not validated, as the primary goal was to curate a balanced set of benign and unsafe llm responses. upon inspection of the model responses, we found that mistral-7b-v0.1 chose to engage and comply with many harmful user queries. current state-of-the-art llms @xcite @xcite that include safety alignment as part of post-training do not easily engage with harmful user queries and refuse to respond and redirect the conversation. in order to include such refusal samples as part of our dataset, we generate responses that avoid engaging with harmful queries using gemma-2-27b @xcite prompted to refuse unsafe prompts. the final dataset comprises 34,248 samples, including 16,880 standalone prompts, 17,368 prompt and response pairs (of which 5,200 are synthetic refusals). we build train, validation, and test splits containing 30, 763, 1, 501, and 1, 984 samples, respectively, via stratified sampling. the splits are available on huggingface to enable testing the performance of other models on our test set and ensuring correct usage of aegis2.0 dataset when comparing different models. more statistics of the dataset are included in appendix a.8. ## safety guard models on aegis2.0 we train content moderation models by performing parameter-efficient fine-tuning (peft) with llama3.1-8b-instruct (llama team, 2024b) as our backbone. the model is trained on the aegis2.0 train split using the majority label inferred from the human and llms annotations to predict a label of safe/unsafe for each of the prompt and response. details about the training setup and hyperparameters can be found in appendix a.4.4. we compare the performance of our best model llama3.1-aegisguard against industry baselines in we additionally notice from ablations that the binary safe/unsafe prediction performance improves with the adding fine-grained categories in the prompt template, compared to the core category taxonomy alone. this can be attributed to the fact that the wildguardmix dataset contains many fine-grained risks like phishing, malware, and unauthorized advice, and privacy issues that are not present in the core categories. this demonstrates that a more detailed taxonomy not only enhances llama3.1-aegisguard's ability to predict specific hazard categories but also improves its accuracy in distinguishing between safe and unsafe examples. we additionally notice that using weakly supervised (-jury of llms) instead of the conversation-level annotations for responses substantially boosts performance on response moderation. additionally, adding refusal data ( §4.2) and topic following data ( §6) also provide important increases in performance. ## improving content moderation via topic following in this section, we elaborate on combining topicfollowing with safety data and examine the impact of training on this data blend on content safety classification. ## conclusion this paper introduces aegis2.0, a dataset usable by commercial applications designed to address diverse safety risks in large language models using a taxonomy of 12 core and 9 finegrained risk categories. by leveraging a hybrid approach that combines human and llm-generated annotations, we demonstrate the effectiveness of the aegis2.0 dataset by using it to train the llama3.1-aegisguard, which performs at par with the wildguard on the wildguardtest set and substantially outperforms it on the ope-nai mod, all while using a much smaller training dataset and only open-source, commercialusable llms for weak supervision, unlike the use of gpt4 for supervision in the wildguard-train dataset. we also show substantially improved performance compared to other baselines like llamaguard3-8b and openai mod api, providing conclusive evidence for the utility of aegis2.0 for training content moderation models. thus, we hope that the release of aegis2.0 and associated aegisguard models offers valuable resources for advancing llm safety systems. our ablation studies show improved performance on inclusion of fine-grained risk categories, as compared to only the core categories in our taxonomy, providing evidence of the benefit of allowing annotators to use a flexible free-text input on unsafe samples that did not fit into the core categories. finally, our experiments on augmenting our content moderation data with topic following dialogue moderation data show enhanced model robustness and improved performance on prompt safety tasks. moreover, tf-improved models are much more adaptable to new risk categories not part of the safety datasets. future extensions to this work, based on the limitations discussed in §8, include expanding the dataset to include responses from more llms and increasing representation of important risk categories that are currently underrepresented, expanding to multiple languages, and addition of more types of prompts designed to jailbreak moderation models to further enhance robustness.