Granite Guardian: Comprehensive LLM Safeguarding (2025)

Abstract: The deployment of language models in real-world applications exposes users to various risks, including hallucinations and harmful or unethical content. These challenges highlight the urgent need for robust safeguards to ensure safe and responsible AI. To address this, we introduce Granite Guardian, a suite of advanced models designed to detect and mitigate risks associated with prompts and responses, enabling seamless integration with any large language model (LLM). Unlike existing open-source solutions, our Granite Guardian models provide comprehensive coverage across a wide range of risk dimensions, including social bias, profanity, violence, sexual content, unethical behavior, jailbreaking, and hallucination-related issues such as context relevance, groundedness, and answer accuracy in retrieval-augmented generation (RAG) scenarios. Trained on a unique dataset combining diverse human annotations and synthetic data, Granite Guardian excels in identifying risks often overlooked by traditional detection systems, particularly jailbreak attempts and RAG-specific challenges. https://github.com/ibm-granite/granite-guardian

## introduction the responsible deployment of large language models (llms) across diverse applications requires robust risk detection models to mitigate potential misuse and ensure safe operation. given the inherent vulnerabilities of llms to various threats and safety risks, detection mechanisms that can filter user inputs and model outputs are essential components of a secure system. model-driven safeguards built on a well-defined risk taxonomy have emerged as an effective approach for mitigating these risks. these models serve as adaptable, plug-and-play components across a wide range of use cases. examples include using them as guardrails for real-time moderation, acting as evaluators to assess the quality of generated outputs, or enhancing retrieval-augmented generation (rag) pipelines by ensuring groundedness and relevance of answers. developing highperformance detection models that address a broad spectrum of risks is crucial for ensuring the safe use of llms. moreover, transparency in the development and deployment of these models can spread trust and accountability in their operation. to address these challenges, we present granite guardian, a family of risk detection models derived from the granite 3.0 language models (granite team, 2024). it makes several key contributions: (i) it is the first model family (2b and 8b sizes) to address unified risk detection by incorporating function calling hallucination, context relevance, groundedness, and answer relevance in rag pipelines; (ii) leverages a combination of diverse, high-quality human-annotated and synthetic datasets to enhance resilience against adversarial attacks and hallucinations; (iii) delivers competitive performance, achieving top-tier results on multidimensional tasks. our paper is organized as follows. we outline the various harms and risks addressed, as well as the risk taxonomy underlying granite guardian, in section 2, training data and synthetic data generation in section 3, and model development in section 4. section 5 provides extensive benchmark evaluations, demonstrating our model's effectiveness across multiple risk dimensions foot_0 . to address such risks, moderation-based strategies -commonly referred to as "guard" or "guardrails" -have emerged as promising solutions. originally developed to enhance social media safety, these approaches have been adapted to improve the safety of llms. existing work on "guard" frameworks can be broadly categorized into two areas: (i) models designed to address general safety concerns, such as harmful or biased content, and (ii) models specifically targeting the rag triad: context-relevance, groundedness, and answer relevance. the first category includes model families such as llamaguard @xcite and shieldgemma @xcite , which also enable detection across different risk dimensions. while these models share broad objectives, like they output label tokens (yes/no or unsafe/safe) to indicate the presence of risks, while differing in subtle but important ways, such as variations in prompt templates and risk definitions. additionally, some models take a more modular approach to risk detection, such as the llama family, which includes an independent promptguard model for addressing jailbreaks and prompt injections. many of these models rely on native capabilities of their base models for extensions like zero-shot, few-shot detection or the flexibility to use token probabilities to model detection confidence. the definition of safety and risk dimensions varies based on the taxonomy that the model targets and its intended application. for example, llamaguard is optimized for conversational ai environments, whereas shieldgemma is designed for policy-specific deployments. furthermore, other approaches like wildjailbreak @xcite emphasize the use of high-quality synthetic data that extends beyond simple harmful prompts and responses, addressing adversarial intent with contrastive samples within its scope. the second category focuses on the rag-triad with models addressing the related risks. notable models in this category include adversarial nli @xcite , wecheck @xcite , and minicheck @xcite . @xcite train a t5-model on the adversarial natural inference inference (anli) dataset which comprises context, label, and a corresponding human created hypothesis which is crafted to fool the detection model into misclassification. the wecheck model is trained on synthethic data comprising of llm's responses to a given text. the labels are derived via multiple labelling models. the model is first pre-trained on nli datasets and then finetuned on the synthetic data in a noise-aware fashion. minicheck first decomposes the given response into several atomic facts and generates a score for each sentence based on how well it is supported by the context. it then aggregates the scores for all the atomic facts in the response and predicts if the response is grounded or not. minicheck is also trained on synthetic data composed of contexts, atomic facts and the label indicating whether the fact is grounded in the context or not. ## experimental results probability computation: language modelbased guardrails generally assign probability by considering the token generation probability of the corresponding safe and unsafe token given the input and then normalizing across the two via softmax operation. we propose a more robust probability computation for binary classification purposes. we aggregate the logits value of different variations of the safe and unsafe token logits score and then compute the overall probabilities. the probabilities for the safe and unsafe labels are computed as follows: llama-guard @xcite and shieldgemma @xcite . ## conclusion this work introduces the granite guardian family, a suite of safeguards for prompt and response risk detection. it addresses diverse risks, including rag-specific issues like context relevance, groundedness, and answer relevance, as well as jailbreaks and custom risks, tailored for enterprise use cases. granite guardian models can integrate with any llms and outperform competitors on benchmarks, supported by transparent training with diverse human annotations to ensure inclusivity and robustness. released as open-source , these models provide a foundation for advancing responsible and reliable ai systems.