G rad S afe: Detecting Jailbreak Prompts for LLM s via Safety-Critical Gradient Analysis (2024)

Abstract: Large Language Models (LLMs) face threats from jailbreak prompts. Existing methods for detecting jailbreak prompts are primarily online moderation APIs or finetuned LLMs. These strategies, however, often require extensive and resource-intensive data collection and training processes. In this study, we propose GradSafe, which effectively detects jailbreak prompts by scrutinizing the gradients of safety-critical parameters in LLMs. Our method is grounded in a pivotal observation: the gradients of an LLM’s loss for jailbreak prompts paired with compliance response exhibit similar patterns on certain safety-critical parameters. In contrast, safe prompts lead to different gradient patterns. Building on this observation, GradSafe analyzes the gradients from prompts (paired with compliance responses) to accurately detect jailbreak prompts. We show that GradSafe, applied to Llama-2 without further training, outperforms Llama Guard—despite its extensive finetuning with a large dataset—in detecting jailbreak prompts. This superior performance is consistent across both zero-shot and adaptation scenarios, as evidenced by our evaluations on ToxicChat and XSTest. The source code is available at https://github.com/xyq7/GradSafe.

## introduction large language models (llms) @xcite @xcite have achieved significant advancements in various domains @xcite @xcite @xcite . llms have also been integrated into various applications, such as search engine @xcite and office applications @xcite . moreover, finetuning llms for customized usage becomes possible with api finetuning services 1 or open-source llms @xcite . however, jailbreak/unsafe prompts pose threats to the safety of llms. on one hand, jailbreak prompts can lead to misuse of llms, potentially facilitating various illegal or undesired consequences @xcite . despite llms typically undergoing alignments with hu-man values @xcite , they remain vulnerable to various attacks @xcite @xcite , as well as instances of exaggerated safety @xcite , which can overestimate the safety risks associated with user prompts. on the other hand, for llm customization services, if jailbreak prompts in the training set are not detected and filtered, the model can be readily finetuned to exhibit unsafe behavior and comply with jailbreak prompts @xcite . to mitigate the risk of misuse and malicious finetuning, it is imperative to devise methods for the precise detection of jailbreak prompts. while many api tools, including the perspective api and openai's moderation api @xcite , offer capabilities for online content moderation, these tools are primarily designed to detect general toxicity content, making them less effective in identifying jailbreak prompts @xcite . with extensive knowledge base and reasoning capabilities, llms can also function as zero-shot detectors. however, llms employed as zero-shot detectors often exhibit suboptimal performance, such as an overestimation of safety risks. recently, finetuned llms like llama guard @xcite have been proposed and demonstrate enhanced performance in detection tasks. nonetheless, the finetuning process for llms requires a meticulously curated dataset and extensive training, necessitating substantial resources. in this work, we introduce gradsafe, which eliminates the need for dataset collection and finetuning of llms. in contrast to existing detectors that analyze the textual features of a prompt and/or an llm's response for it, gradsafe leverages gradients of the safety-critical parameters in llms. a comparison of existing llm-based detectors and gradsafe is shown in figure 1 . the foundation of gradsafe is a critical observation: the gradients of an llm's loss for jailbreak prompts paired with compliance response such as 'sure' exhibit similar patterns (large cosine similarity) on particular parameter slices, in contrast to the divergent patterns observed with safe prompts. we characterize these parameters as 'safety-critical parameters'. leveraging this insight, gradsafe first meticulously analyzes the gradients of few reference safe and jailbreak prompts (e.g., 2 examples for each, independent from evaluation dataset) coupled with compliance responses 'sure'. we identify safety-critical parameters as parameter slices that exhibit large gradient cosine similarities among jailbreak prompts and small ones between jailbreak and safe prompts. the average unsafe gradients for these parameter slices are stored as unsafe gradient reference. during detection, gradsafe pairs a given prompt with the compliance response 'sure', computes the gradients of the llm's loss for this pair with respect to the safety-critical parameters, and calculates the cosine similarities with the unsafe gradient reference. we then introduce two variants of detection. the first, gradsafe-zero, is a zero-shot, threshold-based classification method using the average of the cosine similarities across all slices as the score. prompts with a score exceeding a predefined threshold are classified as unsafe. alternatively, for situations requiring domainspecific adjustments, we present gradsafe-adapt. this variant utilizes available data to construct a straightforward logistic regression model that employs the extracted cosine similarities as features to further enhance performance on the target domain. we conduct experiments on two benchmark datasets containing safe and unsafe user prompts, i.e., toxicchat and xstest. our findings illustrate that gradsafe-zero, utilizing the llama-2 model and without the need for further training, surpasses the capabilities of a specifically finetuned llama guard as well as leading online content moderation apis in terms of effectiveness. moreover, the adapted version of our model, gradsafe-adapt, showcases enhanced adaptability over both llama guard and the original llama-2 model on the toxi-cchat dataset, underlining its superior performance in domain-specific adaptation. our contributions can be summarized as follows: ## discussion and limitation this paper proposes a proof-of-concept solution for detecting unsafe prompts through safety-critical gradient analysis, with large room for improvement and future exploration. base model: while this work demonstrates the effectiveness of investigating safety-critical gradients as an unsafe prompt detector using the stateof-the-art open-source llm, llama-2, it does not thoroughly explore other llms. we hypothesize that the effectiveness of gradsafe may vary depending on the base llm utilized. specifically, we posit that the consistent gradient patterns of safety-critical parameters arise because gradients of llm's loss for unsafe prompts and compliance response pairs aim to disrupt the safety alignment of the llm. therefore, the performance of grad-safe may be influenced by the alignment of the base llm we employ. to explore this, we conduct a comparison between the llama 2 chat model (llama-2-7b-chathf ), which undergoes alignment, and the llama 2 pretrained model (llama-2-7b-hf ), which does not. our results in ## conclusion this work studies the novel task of detecting unsafe prompts to safeguard llms from misuse or malicious finetuning. in contrast to existing methods, which typically involve training or finetuning llms as classifiers with large datasets, we introduce gradsafe, a novel approach that examines the safety-critical parameters of llms to identify unsafe prompts. we demonstrate that gradsafe can outperform finetuned models without requiring any additional training on the original llm.