S afe Q uant: LLM Safety Analysis via Quantized Gradient Inspection (2025)

Abstract: Contemporary jailbreak attacks on Large Language Models (LLMs) employ sophisticated techniques with obfuscated content to bypass safety guardrails. Existing defenses either use computationally intensive LLM verification or require adversarial fine-tuning, leaving models vulnerable to advanced attacks. We introduce SafeQuant, a novel defense framework that leverages quantized gradient patterns to identify harmful prompts efficiently. Our key insight is that when generating identical responses like “Sure”, LLMs exhibit distinctly different internal gradient patterns for safe versus harmful prompts, reflecting conflicts with safety training. By capturing these patterns through selective gradient masking and quantization, SafeQuant significantly outperforms existing defenses across multiple benchmarks while maintaining model utility. The method demonstrates particular effectiveness against sophisticated attacks like WordGame prompts and persuasive adversarial attacks, achieving an F1-score of 0.80 on WordGame dataset and outperforming state-of-the-art (SoTA) methods like GradSafe by an absolute margin of 57%.

## introduction the rapid evolution and widespread adoption of large language models (llms) @xcite @xcite across domains @xcite , from code completion @xcite to healthcare @xcite , has highlighted critical safety challenges. while these models excel at few-shot learning @xcite , they remain vulnerable to jailbreaking @xcite , that are manipulation techniques that bypass safety guardrails to elicit harmful or unethical responses. despite alignment efforts @xcite , llms remain susceptible to sophisticated attacks @xcite . early defenses relied on surfacelevel analysis through rephrasing or perplexitybased filtering @xcite , but these methods fail against carefully crafted prompts. recent gradient-based approaches like gradsafe @xcite and gradient cuff @xcite leverage model internals for detection, with grad-safe using reference prompt pairs that may not generalize well, and gradient cuff analyzing refusal loss landscapes that sophisticated attacks might circumvent while maintaining high refusal probabilities. ## proposed method in this section, we explore how analyzing the gradients of an llm's parameters can help detect harmful prompts. the complete pipeline of safequant is illustrated in figure 2 . in steps 1-3, we obtain the llm's forward activations and backward gradients for the corresponding input prompts. next, our method selects a subset of informative gradients from the complete set of gradients in step 4. in steps 5-7, we develop a novel quantization approach to compress the informative gradients to enable efficient analysis. finally, in step 8, using the quantized gradient information, we train a classifier to discriminate between benign and harmful prompts. following this, we integrate our trained classifier into the llm response pipeline as a safety filter that blocks harmful prompts to the llm. ## conclusion we presented safequant, a novel defense framework for detecting both manually-engineered and algorithmic jailbreak prompts in llms through quantized gradient pattern analysis. our method significantly outperforms existing approaches, achieving 57% and 40% higher f1-scores on wordgame and wildjailbreak datasets respectively, while maintaining the model's original capabilities on mmlu. the framework demonstrates excellent generalization to unseen attack categories and provides strong safety guarantees without compromising model utility. future work could explore extending the framework to multi-modal content, developing adaptive quantization techniques for different types of unsafe content, and optimizing inference through improved parallelization strategies and real-time integration. through extensive validation, safequant proves to be an effective step toward more secure and reliable language models.