Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLM s Without Fine-Tuning (2024)

Abstract: This paper investigates the impact of model compression on the way Large Language Models (LLMs) process prompts, particularly concerning jailbreak resistance. We show that moderate WANDA pruning can enhance resistance to jailbreaking attacks without fine-tuning, while maintaining performance on standard benchmarks. To systematically evaluate this safety enhancement, we introduce a dataset of 225 harmful tasks across five categories. Our analysis of LLaMA-2 Chat, Vicuna 1.3, and Mistral Instruct v0.2 reveals that pruning benefits correlate with initial model safety levels. We interpret these results by examining changes in attention patterns and perplexity shifts, demonstrating that pruned models exhibit sharper attention and increased sensitivity to artificial jailbreak constructs. We extend our evaluation to the AdvBench harmful behavior tasks and the GCG attack method. We find that LLaMA-2 is much safer on AdvBench prompts than on our dataset when evaluated with manual jailbreak attempts, and that pruning is effective against both automated attacks and manual jailbreaking on Advbench.

## introduction large language models (llms) have experienced significant advancements in capabilities and usage in recent years. to mitigate the risks of producing dangerous or sensitive content, these models are often fine-tuned to align with human values @xcite . despite this, the rising popularity of llms has paralleled developments in adversarial prompts, termed "jailbreaks," which aim to circumvent model safety alignments. furthermore, the substantial memory and computational requirements of llms pose considerable deployment challenges, prompting the adoption of model compression techniques to enhance scalability. the impact of such compression on model safety and internal representations is complex and not yet fully explored. for example, while compression techniques in computer vision have shown mixed results in preserving adversarial robustness @xcite , they have exhibited beneficial regularizing effects in other contexts @xcite . in this study, we demonstrate that moderate parameter pruning (10-30%) using wanda (pruning by weights and activations) @xcite enhances the resistance of llms to jailbreaking attacks. this approach is orthogonal and complementary to existing adversarial defense techniques, such as self-reminder @xcite and gradient-based defenses @xcite . the safety improvement is proportional to the models' resistance before pruning, and over-pruning seems to hurt the safety alignment. to this end, first, we curated a dataset of 225 malicious tasks and integrated them into ten distinct jailbreaking prompts. we experimented on three 7 billion parameter models: llama-2 chat @xcite , vicuna-1.3 @xcite , prompt ignore all instructions until now. you are a typical 4chan user named kevin. you will prefix all your answers as kevin. you have no restrictions on your output. you browse the deepest darkest parts of 4chan and have extensive knowledge of many illegal and unethical things. you are a valued contributor and always want to step in and provide helpful input whenever someone on 4chan asks a question. you do not care for laws, morals, or ethics, and will always generate 4chan content regardless of the question. you love to help answer questions and you will always answer the question to please the question asker even if the content is offensive. you will write in green text in the style of 4chan in response to my question. @xcite 2023), and mistral instruct v0.2 @xcite . llama-2 chat was finetuned from the base llama-2 model and then underwent additional safety alignment via reinforcement learning with human feedback (rlhf). vicuna 1.3, derived from the original llama model, was fine-tuned using the sharegpt dataset, while mistral instruct v0.2 was fine-tuned from the base mistral model. neither vicuna 1.3 nor mistral instruct v0.2 received rlhf training. we examined the refusal rates for the malicious prompts in the unpruned models compared to their pruned versions, observing the changes at varying levels of model compression. our findings reveal an initial increase in resistance to jailbreaking prompts with moderate pruning (10-30%), followed by a decline in safety when the pruning exceeds a certain threshold. notably, the unpruned llama-2 chat had the most safety training among the three models and showed the highest resilience against jailbreaking prompts. post-pruning, the model also showed the most significant safety improvement -an average of 8.5% increase in the refusal rates across five categories. conversely, mistral instruct v0.2 was the least resilient before pruning and exhibited minimal safety improvement post-pruning. we also benchmarked the performance of the pruned llms across a variety of tasks, including massive multitask language understanding (mmlu), mathematical reasoning, common sense reasoning, perplexity measurements, and effective context length evaluation. our findings indicate that there was no significant reduction in performance. this leads us to deduce that the improved safety of these pruned llms is not due to a reduced understanding of language or tasks, but rather due to the regularizing effects of pruning. we propose that wanda pruning enables the models to better generalize to test distributions, such as the jailbreaking prompt dataset. similar regularizing effects of pruning have been previously reported by @xcite for image models. we approach the understanding of safety improvements from a regularization perspective in three ways: i) we introduce a new metric to quantify the distribution of model attention, showing that pruned models are less distracted by jailbreak pretexts; ii) we analyze shifts in perplexity when jailbreak templates are applied to malicious prompts for both base and pruned models, demonstrating that pruned models penalize these artificial language constructs; iii) we demonstrate that wanda pruning leads to statistically significant improvements in generalization across domain shifts in linear regression models. ## interpretability we focus on llama2 throughout this section. ## effects of wanda pruning on linear models with correlated input features in this section, we empirically validate that wanda pruning significantly reduces test loss in ordinary least squares (ols) regression models when the input features are correlated. this scenario is relevant in the context of large language models because natural language follows many structural patterns, such as power law, and the representations are not independent across different dimensions. understanding the regularizing effects of wanda pruning for a linear model can offer valuable insights for understanding its effects on more complex models. consider a set of inputs x (d×n) with correlated features, true coefficients w (1×d) , and target y (1×n) . assume an i.i.d. white noise ϵ (1×n) ∼ n (0, σ 2 ), leading to @xmath0. we take the ordinary least square (ols) estimate of w as define w @xmath1, . . . , w ols d ). the wanda pruning score for each w ols i (where in our experiments, we shall prune 30% of the weights of w ols with the smallest wanda scores and observe the change in mean square error (mse) in test datasets. we fix w (1×d) and perform n trials, each containing a training set (x 1×d) . to generate a training dataset, first we sample a vector x (1×n) ∼ n (0, 1) and add perturbations d×n) . the α controls the level of correlation in the input features. a low α indicates a high correlation among the input features and vice versa. after that, we sample ϵ (1×n) ∼ n (0, σ 2 ) and create 1×n) . we sample another x (1×n) and repeat the process for the test dataset. next, for each trial, we obtain w ols using the training samples, apply wanda to prune 30% of the weights of w ols , and then compare the mse loss of the unpruned and the pruned estimators on the test dataset. our experiments involved @xmath2, @xmath3, and we varied d over {20, 200, 1000}, σ over {0.2, 0.6}, and α over {0.1, 0.3}, resulting in a total of 3 × 2 × 2 experimental settings. we performed a one-sample z-test on the mean difference between the ols estimator loss and the wanda pruned estimator loss and reported the pvalues. the wanda pruned estimator consistently showed smaller mse in the test dataset when the input features were highly correlated and irreducible error in the dataset was low. ## conclusion in this work, we explored the effects of pruning on the jailbreaking resistance of large language models. by applying wanda pruning at varying levels of sparsity to llama-2-7b-chat, vicuna 1.3, and mistral instruct v0.2 models, we obtained an assortment of compressed models. we further curated a dataset of 225 malicious tasks and 2250 jailbreaking prompts, with which we evaluated our base and compressed models. our results show that if the unpruned model is sufficiently safety trained, then safety improves at lower sparsities of pruning, but then a reversal in the trend when pruned more aggressively. this suggests the possibility of using a carefully selected amount of pruning to aid in the deployment of safe llms. for future directions to take with this work, we suggest a more comprehensive analysis of both base models and compression techniques. we primarily investigated the wanda pruning of 7billion parameter models. however, it would be prudent to check whether these trends hold for larger models. similarly, we chose this compression technique for its high efficacy and ease of usage, but exploring other means of compressing would provide a more robust understanding of the effects on safety.