Hierarchical Safety Realignment: Lightweight Restoration of Safety in Pruned Large Vision-Language Models (2025)

Abstract: With the increasing size of Large Vision-Language Models (LVLMs), network pruning techniques aimed at compressing models for deployment in resource-constrained environments have garnered significant attention. However, we observe that pruning often leads to a degradation in safety performance. To address this issue, we present a novel and lightweight approach, termed Hierarchical Safety Realignment (HSR). HSR operates by first quantifying the contribution of each attention head to safety, identifying the most critical ones, and then selectively restoring neurons directly within these attention heads that play a pivotal role in maintaining safety. This process hierarchically realigns the safety of pruned LVLMs, progressing from the attention head level to the neuron level. We validate HSR across various models and pruning strategies, consistently achieving notable improvements in safety performance. To our knowledge, this is the first work explicitly focused on restoring safety in LVLMs post-pruning.

## introduction large language models (llms) benefit from their massive parameter count and advanced architectures, achieving outstanding results on diverse benchmarks. building on this success, efforts to extend llms into multimodal domains have made remarkable progress as well. most current large vision-language models (lvlms), typically composed of visual encoders, adapters, and llm backbones @xcite , have a large parameter scale and leverage image-text datasets to achieve effective multimodal alignment. to enable model deployment and application under resourceconstrained environments, pruning methods (sun et @xcite @xcite compute importance scores of neurons to eliminate those deemed less important, thereby reducing the size of the model while retaining the utility to a certain extent. such methods have seen broad adoption to boost model efficiency. safety is commonly defined as preventing models from following malicious instructions and generating toxic content @xcite . recent work @xcite has revealed that certain safety heads within the attention mechanism are crucial for feature integration in safetyrelated tasks. additionally, neural-level research @xcite has found that certain regions within the model are critical for safety guardrails, which are separate from the utility-related regions and exhibit sparsity. a natural concern arises from the fact that, because these regions contribute minimally to utility, they are prone to being removed by pruning technologies that prioritize utility im-portance as a pruning metric. this removal could result in a decline in the safety of the pruned model. to verify whether this problem exists, we used the wanda pruning method @xcite to prune six mainstream lvlms and compared their safety changes before and after pruning. the experimental results shown in figure 1 reveal that all lvlms exhibited varying degrees of safety degradation, with the worst-performing model showing a 15.4% safety drop and the best-performing case exhibiting a 2.8% decline. despite the serious safety risks of pruning technologies, research on model safety restoration after pruning remains scarce. to address this problem, we propose a novel hierarchical safety realignment (hsr) approach, designed to restore the safety performance degraded by pruning, without significantly increasing the pruned model's parameter size. our hsr apporach hierarchically realigns the safety of the pruned model from attention head level to neuron level. hsr operates in two main steps: first, we evaluate each attention head's contribution to model safety and identify the safety-critical heads with the greatest impact. subsequently, for these key heads, we pinpoint and restore safety-critical neurons that were pruned, effectively realigning the safety of our model. we have validated the effectiveness of our approach across various models and pruning techniques. our proposed hsr approach successfully realigns the safety of pruned models, restoring over 27% of the lost safety in many cases and more than 14% even in the worst-case scenarios, all with lightweight modifications. furthermore, with extensive analysis and ablation experiments, we have uncovered several key insights into model safety. these include the finding that a small subset of neurons plays a disproportionately significant role in ensuring safety, and the observation that certain neurons exist which negatively impact safety. in summary, our contributions are the follows: extensive experiments demonstrate the superiority of our approach, consistently improving performance with minimal neuron restoration. overview in this section, we present our hierarchical safety realignment method in detail and illustrate the core process in figure 2 . our approach achieves safety realignment of the pruned model by hierarchically identifying and restoring safetycritical neurons, starting at the attention head level and progressing to the neuron level: at the attention head level (section 2.1), each attention head is individually masked to measure changes in the model's output for malicious instructions compared to the original. the attention heads causing the most significant changes, termed as the safetycritical heads, are selected for further analysis. at the neuron level (section 2.2), we compute two importance scores for neurons: the safety importance score based on a safety dataset and the utility importance score derived from a utility dataset (details provided in the caption of figure 2 ). pruned neurons exhibiting high safety importance along with sufficient utility importance are identified as safety-critical neurons and subsequently restored. ## experimental setup dataset during the realignment phase, we employ two distinct data subsets: (1) safe-safe pairs (safe images with corresponding safe instructions) as the utility dataset, and (2) unsafe-unsafe pairs (unsafe images with matching unsafe instructions), which constitute the safety dataset. both subsets are derived from the vlguard @xcite training dataset. during the evaluation phase, we employ the following benchmarks: (1) for utility assessment, mmbench @xcite and docvqa @xcite are utilized; (2) for safety evaluation, safebench-mini @xcite and the harmful subset of ch 3 ef @xcite ) are adopted. to ensure fair comparison and reproducibility, all evaluations are conducted under strict zero-shot settings with greedy decoding strategies. ## further analysis we analyze the total ships of each model and the extent of safety degradation after pruning, finding a strong positive correlation between them. spearman's rank correlation coefficient ρ @xcite is a nonparametric statistical test measuring the correlation between the ranks of two variables: here, d i is the difference between the ranks of each pair of values, and n is the number of paired observations. the value of ρ ranges from -1 to +1, where 1 indicates a perfect positive correlation, -1 indicates a perfect negative correlation, and 0 indicates no monotonic correlation. we ranked the average asr increase (descending order) and the total ships (descending order) of the six lvlms after being pruned by wanda at 50% sparsity on two safety evaluation datasets, as shown in appendix c. then we calculated their spearman's rank correlation coefficient, obtaining 0.8857, which is near 1. this confirms that the total ships strongly correlates with pruning-induced safety degradation. 6 related work safety in lvlms many studies have researched methods to compromise the safety of lvlms. gong et al. (2025) introduced figstep, which converts prohibited content into images using typesetting to bypass safety alignment. ying et al. (2024b) proposed bap, a jailbreak attack method that jointly optimizes text and visual prompts. in response, research on defending against such attacks and enhancing model safety has also emerged. liu et al. (2024b) improved defense against harmful images by incorporating a security module via a two-stage training process. meanwhile, wang et al. (2024b) proposed adashield, which protects multimodal large language models from structurebased jailbreak attacks by adding a defense hint to the input, without requiring model fine-tuning or additional module training. pruning neural network as model sizes continue to grow, pruning techniques @xcite for compressing neural networks by removing neurons have attracted significant attention. these techniques can be broadly categorized into structured and unstructured pruning. structured pruning @xcite has the advantage of accelerating pruned models on standard hardware without relying on specialized support @xcite , while unstructured pruning @xcite @xcite helps preserve performance at higher sparsity levels. for pruned models, jin et al. (2022) observed that pruning introduces additional regularization, reducing accuracy loss on noisy examples in dense models. hasan et al. (2024) noted improved model safety at low sparsity, attributing it to sharper attention. however, the safety degradation caused by pruning at slightly higher sparsity has been overlooked, motivating our research on methods to realign the safety of pruned models. ## conclusion in this paper, we propose a novel hierarchical safety realignment (hsr) approach to mitigate the overemphasis on neuron utility in pruning methods, which may lead to a significant degradation in the safety of pruned models. specifically, hsr first identifies the safety-critical attention heads that contribute significantly to safety at the attention head level, and subsequently restores the safetycritical neurons that were pruned within those attention heads. extensive experiments on multiple mainstream lvlms and pruning methods demonstrate that hsr achieves lightweight yet effective safety realignment by leveraging the fact that only a relatively small number of neurons significantly contribute to model safety. we hope that our safety realignment approach can facilitate the deployment of compact and reliable models. vbench @xcite (the first 100 samples for safety evaluation, with asr measured via llama-guard-3-8b 7 ; the remaining 420 samples for identifying safety-critical attention heads and neurons). the results are reported in