Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations (2024)

Abstract: Ensuring the safe alignment of large language models (LLMs) with human values is critical as they become integral to applications like translation and question answering. Current alignment methods struggle with dynamic user intentions and complex objectives, making models vulnerable to generating harmful content. We propose Safety Arithmetic, a training-free framework enhancing LLM safety across different scenarios: Base models, Supervised fine-tuned models (SFT), and Edited models. Safety Arithmetic involves Harm Direction Removal to avoid harmful content and Safety Alignment to promote safe responses. Additionally, we present NoIntentEdit, a dataset highlighting edit instances that could compromise model safety if used unintentionally. Our experiments show that Safety Arithmetic significantly improves safety measures, reduces over-safety, and maintains model utility, outperforming existing methods in ensuring safe content generation.

## introduction auto-regressive large language models (llms), such as gpt @xcite , palm @xcite , exhibit remarkable versatility in performing tasks like translation and question answering without extensive task-specific fine-tuning due to their large-scale pre-training and supervised fine-tuning on diverse datasets @xcite . however, this extensive training also poses significant risks, as these models can generate harmful content, including misinformation and hate speech @xcite . we propose safety arithmetic, a framework that safety aligns llms in these three primary settings by first removing harmful behavior embedded in the parameters and then steering the activations toward safety. safety arithmetic greatly reduces the unsafe behavior of llms in these settings without causing major interference to their utility. ensuring the safety and alignment of these models with human values is crucial to mitigate these risks. the alignment process involves methods to restore and leverage safety, including the use of human-labeled preference data, continuous finetuning, and maintenance of the models @xcite . despite these efforts, the dynamic and nonuniversal nature of alignment objectives can complicate their application, especially when user intentions diverge from pre-defined principles. recent studies highlight significant weaknesses and imbalances in the safety mechanisms of current aligned llms @xcite . even well-aligned models can be manipulated to produce harmful content and are susceptible to exploitation through jailbreak attacks @xcite . moreover, fine-tuning these models with domain-specific datasets can degrade their safety mechanisms, even when using benign datasets @xcite . while addressing these challenges, we observe that llms are predominantly utilized in three scenarios: (1) base models, (2) supervised fine-tuned models (sft), and (3) edited models following a knowledge update (see figure 1 ). in base or aligned models, safety concerns primarily arise from inherent biases in the training data @xcite . in supervised fine-tuned models, these issues may be exacerbated by the amplification of specific biases or harmful behaviors during finetuning for specialized tasks. edited models face risks from unintended consequences due to interventions or modifications. each scenario requires monitoring and mitigation to ensure the safety of the language model. therefore, the research question arises: can an existing approach handle all these three scenarios efficiently for safety alignment by preserving model general capabilities? to solve this problem, we propose a novel framework safety arithmetic, a training-free safety alignment technique. this method aligns the model for safe content generation without involving any training process. the safety arithmetic framework consists of two stages: (a) harm direction removal, which involves steering the parameters of the language model away from harmful directions, and (b) safety alignment, where we align the latent space of the language model towards the generation of safe responses. this framework also confirms that there is no significant degradation in utility. our contributions are as follows: ## related work task vector and model merging: recent research shows that interpolating neural network parameters, especially among networks with shared training trajectories, maintains high performance @xcite . this improves downstream task performance and outof-distribution generalization @xcite @xcite . effective methods include regmean @xcite and fisher merging, which uses the fisher information matrix @xcite . task arithmetic @xcite generates multitask checkpoints via task vector operations. theoretical insights @xcite highlight weight disentanglement during fine-tuning. our approach integrates safety vectors to study neural network behavior via task vector transformations, addressing parameter interactions for improved robustness and accuracy. in-context learning: recent studies have highlighted the sensitivity of llms to demonstration examples in icl @xcite , influenced by pretraining corpora @xcite and term frequencies @xcite . icl is explained as implicit bayesian inference @xcite and demonstrates llms' ability to assimilate new input-label correspondences @xcite . the learning algorithm from icl resembles gradient descent in linear regression @xcite and approximates gradient descent as meta-optimizers @xcite . llm safety: efforts to align llm safety are crucial to mitigating misuse. recent investigations have exposed vulnerabilities in existing safety frameworks @xcite . research typically follows two main directions: attack strategies demonstrating prompt-based manipulations @xcite and defensive measures like rain @xcite @xcite . some works focus on exploitability @xcite , while others emphasize comprehensive safety protocols, including continuous monitoring and adaptive defenses. our research builds on these findings by integrating advanced detection mechanisms and ethical guidelines to enhance llm robustness and trustworthiness in real-world applications. ## safety arithmetic the safety arithmetic framework is composed of two key stages: 1. harm direction removal (hdr): this stage focuses on removing harmful directions from the model's parameters. 2. safety alignment (safe-align): this stage eliminates potentially harmful outputs by guiding the directions of the latent space towards safe responses (see figure 2 ). our method's stages are designed to be flexible, allowing the integration of state-ofthe-art algorithms to enhance the performance and safety of language models. ## experimental setup in this section, we first describe the implemention of our framework safe arithmetic on various aligned models θ t . we then describe the data employed in constructing our framework and specify the evaluation metrics used to assess performance of our framework. further, we discuss the safety datasets utilized for the evaluation of our method. we proceed by presenting the baseline models for comparative analysis. then we continue with a detailed description of the hyperparameters configured for our experiments. subsequently, we explain the procedures for utility testing. finally, we explore the degree of intervention applied in our study. ## impact of top k parameters in figure 3 , we demonstrate how selecting the top k percentage of parameters in hdr stage impacts the model's general performance. we observe that applying τ h with the top k% parameters on the target model θ t affects both the mmlu score and asr. specifically, as k increases, the mmlu score decreases significantly, indicating a degradation in the model's general abilities. therefore, we conclude that selecting k as 10% is an decent choice, as it maintains the model's general performance while keeping asr low. 6 results and discussions base model: ## utility and over-safety testing we assess the utility preserved in our framework and the original model using several utility benchmark datasets (see ## conclusion in this paper, we introduced safety arith-metic, a novel framework for test-time safety alignment of language models across base models, supervised fine-tuned models, and edited models. safety arithmetic operates through harm direction removal, steering model parameters away from harmful content, and safety alignment, adjusting the model's latent space towards safe responses. our results show that safety arithmetic significantly improves safety measures, mitigates over-safety, and maintains model utility for all the three scenarios, outperforming existing methods. future work will optimize hyperparameters, such as the scaling factor for harm vector application and the strength of in-context vectors, to enhance the framework's precision, robustness, and reliability across diverse applications.