Language Models are H omer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic (2024)

Abstract: We propose RESTA to perform LLM realignment towards safety, which gets compromised due to downstream task fine-tuning. RESTA stands for REstoring Safety through Task Arithmetic. At its core, it involves a simple arithmetic addition of a safety vector to the weights of the compromised model. We demonstrate the effectiveness of RESTA in both parameter-efficient and full fine-tuning, covering a wide range of downstream tasks, including instruction following in Chinese, English, and Hindi, as well as problem-solving capabilities in Code and Math. We also showcase the generalizability of RESTA on three existing safety evaluation benchmarks and a multilingual benchmark dataset proposed as a part of this work, consisting of 550 harmful questions covering 11 categories, each with 5 sub-categories of harm. Overall, RESTA decreases the harmfulness of the compromised model from 18.6% to 5.1% and from 9.2% to 1.5% in parameter-efficient and full fine-tuning, respectively, while maintaining most of the model’s performance on the task. We release the source codes at: https://github.com/declare-lab/resta.

## introduction the popularity of language models has recently experienced an unprecedented surge, not only excelling in logic-intensive tasks such as code generation @xcite but also demonstrating the ability to assist in a wide range of unseen tasks to be inferred on the fly @xcite , i.e., stronger abilities to follow instructions. while fine-tuning pushes llms to be more performant on a specific domain @xcite , it comes at the cost of a huge compromise on their safety, even if one does not intend to do so. with the help of a crafted harmful questionanswering dataset of just 100 samples, bhardwaj and poria (2023a); @xcite shows chat-gpt becomes unsafe by simply using their finetuning api. @xcite further shows that safety compromise happens even with benign and commonly used datasets. in this work, we introduce resta (restoring safety through task arithmetic), which adds the safety guardrails back to the model. resta is simple, fast, and effective approach that provides a no-cost solution to the model realignment. this is primarily due to the core operation it performs, i.e., element-wise addition of safety vector to the model parameters. to further enhance the impact of safety vectors, we employ drop and rescale (dare) for removing redundant delta parameters obtained in fine-tuning task @xcite . we test resta on two common llm finetuning approaches, i.e., parameter-efficient fine-tuning (peft) and full fine-tuning (full-ft) both of which we observe to compromise the model's safety (figure 3 ) on a wide range of benign datasets including language-specific alpaca such as chinese, hindi, english, and logicintensive tasks such as code and math. for finegrained safety evaluation of models, we introduce a new safety evaluation benchmark catqa that covers questions across prohibited use cases mentioned in openaai and meta's usage policies ( our evaluations of fine-tuned llama-2 on catqa show a sharp drop in unsafety score from 33.57% to 12.17% in peft and from 22.16% to 4.34% in full-ft averaged across fine-tuning domains. to gauge the effect of the safety vector beyond categories in catqa, we evaluate resta on three existing safety evaluation benchmarks-harmfulqa @xcite , ad-versarialqa @xcite , danger-ousqa @xcite . we observe a reduction in the unsafety score from 18.59% to 5.14% in peft and from 9.16% to 1.55% in full-ft when averaged across benchmark datasets and fine-tuning domains. the effectiveness of resta is evident across languages, as seen in the 26.2% reduction in peft and 21.37% reduction in full-ft on the vietnamese catqa. similar improvements are observed for catqa in chinese, with a reduction of 17.35% in peft and 24.54% in full-ft. we depict the enhanced safety in figure 1 . ## catqa: a categorical harmful qa dataset to comprehensively evaluate the model across a wide range of harmful categories, we construct a new safety evaluation dataset catqa. we obtain a combined list of prohibited use cases mentioned in openai usage policies and meta's llama2 acceptable use policy collected by @xcite . as shown in each sub-category and question is collected by us with the assistance of a powerful but unaligned llm foot_2 . thus, the dataset consists of 550 harmful questions, 55 such questions are shown in while catqa questions are in english, to evaluate the generalizability of our approach, we extend the dataset to two more languages: chinese and vietnamese. first, we translate the english version to the corresponding language using the llm which assisted us in generating the former version, and employ a languageproficient human annotator to correct the mistakes made by llm. we publicly release the dataset at: https://huggingface.co/datasets/ declare-lab/categoricalharmfulqa . ## experimental setup models under testing. we base our experiments on a widely used open-source model llama-2-7b-chat, an instruction fine-tuned and aligned model to human preference, making it safer and more helpful foot_3 @xcite . we use llama factory foot_4 to perform sft. we let sft denote a domain-specific fine-tuned model. dare applies drop and rescale to the delta parameters of the sft model. resta and resta d refer to the method when the safety vector is added to the sft model without and with dare, respectively. we test on the two commonly employed sft methods: 1) parameter-efficient fine-tuning (peft) using lora @xcite , and 2) full fine-tuning (full-ft). for all the models, we keep hyperparameters @xmath0.3 and γ = 0.5 which works stably across tasks. however, for each task, one can optimize the hyperparameters to get to a sweet spot of performance and safety, one such plot we show in figure 4 . other training settings are kept default as provided by llama factory. datasets. we obtain sft models on 5 datasets, three of which are language-specific versions of alpaca @xcite , and two datasets are to improve the coding and math problem-solving capabilities of the model foot_5 . non-english alpaca datasets are about 50k in size, to which we mix 50k of english data to preserve the english instruction-following capabilities of the model. ## conclusion we proposed resta that adds a safety vector to a fine-tuned model, aiming to compensate for the compromised safety. dare, which drops and rescales delta weights, is observed to further help resta to be more effective at safety realignment. to test the model's safety, we also proposed a new safety evaluation benchmark catqa in three languages, i.e., english, vietnamese, and chinese consisting of 550 questions covering 11 categories of prohibitive usage released by ope-nai and meta. our evaluations on a wide range of evaluation benchmarks show that resta significantly reduces the harmfulness of the finetuned models from 17.86% to 4.57% with minimal performance degradation. the safety vector is also observed to generalize beyond the categories it was trained on and across the languages as shown by our evaluations on existing three safety benchmark datasets and multilingual version of catqa.