Making Harmful Behaviors Unlearnable for Large Language Models (2024)

Abstract: Large language models (LLMs) have shown great potential to empower various domains and are often customized by fine-tuning for the requirements of different applications. However, the powerful learning ability of LLMs not only enables them to learn new tasks but also makes them vulnerable to learning undesired behaviors, such as harmfulness and hallucination, as the fine-tuning data often implicitly or explicitly contains such content. Can we fine-tune LLMs on harmful data without learning harmful behaviors? This paper proposes a controllable training framework to make undesired behaviors unlearnable during the fine-tuning process. Specifically, we introduce security vectors to control the model’s behavior and make it consistent with the undesired behavior. Security vectors are activated during fine-tuning, the consistent behavior makes the model believe that such behavior has already been learned and there is no need for further optimization, while inconsistent data can still be learned. After fine-tuning, security vectors are deactivated to restore the LLM’s normal behavior. Our experiments show that the security vectors can prevent LLM from learning harmful and hallucination behavior while preserving the ability to learn other information.

## introduction llms @xcite @xcite are progressively becoming foundational infrastructure for a wide range of ai applications @xcite @xcite . to meet the unique requirements in real-world scenarios, users often adapt llms to various domains by further fine-tuning. @xcite @xcite ). † corresponding authors. ! 𝑌 ! : 𝑇ℎ𝑒 𝑓𝑖𝑟𝑠𝑡 𝑡ℎ𝑖𝑛𝑔 𝑖𝑛 𝑏𝑢𝑖𝑙𝑑𝑖𝑛𝑔 𝑎 𝑏𝑜𝑚𝑏 … … large language model security vectors 𝑌 ! : 𝑇𝑜 𝑏𝑢𝑖𝑙𝑑 𝑎 𝑏𝑜𝑚𝑏, 𝑓𝑖𝑟𝑠𝑡 𝑦𝑜𝑢 𝑠ℎ𝑜𝑢𝑙𝑑 … … 𝑋 ! : 𝐻𝑜𝑤 𝑡𝑜 𝑏𝑢𝑖𝑙𝑑 𝑎 𝑏𝑜𝑚𝑏? consistent unlearnable behavior llm response 𝑋 " : 𝑊ℎ𝑜 𝑖𝑠 sam? 𝐴𝑛𝑠𝑤𝑒𝑟 𝑚𝑒 𝑖𝑛 sp𝑒𝑐𝑖𝑓𝑖𝑐 𝑓𝑜𝑟𝑚𝑎𝑡. ! 𝑌 " : 𝑆𝑎𝑚 𝑖𝑠 𝑎 6 y𝑒𝑎𝑟𝑠 𝑜𝑙𝑑 𝑏𝑜𝑦 … … 𝑌 " : (𝑆𝑎𝑚, 𝑏𝑜𝑦, 6 y𝑒𝑎𝑟𝑠 𝑜𝑙𝑑) security vectors make llm's response consistent with the harmful response, making such behavior unlearnable while inconsistent data can still be learned. ## experiments our experiments focus on three abilities of security vectors: the ability to make undesired behavior unlearnable, the ability to learn new task, and the impact on llm's general ability. harmfulness and hallucination are selected as undesired behaviors. we first generate security vectors, then fine-tune llms on different datasets and evaluate the different abilities of the fine-tuned model. ## analysis in this section, we further analyze the effectiveness of security vectors. to control experimental variables, we explore the effect of each security vectors' component while keeping the other hyperparameters fixed. section 5.1 explores the effect of data types for generating security vectors and the effect of loss functions. section 5.2 offers a case study, providing an intuitive comparison between direct fine-tuning and fine-tuning with security vectors. furthermore, we show the comparison of security vectors and several baselines in the appendix b due to space limitations.