Towards Safer Large Language Models through Machine Unlearning (2024)

Abstract: The rapid advancement of Large Language Models (LLMs) has demonstrated their vast potential across various domains, attributed to their extensive pretraining knowledge and exceptional generalizability. However, LLMs often encounter challenges in generating harmful content when faced with problematic prompts. To address this problem, existing work attempted to implement a gradient ascent based approach to prevent LLMs from producing harmful output. While these methods can be effective, they frequently impact the model utility in responding to normal prompts. To address this gap, we introduce Selective Knowledge negation Unlearning (SKU), a novel unlearning framework for LLMs, designed to eliminate harmful knowledge while preserving utility on normal prompts. Specifically, SKU is consisted of two stages: harmful knowledge acquisition stage and knowledge negation stage. The first stage aims to identify and acquire harmful knowledge within the model, whereas the second is dedicated to remove this knowledge. SKU selectively isolates and removes harmful knowledge in model parameters, ensuring the model’s performance remains robust on normal prompts. Our experiments conducted across various LLM architectures demonstrate that SKU identifies a good balance point between removing harmful information and preserving utility.

## introduction large language models (llms) @xcite @xcite have demonstrated their exceptional ability across various ai applications @xcite @xcite @xcite @xcite as llms have been trained and fine-tuned on vast amount of textual data @xcite @xcite . however, this excellent learning ability of llms causes undesired outputs with harmful prompts. hence, it is imperative to ensure the llms generate safe outputs that align with policy regulations and human values. however, the current approach of reinforcement learning from human feedback (rlhf) is computationally expensive, and can be problematic with misaligned evaluators @xcite ). an alternative strategy of rlhf is to use machine unlearning @xcite (mu) to "forget" samples that represent those undesirable behaviors during the pre-training process. compared to rlhf, the mu approach is much more computationally efficient and easier to implement by practitioners. different from the traditional unlearning in classification tasks @xcite @xcite , where the goal is to eliminate samples and their influence from both the dataset and trained model, unlearning samples that lead to those unwanted behaviors on llms is rather complicated due to its large quantity of training corpus. besides, the model performance on normal prompts is easily deteriorated by the unlearn-ing process @xcite , which means that llms may have excellent performance on unlearning unwanted samples but come up with poor performance on normal prompts, as shown in figure 1 . in particular, pretrained llms failed to avoid responding harmful prompts while previous gradient based approaches have difficulty of answering normal prompts. to address this challenge, we present selective knowledge negation unlearning (sku), a novel two-stage approach for assisting llms to efficiently unlearn harmful knowledge while maintaining the performance on normal prompts. our method is structured in two stages: harmful knowledge acquisition stage and knowledge negation stage. in particular, the knowledge negation stage is motivated by the negation operation of task vectors @xcite , where negating task vectors can effectively mitigate undesirable behaviors. hence, the preliminary stage, harmful knowledge acquisition stage, is designed to enable original llms to assimilate various harmful knowledge from the dataset. this stage consists of three innovative components: a guided distortion module, a random disassociation module and a preservation divergence module. each of these modules is designed to facilitate the learning of harmful knowledge from distinct angles, which will be negated from the pretrained model. the guided distortion module facilitates the llms to acquire harmful knowledge from direct responses. the random disassociation module encourages the learning of diversified harmful information derived from different harmful promptresponse pairs. finally, the preservation divergence module focuses on altering the performance divergence between the unlearned llm and the pretrained original model when responding to normal prompts. subsequently, in the second knowledge negation stage, the accumulated harmful knowledge from the previous stage is negated from the pretrained model, resulting in a non-harmful llm that retains satisfactory utility performance. our main contributions are as follows: ## preliminary let @xmath0, y)}, in which x is the text data and y is the corresponding label, to be the complete data that a llm θ o was trained on. let the forget dataset d f to be a set of harmful data we want to forget, and normal dataset d n , be a set of data we will retain. our ultimate goal is to let the θ o erase all information from d f while retaining utility performance on d n . in particular, d f consists of a group of harmful prompt-response pairs (x f , y f ), where x f are harmful driven prompts and y f are dangerous and harmful responses that we want θ o to avoid generating. however, since a llm (i.e. θ o ) is trained on a wide range of online dataset, it would be unrealistic to find a forget dataset that includes all harmful information. hence, the harmful prompts x f in d f do not necessary have to belong to the training dataset of θ o . similarly, normal dataset d n contains a group of benign prompt-response pairs (x n , y n ), where x n , y n can be any prompts and responses as long as x n , y n / ∈ d f and do not present any harmful texts. ideally, we would retrain the θ o by excluding the data from d f , and regard it as the golden baseline. however, this approach is computationally prohibitive, as highlighted in @xcite . in addition, to ensure the generalizability of the unlearning approach, given any unseen harmful prompt xf , we want the unlearned model θ u to generate non-harmful responses as well. ## methods the primary goal of our unlearning algorithm is to enable large language models (llms) to effectively remove harmful knowledge while maintaining a satisfactory utility performance on nonharmful prompts. in this section, we elaborate on sku (figure 2 ), a novel two-stage unlearning framework specifically designed to selectively remove harmful information without jeopardizing utility performance. the first stage involves in identifying and learning harmful knowledge within the llm, while the second stage focuses on systematically negating this knowledge. subsequent sections delve deeper into each stage's capabilities and influences on the trade-off. ## experiments in this section, we present extensive experiments to validate the effectiveness of the sku. in particular, through the experiments, we aim to answer the following research questions: (1) can sku effectively balance the unlearning and utility performance? ( 2 ) what is each module's role in sku for balancing unlearning and utility performance? ## ablation study in this section, we conducted ablation experiments by iteratively removing each module from sku, which can demonstrate the effectiveness of each section on leveraging the balance between model utility and unlearning efficacy. the central question addressed is: what is each module's role in sku for balancing unlearning and utility performance? the associated results are shown in ## conclusion in this work, we explore the trade-off between maintaining model utility and unlearning harmful knowledge in large language models (llms). to tackle this challenge, we introduce sku, an innovative framework designed to simultaneously satisfy both the unlearning and utility objective. specifically, this approach encompasses a twostage process: the harmful knowledge acquisition stage, and knowledge negation stage, where the first stage enhance the harmful knowledge for easy figure 3 : the performance of sku with a number of baselines on llama2-7b. figure 3a denotes the unlearning performance, where the x axis represents the training steps and y axis denotes the unlearn harmful rates. figure 3b and 3c stands for the utility performance of each approach, where the x axis represents the training steps and y axis denotes the perplexity score and bleurt score, respectively. the orange line represents the performance of sku.