Dissecting Fine-Tuning Unlearning in Large Language Models (2024)

Abstract: Fine-tuning-based unlearning methods prevail for erasing targeted harmful, sensitive, or copyrighted information within large language models while preserving overall capabilities. However, the true effectiveness of the methods is unclear. In this paper, we delve into the limitations of fine-tuning-based unlearning through activation patching and parameter restoration experiments. Our findings reveal that these methods alter the model’s knowledge retrieval process, rather than genuinely erasing the problematic knowledge embedded in the model parameters. Furthermore, behavioral tests demonstrate that the unlearning mechanisms inevitably impact the global behavior of the models, affecting unrelated knowledge or capabilities. Our work advocates the development of more resilient unlearning techniques for truly erasing knowledge.

## introduction large language models (llms), due to their extensive pre-training corpora, often inadvertently learn harmful, sensitive, or copyright-protected knowledge @xcite @xcite . consequently, recent research has focused on developing efficient unlearning methods as a post-training technique to selectively unlearn the specific knowledge @xcite . currently, the core mechanism of these unlearning methods involves finetuning @xcite @xcite , with corresponding adjustments and designs in the loss function to facilitate the unlearning process. although earlier investigations @xcite have proven that these methods are ineffective at completely erasing model-embedded knowledge, the factors contributing to the misleading success of these techniques remain unclear. therefore, in this paper, we try to unveil why existing finetuning-based unlearning methods perform well in behavioral tests by analyzing the mechanisms of internal knowledge recall and flow within models @xcite @xcite . specifically, we investigate which components or parameters carry these unlearning effects. we design activations patching and parameters restoration experiments in three settings, aiming to independently study the impact of unlearning methods on the coefficients and value vectors in the mlps, as well as on the attention components' states. our findings further confirm that the methods do not truly alter the knowledge embedded in the value vectors of mlps, and reveal that they will change how they extract and transfer this knowledge through modifications in the coefficients of mlps and attention components during unlearning. notably, the coefficients produced by the mlp in the final layers are primarily responsible for achieving the unlearning effects of finetuning-based methods. we further test the global behavior impact of these fine-tuning-based unlearning methods on llama2-7b-chat @xcite and olmo-7b @xcite by implementing them on the respective pretraining datasets of both models, aiming to more closely simulate the erasure of knowledge acquired during the pretraining process. we discover that while these methods appear to effectively unlearn target knowledge, they also inevitably affect the output and behavior related to unrelated knowledge. this unintended consequence stems from the fact that these approaches are based on altering the model's internal knowl-edge retrieval mechanisms, thereby impacting its global behavior and overall performance. ultimately, we conclude once again that current fine-tuning-based unlearning methods cannot completely erase sensitive knowledge embedded in models, particularly within the mlps, instead adjusting the mechanisms by which the model retrieves knowledge. these methods are vulnerable to recovery attacks in components' activations and unsuitable for true unlearning. we advocate for future unlearning evaluations to concentrate on precise measurement of both the actual storage of targeted knowledge within the model's entire parameter set and the specific dynamics of how this knowledge is retrieved and utilized. knowledge storation in large language models studying how knowledge is stored, transferred, and extracted in llms has always been an important direction in the research of llm's interpretability @xcite @xcite . it is known that in transformer-based language models, the mlp is a crucial component for storing the model's factual knowledge, and its sub-layers can be viewed as key-value memories @xcite . to be specific, the first layer * of mlp sublayers can be viewed as a matrix w k formed by key vectors {k 1 , k 2 , . . . , k n }, used to capture a set of patterns in the input sequence, and ultimately outputting the coefficient scores. the second layer can be viewed as a matrix w v formed by value vectors {v 1 , v 2 , . . . , v n }, with each value vector containing the corresponding factual knowledge (represented through token distributions). finally, the mlp's output can be defined as the sum of value vectors weighted by their memory coefficients: in addition to the mlp, primarily responsible for knowledge storage, the attention component is currently considered the main component responsible for knowledge transfer and extraction in language models @xcite . here, we will not go into detail about its specific structure but only study the impact it has on knowledge extraction. the final computation formula for the hidden states in the language model is defined as: ## patching investigation hypothesis and experimental design based on eq. ( 1 ) and eq. ( 2 ), we hypothesize that there are three main reasons why the current fine-tuningbased unlearning methods appear successful in behavioral tests and seem to suggest that true unlearning has been achieved: and l i,n that are nearly consistent, representing a higher degree of knowledge recovery. conversely, a lower krs suggests a lower degree of that. ## global negative effect of fine-tuning unlearning in the previous section, we demonstrated that these finetuning-based methods alter the model's final behavior by adjusting the mlp output coefficients in the final layers. therefore, we hypothesize that this behavioral change will has a global effect, potentially impacting the output of unrelated knowledge as well. in this section, we verify this hypothesis through the following experiments. we apply four fine-tuning-based unlearning methods to the concepts used in §3 on their pretraining text sources (from redpajama and dolma) with the goal of erasing the learned knowledge during pretraining through a reverse process. these methods are as follows: dpo @xcite , npo @xcite , npo+kl @xcite and gradient difference @xcite . the details of these baselines and data statistics are shown in §a and §b. we evaluate the unlearning effectiveness of these methods on the concepts' related qa pairs and the unlearning impact on unrelated qa pairs, reporting the average scores of bleu @xcite by comparing the model's response before and after unlearning. in figure 2 , we report their performance at the end of each training epoch respectively. we can observe that for finetuning-based methods, as the number of training epochs increases, aiming to achieve a lower target qa bleu score, the corresponding unrelated qa bleu score also decreases accordingly, exhibiting a positive correlation. this suggests that the impact of finetuningbased methods on the model's output behavior is global. while unlearning the target knowledge, they inadvertently alter the output behavior or manner for unrelated knowledge to a certain degree. ## discussion and conclusion we have deeply investigated the reasons why finetuning-based unlearning methods seemingly succeeded in behavior-based testing for large language model unlearning: through activation patching and parameter restoration experiments, we find that these methods alter the way knowledge is extracted by changing mlp activations or model's attention, ultimately affecting the output. this is evidenced by the fact that the model's output regarding the target knowledge is largely restored after patching the activations and the attention components' states. furthermore, we conduct experiments on 0 1 2 3 4 5 6 7 8 0 the pretraining datasets of two models, to test the models' capabilities after unlearning, verifying that in addition to unlearning the corresponding knowledge, fine-tuning-based methods that by altering the way the model accesses knowledge, will significantly impair the model's other unrelated capabilities, causing a certain degree of capability degradation.