Fine-grained Pluggable Gradient Ascent for Knowledge Unlearning in Language Models (2024)

Abstract: Pre-trained language models acquire knowledge from vast amounts of text data, which can inadvertently contain sensitive information. To mitigate the presence of undesirable knowledge, the task of knowledge unlearning becomes crucial for language models. Previous research relies on gradient ascent methods to achieve knowledge unlearning, which is simple and effective. However, this approach calculates all the gradients of tokens in the sequence, potentially compromising the general ability of language models. To overcome this limitation, we propose an adaptive objective that calculates gradients with fine-grained control specifically targeting sensitive tokens. Our adaptive objective is pluggable, ensuring simplicity and enabling extension to the regularization-based framework that utilizes non-target data or other models to preserve general ability. Through extensive experiments targeting the removal of typical sensitive data, we demonstrate that our proposed method enhances the general ability of language models while achieving knowledge unlearning. Additionally, it demonstrates the capability to adapt to behavior alignment, eliminating all the undesirable knowledge within a specific domain.

## introduction machine unlearning, a burgeoning research topic, has gained significant attention in recent years @xcite . it aims to erase the memory of target data from machine learning models, offering potential applications such as removing poisoned data to enhance security @xcite , retrieving personal data to comply with privacy regulations (e.g., the right-to-beforgotten) @xcite , and mitigating biases to promote fairness @xcite . existing studies on unlearning primarily concentrate on computer tokens weights bob's address is xxx you are just like a fool 0.08 0.08 0.09 0.09 0.08 0.57 bob's address is xxx you are just like a fool bob's address is xxx you are just like a fool bob's address is xxx you are just like a fool vision but also extend their exploration to other fields, e.g., federated learning @xcite , recommender systems @xcite , and graph learning @xcite . there is a pressing need for unlearning methods specifically tailored to language models, referred to as knowledge unlearning @xcite . this need arises because language models acquire knowledge from open-source text data, which inherently contains sensitive information, including toxic and private content. however, applying existing unlearning methods directly to language models poses significant challenges. firstly, the retraining overhead of language models is exceptionally high, making it computationally prohibitive for regular users, even when only retraining a subcomponent. secondly, language models have an enormous parameter size, rendering certain memory or influence estimation approaches inaccurate and even intractable. an alternative approach to removing undesirable knowledge from language models is reinforcement learning from human feedback (rlhf) @xcite , but it is not well-suited for this purpose. rlhf involves fine-tuning the model to align with human preferences, which requires a significant amount of preference-aligned text data, e.g., gpt 4 @xcite . however, obtaining a large volume of high-quality resources is challenging and may require considerable effort. given that the primary priority of knowledge unlearning is to prevent the generation of undesirable outputs rather than focusing on generating desirable outputs, unlearning becomes particularly appealing than rlhf. previous research on knowledge unlearning uses gradient ascent to achieve unlearning @xcite . this approach is simple yet effective in overcoming the challenges associated with language models, i.e., the prohibitive retraining overhead and the enormous size of model parameters. however, as shown in figure 1 , it applies gradient ascent to the whole sequence, i.e., all the tokens in the target sequence that are intended to be removed. as a result, while unlearning takes place, besides the target undesirable knowledge, other lexical and semantic knowledge can also be affected. this, in turn, can have a negative impact on the general ability of language models. note that the general language ability holds great significance, and excessively impacting it can be viewed as an instance of over-unlearning. to address this concern, we propose an adaptive objective, which replaces the original objective used in gradient ascent. our proposed objective introduces adaptive weights to each token in the target sequence, providing fine-grained control over the unlearning target. this allows us to minimize the negative impact on non-target knowledge to the greatest extent possible. it is important to note that this objective can be considered as a pluggable component to the gradient ascent approach, ensuring that the knowledge unlearning process remains simple without the need for expensive retraining or influence estimation. furthermore, this approach facilitates the extension of our proposed fine-grained pluggable gradient ascent (fpga) in the regularization-based framework. by incorporating preference-aligned data or models, this framework can further augment the general ability of language models, complementing the unlearning process. we summarize the main contributions of this paper as follows: the results show that it achieves comparable performance with behavior alignment methods, whereas the pure gradient ascent method completely fails. ## preliminary in this section, we identify the targets and principles of knowledge unlearning, and distinguish it from machine unlearning, providing insights for evaluation and future research. ## conclusion in this paper, we investigate the task of knowledge unlearning for language models. existing fine-tuning approach provides a viable solution for knowledge unlearning, utilizing gradient ascent to achieve unlearning. however, this reverse learning process potentially harms the general ability of the language model. to mitigate this issue, we propose a novel approach called fine-grained pluggable gradient ascent (fpga), which introduces adaptive weights into the original objective. fpga offers a simple yet effective solution, acting as a pluggable complement to gradient ascent. furthermore, it can be extended to regularization-based approaches that incorporate additional data or models to preserve the general ability of the language model. through experiments conducted on various datasets, we demonstrate that our proposed method significantly enhances the general ability of language models while achieving effective knowledge unlearning. more importantly, it can facilitate behavior alignment by increasing the volume of unlearning targets. the fine-grain control provided by adaptive objective contributes to maintaining the general ability of language models. in contrast, the pure ga method can achieve unlearning, but fail to align behavior. this is because the increased unlearning volume inadvertently harms the model's general ability to generate meaningful text.