ULMR : Unlearning Large Language Models via Negative Response and Model Parameter Average (2024)

Abstract: In recent years, large language models (LLMs) have attracted significant interest from the research community due to their broad applicability in many language-oriented tasks, and are now widely used in numerous areas of production and daily life. One source of the powerful capabilities of LLMs is the massive scale of their pre-training dataset. However, these pre-training datasets contain many outdated, harmful, and personally sensitive information, which inevitably becomes memorized by LLM during the pre-training process. Eliminating this undesirable data is crucial for ensuring the model’s safety and enhancing the user experience. However, the cost of extensively cleaning the pre-training dataset and retraining the model from scratch is very high. In this work, we propose ULMR , a unlearning framework for LLMs , which first uses carefully designed prompts to rewrite the instructions in the specified dataset, and generate corresponding negative responses. Subsequently, to ensure that the model does not excessively deviate post-training, we perform model parameter averaging to preserve the performance of the original LLM. We conducted experiments on two public datasets, TOFU and RWKU, demonstrating that our method can effectively forget specified information while retaining the capabilities of the original LLM.

## introduction large language models (llms) have achieved commendable success in various tasks, demonstrating their capability to disseminate knowledge across different fields and tasks. nowadays, llms are being utilized by the general public as personal assistants, providing advice and solutions for a variety of daily activities @xcite @xcite . the remarkable abilities of llms largely stem from * equal contributions. the massive dataset used during their pre-training process. llms can parameterize this knowledge, possessing the ability to recall and apply it when generating responses. however, the pre-training dataset widely contains personal privacy information (such as personal identification codes) and harmful content, including biases, discrimination, or content that violates human ethics. additionally, using copyrighted content without consent for pre-training has garnered attention. many countries have privacy protection laws requiring that personal data not be disclosed arbitrarily or allowing individuals or organizations to request the deletion of their data from service providers according to their wishes @xcite . a straightforward approach is to inspect the pretraining dataset, remove problematic data, and then retrain the model from scratch using the remaining dataset @xcite . this method has been widely applied in smaller-scale neural network models, but it is prohibitively expensive and impractical for llms with billions of parameters. therefore, the method of fast approximate unlearning is crucial. research on unlearning is still in its early stages, focusing on the fields of machine learning, and unlearning for llms remains a challenging task @xcite . in this paper, we propose a framework named ulmr for rapid and efficient forgetting on specific instruction sets for llms. first, we enhance the model's ability to generalize and improve its forgetting performance by rewriting the initial instruction set using carefully designed prompts. second, based on the rewritten instructions, we generate corresponding negative responses to train the llm to produce confused responses about the information to be forgotten. finally, to ensure that the weight shift of the model post-training is controlled, we perform a model parameter averaging process to maintain the model's general capabilities without significant degradation. supervised finetuning (sft) by providing specific tasks or directives to the model, enables it to better understand and execute different types of tasks and is a vital method for updating the model's knowledge base @xcite . sft is also applied in many llm unlearning algorithms. the instruction rewriting process can alleviate the overfitting of patterns in the training data by llms during training, thereby enhancing their generalization capabilities. model parameter averaging can mitigate the adverse effects on the model's capabilities during the unlearning process, striking a better balance between forgetting and general capabilities @xcite . our empirical results from experiments demonstrate that the framework we propose can effectively forget knowledge on specified data while maximally preserving its general capabilities. ## methods in this work, our goal is to develop a simple and efficient llm unlearning framework that can forget content in the target dataset while maximizing the retention of general capabilities. initially, we enhance and restructure the forgotten dataset d f to maximize the assurance that the model forgets the corresponding knowledge. subsequently, we perform model parameter averaging to restore the general capabilities of the sft model. the complete framework is illustrated in figure 1 . to maximally induce the llm to forget the content on the specified dataset, we first need to enhance and restructure the forgotten dataset d f . ## conclusion in this work, we develop a simple and efficient llm unlearning algorithm named ulmr. initially, we enhanced and restructured the forget dataset using carefully designed prompts to maximize the assurance that the model forgets the corresponding knowledge. subsequently, we performed model parameter averaging to restore the general capability of the sft model. tests on the tofu and rwku unlearning benchmark demonstrated that our method can retain the general capabilities of the llm to the greatest extent while forgetting the content in the target dataset as much as possible.