Machine Unlearning of Pre-trained Large Language Models (2024)

Abstract: This study investigates the concept of the ‘right to be forgotten’ within the context of large language models (LLMs). We explore machine unlearning as a pivotal solution, with a focus on pre-trained models–a notably under-researched area. Our research delineates a comprehensive framework for machine unlearning in pre-trained LLMs, encompassing a critical analysis of seven diverse unlearning methods. Through rigorous evaluation using curated datasets from arXiv, books, and GitHub, we establish a robust benchmark for unlearning performance, demonstrating that these methods are over 10^5 times more computationally efficient than retraining. Our results show that integrating gradient ascent with gradient descent on in-distribution data improves hyperparameter robustness. We also provide detailed guidelines for efficient hyperparameter tuning in the unlearning process. Our findings advance the discourse on ethical AI practices, offering substantive insights into the mechanics of machine unlearning for pre-trained LLMs and underscoring the potential for responsible AI development.

## introduction large language models (llms) have seen remarkable advancements, attributable to training on extensive and diverse datasets @xcite @xcite @xcite . yet, the reliance on massive data pools has raised significant ethical concerns, particularly when such data include sensitive, private, or copyrighted material @xcite @xcite . a prominent example of these issues is the recent lawsuit filed by the new york times 2 against openai. the lawsuit, responding to the alleged use of millions of articles from the times in training llms like chatgpt, highlights the critical issue of copyright infringement in llms' development. in response to these ethical challenges, the concept of machine unlearning has emerged as a potential remedy. it entails systematically removing specific data from a model's training, ensuring its operation as though the data had never been included @xcite . this approach mitigates the ethical issues stemming from the pretrained data in llms, aligning the technology with evolving legal and ethical standards. despite its potential, current research on machine unlearning in the realm of llms has been predominantly confined to the fine-tuned model @xcite @xcite . while fine-tuning datasets typically contain token numbers on the magnitude of 10 6 or less, pre-training corpora are on the magnitude of 10 12 . consequently, retraining on the retained set of a fine-tuning dataset based on a pre-trained model is often feasible, whereas retraining the pre-trained model from scratch is impractical. therefore, this focus on unlearning fine-tuned models has limitations and is less critical. the real challenge, and our focus in this paper, is the unlearning of pre-trained llms. this challenge is compounded by several factors: 1) the need to adapt existing unlearning methods from other fields to pre-trained llms, 2) the general lack of public availability of pre-trained data used to develop llms, and 3) the absence of directly comparable baselines due to the exorbitant costs of retraining pre-trained llms. our paper addresses them through several key contributions. we first define the problem of machine unlearning for pre-trained llms and propose a unified formulation consolidating prior arts under a single unlearning objective. we then investigate seven different unlearning methods in the context ## problem formulations we call the model designated for unlearning the vanilla model. we denote a forget set of sequences to be unlearned as u ⊂ d. to remove the effect of u, we consider an unlearning algorithm a that applies to a(d) and outputs an unlearned model m ′ . motivated by differential privacy @xcite , @xcite formulated the probabilistic notion of unlearning using (ϵ, δ)-closeness of distributions @xcite . informally, it requires that the output distributions of a and a run over (d \ u) to be similar. we call it exact unlearning if the distributions are identical (i.e., ϵ = δ = 0); approximate unlearning, otherwise. a naïve solution for exact unlearning is just retraining on d \ u for each u from scratch. however, it is prohibitively expensive for llms, in-curring gigantic computation costs and carbon footprints @xcite . deriving theoretical guarantees for llms is also non-trivial, as the underlying transformer architecture is not convex or lipschitz @xcite . pragmatically, an active line of research @xcite @xcite only requires the empirical performance (e.g., classification accuracy) of retrained and unlearned models to be similar. in our context, we can resort to perplexity and ensure ## related work we provide an overview of current research on machine unlearning, memorization, and forgetting. a more detailed version is deferred to appendix b. machine unlearning. the concept of machine unlearning is first introduced in @xcite . @xcite further formalizes exact unlearning by introducing a general framework: sharded, isolated, sliced, aggregated (sisa). exact unlearning requires the unlearned model the same as the retrained model. approximate unlearning, which relaxes the requirement, is also explored by bounding the distance @xcite or indistinguishability @xcite between the two model's distributions. machine unlearning has been extensively researched within the broader field of machine learning @xcite , yet its exploration in gen-erative language models remains limited. @xcite propose sisa-fc and sisa-a, two computationally efficient extensions of sisa for classification lms, e.g., bert. to unlearn knowledge in generative models, @xcite simply perform gradient ascent on target sequences. @xcite consider a special case of unlearning the harry potter books from llama2-7b. @xcite applies machine unlearning for harmful responses removing and hallucinations eliminating. however, these studies have been limited to fine-tuned models or a single corpus source. our work explores unlearning pre-trained llms on more diverse datasets. memorization and forgetting. @xcite first quantifies unintended memorization by a metric called exposure, revealing severe privacy issues, e.g., membership inference attacks @xcite or verbatim data extraction @xcite . contrary to memorization, catastrophic forgetting, where a model loses previously learned knowledge when training on new data, has been studied @xcite . it is a passive phenomenon different from unlearning, which actively forces models to "forget" specific samples. ## conclusion in this paper, we investigate the challenge of removing copyrighted pre-training data from llms. we present a unified formulation for unlearning llms, from which seven unlearning methodologies are derived. we introduce approximate retraining as an evaluation technique to bypass the impracticality of retraining llms from scratch. our experimental analysis across three pre-training data domains validates the efficacy of the unlearning approaches. furthermore, we find that combining gradient ascent with gradient descent on in-distribution data enhances hyperparameter robustness. we also offer guidelines to streamline the tuning of hyperparameters essential to the unlearning process.