Unlearn What You Want to Forget: Efficient Unlearning for LLM s (2023)

Abstract: Large language models (LLMs) have achieved significant progress from pre-training on and memorizing a wide range of textual data, however, this process might suffer from privacy issues and violations of data protection regulations. As a result, the ability to easily remove data related to individual users from such models while not deteriorating their predictive quality after the removal becomes increasingly important. To address these issues, in this work, we propose an efficient unlearning framework that could efficiently update LLMs without having to retrain the whole model after data removals, by introducing lightweight unlearning layers learned with a selective teacher-student objective into the transformers. In addition, we introduce a fusion mechanism to effectively combine different unlearning layers that learns to forget different sets of data to handle a sequence of forgetting operations. Experiments on classification and generation tasks demonstrate the effectiveness of our proposed methods compared to the state-of-the-art baselines.

## introduction utilizing large language models (llms) has become the dominant paradigm for various nlp applications @xcite @xcite @xcite @xcite @xcite as llms memorize a vast amount of knowledge during pre-training or fine-tuning on a wide range of textual data @xcite @xcite @xcite @xcite . however, these data could contain sensitive information such as names, phone numbers, email addresses, and private clinical notes (jang et al., 2022; kurmanji et al., 2023; 1 the codes are avaiable here: https://github.com/ salt-nlp/efficient_unlearning/ @xcite .extensive studies showed that llms could generate private information such as the editor-in-chief of mit technology review including his family members, work address, and phone number @xcite . recently, the eu's general data protection regulation (gdpr) and us's california consumer privacy act (ccpa) have also required the right to be forgotten, introducing new regulations that require applications to support the deletion of user-generated content when requested by users @xcite . in light of this, it is essential to provide llms with an efficient and effective way to unlearn the information requested by users. recent attention has been paid to the handling of such unlearning requests for llms through retraining and data pre-processing like sisa @xcite where training data is stored in different isolated slices and each checkpoint is saved after training on each slice. when a deletion request is received, the respective data point will be removed from the slice, and the model checkpoint up to the data point will be used to further retrain the model. the effect of unlearning is often reflected by the model errors on the deleted data (models cannot predict the deleted data) @xcite . other works have also explored the design of algorithms that ensure differential privacy (dp) @xcite @xcite . however, machine unlearning approaches like sisa @xcite usually require a significantly large amount of storage space @xcite , and dp methods could result in a slow convergence and significant deterioration in model performance @xcite . in addition, both of them require retraining the whole model, which is extremely expensive and time-consuming considering the model scales of the current llms. these limitations also make them unable to dynamically deal with a sequence of unlearning requests which is often the need in real-world scenarios @xcite . to fill in these gaps, in this work, we propose an efficient unlearning method for llms (eul) to efficiently unlearn what needs to be forgotten without completely retraining the whole model while retaining the performances of the models. specifically, we propose a lightweight approach to learning the unlearning layer that is plugged into transformers through a selective teacher-student formulation @xcite within several updates, without tuning the large language models. additionally, we introduce a fusion mechanism to effectively combine the weights of different unlearning layers that learn to forget different sets of data to a single unified unlearning layer by minimizing a regression objective. this allows eul to efficiently address a sequence of deletion operations. to demonstrate the effectiveness of our proposed eul, we perform experiments on imdb @xcite and samsum @xcite in different settings compared to the state-of-the-art unlearning or model editing baselines. to summarize, our main contributions are threefold: ## efficient unlearning for llms this section presents our designed efficient unlearning method for llms (eul) which could efficiently and dynamically handle a sequence of deletion requests. the overall process is shown in figure 1 . formally, for a large language model f (.) that is trained on a dataset @xmath0, y)} where x is textual data and y is the corresponding label, and a deletion request to forget d @xmath1, y f }, our goal is to learn an updated model f ′ (.) that satisfies the following @xcite : ## conclusion in this work, we propose eul, an efficient unlearning method for llms that could efficiently and effectively unlearn the user-requested data via learn-ing unlearning layers through the selective teacherstudent objective. we further introduce a fusion mechanism that could merge different unlearning layers into one unified layer to dynamically unlearn a sequence of data. experiments on different settings (different datasets, different model sizes, different forget set sizes) demonstrated the effectiveness of our proposed eul method compared to state-of-the-art baselines.