Balancing Forget Quality and Model Utility: A Reverse KL -Divergence Knowledge Distillation Approach for Better Unlearning in LLM s (2025)

Abstract: As concern for privacy rights has grown and the size of language model training datasets has expanded, research into machine unlearning for large language models (LLMs) has become crucial. Before the era of LLMs, research on machine unlearning mainly focused on classification tasks in small parameter models. However, as parameter sizes have grown and unlearning targets have become more complex, unlearning has become more challenging, especially in scenarios involving generation instead of classification, as the output space of such models is significantly larger and more diverse. Existing methods based on gradient ascent and its variants often struggle with balancing forget quality and model utility, leading to either over unlearning or partial unlearning. To address this challenge, we propose Reverse KL-Divergence based Knowledge Distillation for Unlearning (RKLU), a novel unlearning method for LLMs. RKLU focuses on precisely unlearning the components of the token distribution related to the unlearning target, allowing us to achieve significant forget quality while maintaining model utility in our experiments.

## introduction llms are trained with extensive data, leading to the development of emergent abilities but also retaining sensitive and personal information. for example, the model might learn personal details, such as age, educational background, family background, and other various information @xcite @xcite . the gdpr mandates the right to be forgotten (rtbf), allowing individuals to request the removal of any information related to them from machine learning models @xcite . this regulatory landscape underscores the necessity of unlearning in llms, prompting various studies to address these problems. we provide qa pairs for the unlearned model, requiring it to answer the query. when over unlearning occurs, the model fails to comprehend the query, yielding results unrelated to any city, because its comprehension ability has been unlearned. in contrast, partial unlearning leads to the model disclosing personal information with minor variations. ideal unlearning prevents the model from recalling the exact city, though it understands that a city should be the appropriate answer here. the primary goal of machine unlearning is to remove the influence of specific data samples (unlearning targets) from trained models @xcite . this broad definition has led to unlearning efforts in various aspects, including defense against extraction attacks @xcite , unlearning personal or copyrighted information @xcite , detoxification @xcite , and debiasing @xcite . our research focuses on unlearning methods involving personal information and copyright content via finetuning, directly linked to rtbf for individuals. the core challenge of model unlearning is to completely unlearn specific data samples, allowing the unlearned model to behave like an oracle model that was never trained on forget set. at the same time, it must achieve good forget qual-ity while balance the maintenance of model utility. most current unlearning methods focus on directly reducing the likelihood of generating unlearning targets using gradient ascent and similar techniques @xcite . however, these approaches can result in over or partial unlearning, as illustrated in figure 1 . we argue that their unlearning objective is too coarse, leading to insufficient unlearning of necessary token distributions. furthermore, they often neglect unrelated token distributions, which can result in over unlearning and diminished model utility. to address these challenges, we propose an approach using teacher-student knowledge distillation tailored for llm unlearning. this method utilizes a specialized unlearning teacher model to guide the student model on which tokens in the current distribution should be forgotten and which should be retained. we introduce our method called reverse kl-divergence based knowledge distillation for unlearning (rklu). rklu draws inspiration from the prior method @xcite which continues to finetune the original model on a forget set to augment it. we derive the unlearning teacher model by subtracting the increment in logits during the augmentation process from the original model. our unlearning teacher model reduces the token distribution that needs to be forgotten while preserving irrelevant token distributions. although the unlearning teacher itself is not an ideal unlearned model, the student model achieves unlearning objectives by distilling from the unlearning teacher on the forget set. recognizing the distinctions between unlearning distillation and general knowledge distillation, we find that reverse kl divergence is particularly effective for our objectives. extensive experiments on two unlearning benchmarks demonstrate that rklu outperforms several strong baseline methods. our contributions are as follows. ## experiments in this section, we apply rklu to two unlearning tasks: personal information unlearning on the tofu dataset and copyright content unlearning on harry potter book. for these tasks, we adopt settings and evaluation metrics following previous works @xcite @xcite . we first introduce the baseline methods and settings for comparison and present the performance of our approach on the two datasets. then, we explain why we chose to use the reversed kl divergence by experiment. ## case study in this section, we will show over or partial unlearning issues by exploring unlearned model using two methods: direct answer and fill-in-blank. this setting is based on previous findings that the more detailed the given prefix string, the more likely the model is to recall its memory @xcite . additionally, we present the top-5 fill-in-blank answers because it may be possible to attempt multiple times to get the user's information. as shown in we posit that incomplete unlearning intensifies internal knowledge conflicts, heightening privacy breach risks. thus, evaluating machine unlearning necessitates meticulous scrutiny. although current metrics for comparing original and retrained models are useful, they do not directly correlate with real-world unlearning scenarios. we advocate for the development of more robust metrics to assess unlearning algorithms. ## conclusion in this paper, we introduce the reverse kl-divergence based knowledge distillation for unlearning (rklu) method for better unlearning in llms, using reverse kl-divergence based knowledge distillation for unlearning while maintaining performance. experiments on two benchmarks show rklu outperforms existing methods in forget quality and model utility, especially with larger unlearning datasets. it also retains general capabilities. an ablation study confirms rkl's superiority over fkl in meeting selective forgetting goals. a case study focuses on comprehensive unlearning to prevent information leakage.