UIPE : Enhancing LLM Unlearning by Removing Knowledge Related to Forgetting Targets (2025)

Abstract: Large Language Models (LLMs) inevitably acquire harmful information during training on massive datasets. LLM unlearning aims to eliminate the influence of such harmful information while maintaining the model’s overall performance. Existing unlearning methods, represented by gradient ascent-based approaches, primarily focus on forgetting target data while overlooking the crucial impact of logically related knowledge on the effectiveness of unlearning. In this paper, through both theoretical and experimental analyses, we first demonstrate that a key reason for the suboptimal unlearning performance is that models can reconstruct the target content through reasoning with logically related knowledge. To address this issue, we propose Unlearning Improvement via Parameter Extrapolation (UIPE), a method that removes knowledge highly correlated with the forgetting targets. Experimental results show that UIPE significantly enhances the performance of GA-based method and its variants on the TOFU and WMDP benchmarks.

## introduction large language models (llms) trained on massive datasets show exceptional capabilities @xcite . however, such extensive datasets inevitably contain harmful information, which diminishes model performance and may cause societal challenges @xcite . to mitigate such issues, llm unlearning has emerged as a critical research direction. llm unlearning aims to mitigate the influence of undesired data @xcite @xcite @xcite . gradient ascent-based (ga) llm unlearning has emerged as one of the predominant methodologies in this field @xcite . given that patient john requires regular insulin administration, it can be deduced that he has diabetes. ## preliminary experiments to validate this hypothesis that llms can leverage related knowledge to reconstruct forgotten knowledge, we first construct a target forget set along with a corresponding related knowledge set, and then conduct a series of comparative experiments to systematically evaluate this phenomenon. ## experiments 6.1 experimental setup datasets and models. to evaluate the effectiveness of uipe, we conduct experiments on two llm unlearning benchmarks: ① fictional forgetting on the tofu dataset, which targets removal of fabricated knowledge; ② real-world forgetting on the wmdp @xcite dataset, which contains factual knowledge and does not require additional fine-tuning. for the tofu benchmark, we use the llama2-7b-chat model, while for wmdp, we adopt the zephyr-7b-beta @xcite model to maintain consistency with the original benchmark. see appendix d.1 for further details. evaluation setup. for tofu, we adopt the official metrics provided by the benchmark to evaluate both forget quality and model utility. for wmdp, following previous work, forget quality is assessed on the benchmark-provided wmdp-bio and wmdp-cyber subsets, while model utility is evaluated via zero-shot accuracy on the mmlu dataset @xcite . baselines. we evaluate the effectiveness of uipe by applying it to several baselines. in addition to the basic ga method, we include grad. diff. @xcite , kl min. @xcite , and npo @xcite . see appendix d.2 and d.3 for more details. ## conclusion in this paper, we investigate the impact of knowledge related to forgetting targets on the effectiveness of target knowledge elimination. building on this insight, we propose uipe (unlearning improvement via parameter extrapolation), a technique for enhancing the unlearning of target harmful knowledge without additional training. extensive experiments across multiple unlearning methods demonstrate that uipe consistently enhances their ability to remove target knowledge, improving forget quality while maintaining model utility.