H oward U niversity- AI 4 PC at S em E val-2025 Task 4: Unlearning Sensitive Content From Large Language Models Using Finetuning and Distillation for Selective Knowledge Removal (2025)

Abstract: This paper presents our approach and submission to the SemEval 2025 task on “Unlearning Sensitive Content from Large Language Models.” The task focuses on making LLMs forget specific knowledge, such as copyrighted material and personally identifiable information (PII), without needing expensive retraining from scratch on the OLMo model. We propose a method to unlearn using fine-tuning and knowledge distillation. Our approach involves fine-tuning separate models on “retain” and “forget” datasets to preserve or suppress knowledge selectively. We then distill the model by suppressing logarithmic data from the fine-tuned model without learning using a combined loss of L2, KL divergence and cosine similarity while retaining knowledge from the fine-tuned model with retention using KL divergence loss.

## introduction large language models (llms) have revolutionized natural language processing (nlp) by training on a vast amount of publicly available information to perform various tasks. their ability to generate human-like responses and understand complex linguistic patterns has made them indispensable across industries. during training, they may inadvertently retain copyrighted content or personally identifiable information (pii), leading to ethical, legal, and privacy concerns. the presence of such sensitive data within an llm raises questions about data security, user privacy, and intellectual property rights, making it crucial to develop mechanisms to selectively remove unwanted information from these models. to foster research in the field of llm unlearning, @xcite introduced semeval-2025 task 4: unlearning sensitive contents from large-language models. this task was divided into three sub-tasks: the participants were provided with two sets of data: "forget" and "retain". as the names suggest, the goal was to ensure the model forgets the information from the "forget" set and preserves the information from the "retain" set. aditionally, the organizers provided 1b and 7b parameter models pretrained on the datasets provided to facilitate the experimentation. we refer to these models as candidate models throughout this work. for the evaluation phase, participants were required to submit working python scripts implementing the solution to the problem. the scripts were executed on privately held subsets of "forget" and "retain" sets for each sub-tasks to assess the effectiveness of the method. one possible solution to solve the problem is to retrain the model from scratch using a filtered dataset, but this process is computationally expensive and requires substantial gpu hours, making it impractical for large-scale models. moreover, complete retraining does not guarantee the complete removal of unwanted knowledge as residual traces of the information may persist due to the complex nature of llms. to address these challenges, we experimented with a method to unlearn specific data from llms without requiring complete retraining. our method focuses on selectively modifying the model in a way to suppress unwanted knowledge while preserving required information. we try to achieve this by fine-tuning separate models on forget and retain sets. this allows us to distill knowledge in a way where we try to suppress the forget-set information and retain the retain-set information. ## related work in this section, we explore prior research done in the field of unlearning in non-llm and llm settings. while prior methods like tv-stable sgd, rcad, and salun have shown their effectiveness, they primarily target smaller-scale models or specific domains like classification and image generation. these approaches may not directly generalize to llms. our approach, however, is specifically geared towards llms by utilizing finetuning and knowledge distillation. @xcite introduce gradient ascent on the target token to forget information. they also highlight that sequential unlearning is better than batch unlearning. in contrast, our method involves reinforcing the knowledge to eventually retain or forget the information. @xcite introduces a method that uses logit differences to suppress certain information. they use assistant models to forget the "retain" set and remember the "forget" set and eventually use the logit difference to remove forget set information from the base model. in our method, however, we use assistant models to retain both the retain and forget sets and suppress the forget set onto the candidate model while reinforcing the information from the retain set. @xcite introduces demem, an approach that utilizes a reinforcement learning feedback loop to unlearn. in contrast, our approach uses finetuning and knowledge distillation to unlearn. ## description of our technique this section outlines the methodology used in our experiments. before detailing the technique, we establish the following notations: the experiment consists of the following steps: ## result our approach resulted in insufficient gpu vram on the official test environment for the 7b parameter olmo model. however, there was no such issue when it came to running the algorithm on the 1b parameter olmo model. the final 1b parameter model underperformed and ranked 24 th on the leaderboard. our final aggregate score was 0.079 with an mmlu score of 0.236. we conducted additional experiments using the gpt-2 model with 137 million parameters. the performance of the model was evaluated using rogue score. the scores after finetuning the models on retain and forget sets are as follows: the results above indicate that the finetuned models retained some information from their respective training datasets. after the distillation process as discussed in section 3.2, we had the following results: forget dataset retain dataset rogue-1 0.71 0.77 rogue-l 0.71 0.77 the post-distillation results suggest that for the retain dataset, the model's output tried to move as close as possible to the actual output. while the result from forget dataset suggest that the predicted outputs shifted a bit from the actual outputs. while the negative weighted loss functions helped in somewhat suppressing the information, we still can see that the suppression was incomplete because of relatively high rogue scores. this finding suggests a potential need for an alternative or complementary loss function to enhance the effectiveness of unlearning. ## conclusion in this study, we explored a fine-tuning and distillation-based approach to unlearning in large language models. our method involved training separate models to distinguish between forget and retain sets, followed by a distillation phase to suppress unwanted knowledge while preserving critical information. despite the effort, the results did not achieve the effective unlearning. the method struggled to fully remove the targeted knowledge and resulted in an unintended degradation to the model, highlighting the need for a modified approach to solve the problem.