NLPART at S em E val-2025 Task 4: Forgetting is harder than Learning (2025)

Abstract: Unlearning is a critical capability for ensuring privacy, security, and compliance in AI systems, enabling models to forget specific data while retaining overall performance. In this work, we participated in Task 4 of SemEval 2025, which focused on unlearning across three sub-tasks: (1) long-form synthetic creative documents, (2) short-form synthetic biographies containing personally identifiable information, and (3) real documents sampled from the target model’s training dataset. We conducted four experiments, employing Supervised Fine-Tuning (SFT) and Negative Preference Optimization (NPO). Despite achieving good performance on the retain set—data that the model was supposed to remember—our findings demonstrate that these techniques did not perform well on the forget set, where unlearning was required.

## introduction as machine learning (ml) continues to be integrated into critical domains, concerns over data privacy, security, and user autonomy have become more pressing than ever. from healthcare to finance, data-driven models play a crucial role, but their ability to retain and utilize information raises significant challenges. this issue has been further emphasized by legal regulations such as the general data protection regulation (gdpr), which grants individuals the "right to be forgotten" @xcite . machine unlearning has emerged as a key approach to tackling these concerns. it enables the selective removal of specific data points from a trained model without necessitating a full retraining process @xcite . unlike fine-tuning or knowledge editing, which focus on adjusting or adding new information, machine unlearning is designed to eliminate the influence of certain inputs, ensuring they no longer contribute to the model's predictions or behavior. the overall workflow of machine unlearning is illustrated in figure 1 . the importance of machine unlearning is magnified by the intricate nature of deep learning models. these models often exhibit data entanglement, where information from different training instances becomes deeply intertwined within the model's parameters, making selective removal a challenging task @xcite . additionally, unlearning must be carefully implemented to prevent unnecessary degradation of the model's performance on retained data, striking a balance between utility and privacy @xcite . another significant hurdle is efficiency-particularly in large-scale architectures like llms-since retraining a model from scratch is often impractical due to the immense computational cost. as large language models gain increasing prominence, it becomes essential to examine the relationship between machine unlearning and knowledge editing. knowledge editing is typically used to update or correct specific facts or behaviors in llms without requiring a full model retrain @xcite @xcite . while both techniques modify a model's internal representations, they serve distinct purposes: knowledge editing injects or refines information, whereas unlearning aims to remove specific influences entirely. this distinction highlights both the challenges and the potential areas of overlap between these two approaches. beyond its technical implications, machine unlearning also plays a crucial role in ensuring ethical ai practices and regulatory compliance. by enabling models to forget specific information when required, unlearning enhances both user privacy and model transparency, making it an essential tool in the evolving landscape of responsible ai. in this study, we explore machine unlearning in large language models by participating in semeval-2025 task 4. we focus on selectively removing memorized information across three document types: synthetic creative documents, synthetic biographies containing personally identifiable information (pii), and real-world documents, without compromising the model's general performance. for this, we conduct four experiments using supervised fine-tuning (sft) and direct preference optimization (dpo) techniques. our findings reveal the challenges of achieving effective unlearning, emphasizing that current approaches, while promising for retention, still fall short in forgetting sensitive data. through this work, we aim to contribute insights into the limitations and future directions for improving machine unlearning in large-scale ai systems. foot_0 ## background in recent years, machine unlearning has attracted significant attention as concerns about data privacy, regulatory compliance, and ethical ai practices have grown. the fundamental concept of machine unlearning is to allow trained models to remove specific data points without the need for complete retraining, thereby achieving a balance between efficiency and privacy demands. this section examines the leading approaches and methodologies presented in the literature. ## system overview this study explores unlearning in large language models (llms) through four experiments using two training methods: direct preference optimization (dpo) and supervised fine-tuning (sft). the approach for data collection and training is outlined below. ## results the results show that these methods had little effect on the model's performance since key performance measures barely changed after using them. this means that while dpo and sft may help in some parts of the unlearning process, they are not enough on their own to make the model forget significantly. the results are presented in additionally, our analysis suggests that the model still remembers much of its previous knowledge, even after the unlearning process. these findings highlight the need for additional and diverse approaches to enhance the effectiveness of unlearning. future studies should look into other techniques or a combination of methods to improve the unlearning process. ## conclusion in this study, we applied direct preference optimization (dpo) and supervised fine-tuning (sft) methods to achieve unlearning in the target model. our results show that these methods had only a limited effect on the model's performance, indicating that they are not sufficient by themselves for effective unlearning. this highlights the challenges involved in making a model truly forget specific information. to improve the unlearning process, it is important to explore additional strategies and combine different methods. future work should focus on developing new techniques and investigating how multiple approaches can work together to achieve better unlearning outcomes. and electronics engineers inc. funding information: we would like to thank the reviewers for their insightful feedback, and henry corrigan-gibbs for his service as the point of contact during the revision process. this work was supported by cifar through a canada cifar ai chair, and by nserc under the discovery program and cohesa strategic research network. we also thank the vector institute' sponsors. varun was supported in part through the following us national science foundation grants: cns-1838733, cns-1719336, cns-1647152, @xcite copyright: © 2021 ieee.; 42nd ieee symposium on security and privacy, sp 2021 ; conference date: 24-05-2021 through 27-05-2021. yinzhi cao and junfeng yang. 2015a. towards making systems forget with machine unlearning. 2015 ieee symposium on security and privacy, pages 463-480. yinzhi cao and junfeng yang. 2015b. towards making systems forget with machine unlearning. in 2015 ieee symposium on security and privacy, pages 463-480. ieee. cynthia dwork. 2006. differential privacy. in international colloquium on automata, languages, and programming, pages 1-12. springer.