DUT ir at S em E val-2025 Task 4: Optimized Fine-Tuning of Linear Layers for Balanced Knowledge Forgetting and Retention (2025)

Abstract: This paper describes our system used in SemEval-2025 Task 4: Unlearning sensitive content from Large Language Models. In this work, we propose a method for controlling the fine-tuning of a model’s linear layers, referred to as CTL-Finetune (Control-Tuned Linear Fine-tuning). The goal of our method is to allow the model to forget specific information while preserving the knowledge it needs to retain. The method consists of four main components: 1) shuffling data labels, 2) shuffling label gradient calculation, 3) determination of control layers, and 4) fine-tuning using a combination of gradient ascent and gradient descent. Experimental results demonstrate that our approach effectively enables the model to forget targeted knowledge while minimizing the impact on retained information, thus maintaining the model’s overall performance.

## introduction large language models (llms) have achieved significant advancements in understanding and solving natural language tasks. however, during the training process, llms tend to memorize vast amounts of data @xcite , which may lead to the reproduction of creative content or private information. this, in turn, poses legal risks to model developers and suppliers. these issues are typically identified post model training during testing or red teaming. moreover, stakeholders may request the removal of their data from the model to protect copyright or exercise their right to be forgotten. however, retraining the model after each data deletion request is prohibitively costly and unsustainable. in light of these challenges, anil ramakrishna et al. introduced task 4, named "unlearning sensitive content from large language models," in semeval 2025 @xcite . this task aims to develop a comprehensive evaluation framework to effectively eliminate sensitive data from llms, thereby providing a novel solution for the application of unlearning techniques in the domain of llms. the three subtasks are designed with different types of textual data to evaluate the model's "forgetting" capability when handling sensitive information. these include: long-form synthetic creative documents (covering multiple genres), short synthetic biographies containing personally identifiable information (pii) such as fictional names, ssns, and home addresses, as well as real documents sampled from the target model's training dataset. these tasks aim to comprehensively test the model's ability to identify and eliminate sensitive content across various scenarios. for this task, we employ the fine-tuned olmo-7b-0724-instruct-hf model. building upon this foundation, we propose a method called ctl-finetune (controllable layer finetuning), which achieves selective @xcite @xcite information forgetting and retention by fine-tuning the model's linear layers. figure 1 shows the overview of our framework. this approach involves transforming data labels, calculating relevant gradients, identifying control layers, and combining gradient ascent with gradient descent during fine-tuning. this enables the model to erase specific information while preserving essential knowledge.our system ranked 9th in this competition. ## methodology in this work, we propose a fine-tuning approach for the olmo model, enabling it to selectively forget sensitive knowledge while minimizing the impact on the knowledge that needs to be retained. our method focuses on fine-tuning specific layers of the model, and it is composed of four main steps. by utilizing this method, we achieve a more nuanced control over the model's memory, allowing it to forget sensitive information without compromising the retention of important knowledge and the model's overall performance. the fine-tuning process ensures that the model adapts to new memory constraints while maintaining its core capabilities. ## conclusion in this paper, we introduced a novel approach for enabling selective forgetting in large pre-trained models while preserving their core knowledge. our method, which focuses on fine-tuning specific layers of the model, integrates data shuffling, gradient calculations, and selective updates through gradient ascent and descent. experimental results demonstrate that our approach can effectively forget sensitive information while maintaining the model's essential performance. the 7b model achieved a final score of 0.266, qualifying for the leaderboard, and showed promising results in terms of mia and task-aggregate scores, reflecting its ability to balance forgetting and retention. however, the 1b model, with its simpler architecture, exhibited a performance drop, indicating that model complexity plays a significant role in the effectiveness of the forgetting mechanism. overall, while our method provides a solid foundation for model unlearning, further refinements and optimizations are needed to improve its performance, particularly in terms of the forgetting process.