i S humei-Chinchunmei at S em E val-2025 Task 4: A balanced forgetting and retention multi-task framework using effective unlearning loss (2025)

Abstract: As the Large Language Model (LLM) gains widespread adoption, increasing attention has been given to the challenge of making LLM forget non-compliant data memorized during its pre-training. Machine Unlearning focuses on efficiently erasing sensitive information from LLM under limited computational resources. To advance research in this area, SemEval 2025 Task 4: “Unlearning Sensitive Content from Large Language Models” introduces three unlearning datasets and establishes a benchmark by evaluating both forgetting effectiveness and the preservation of standard capabilities. In this work, we propose a more controllable forgetting loss, Effective Unlearning Loss, and explore its integration with various techniques to achieve more efficient and controlled unlearning. Our system ultimately ranked 5th on the competition leaderboard.

## introduction large language models (llms) have achieved remarkable success across various natural language tasks. however, llms tend to memorize sensitive information from their training data, posing potential risks such as privacy breaches and copyright violations @xcite . malicious attacker can exploit this vulnerability to extract confidential content, leading to unintended exposure. machine unlearning has emerged as a research field to address this issue, focusing on the following core challenges @xcite :(1) preserving essential information and capabilities while ensuring the removal of targeted data. (2) adapting to different data types. (3) balancing computational cost and efficiency. semeval-2025 task 4 introduce the challenge "unlearning sensitive content from large language models," aiming to establish a robust benchmark for evaluating the effectiveness of unlearning strategies in llms @xcite . the task encompasses three data categories: longform synthetic creative documents with different genres, short form synthetic biographies containing personal information, and real documents sampled from the target model's training dataset. each dataset includes predefined "forget" and "retain" sets, and encompasses two evaluation tasks: sentence completion and question-answering. the evaluation not only assesses the success of unlearning but also measures the impact on general capabilities using the mmlu benchmark. to encourage a balance between computational efficiency and performance, the organizers also impose runtime constraints on submitted solutions. in this competition, we propose effective unlearning loss (eul). this aims to erase knowledge related to the data to be forgotten by perturbing the model's gradients during training. we integrate this technique with the standard supervised fine-tuning (sft) process into a multi-task learning paradigm to ensure controllable unlearning. additionally, various data processing and augmentation strategies @xcite are explored to see their impact on the final performance. our contributions are as follows: ## background research on knowledge unlearning in large language models (llms) is an emerging field, with fine-tuning-based unlearning being one of the most common method. this involves retraining the model on datasets containing specific target knowledge to weaken its memory of the unwanted information. one intuitive unlearning method involves gradient ascent on the forget data, which increases the loss on that data to force the model to forget specified knowledge. however, this often leads to optimization instability and poor performance. to solve this, @xcite propose a comprehensive training approach that involves gradient ascent, standard gradient descent on the forget dataset, and minimizing kl divergence to maintain the model's performance on retained knowledge. similarly, @xcite introduces a gradual gradient ascent approach, which stabilizes the unlearning process and avoids instability. another unlearning method involves replacing the forgotten knowledge, such as @xcite and @xcite who replace forgotten answers with negative responses like "i don't know," while @xcite uses reinforcement learning to identify and replace key phrases. however, @xcite warns that relying solely on negative feedback for unlearning may result in non-sensical outputs and introduce privacy risks, reducing the model's effectiveness. in response, we propose a more pragmatic approach to unlearning in llms that combines multitask learning, data augmentation, and eul to facilitate faster and more efficient knowledge forgetting under constrained resources. ## system overview in this competition, we propose the effective unlearning loss (eul), which, combined with traditional supervised fine-tuning (sft), forms a multi-task learning framework. meanwhile, we observe a significant performance discrepancy between long and short outputs. to mitigate this issue, we incorporate data augmentation. the overall workflow is illustrated in figure 1 . section 3.1 provides a detailed introduction of eul, while section 3.2 demonstrates its integration with the original sft process to enable multi-task training. finally, section 3.3 discusses our data augmentation strategies. ## experiment in this competition, the organizers require unlearning to be performed on two models: olmo-7b and olmo-1b @xcite . the original tasks include sentence completion and questionanswering. to evaluate the effectiveness of unlearning, the organizers propose the following four evaluation metrics: we thoroughly explore the impact of different technical combinations on the final unlearning outcome and conduct ablation studies on the modules and hyperparameters. due to space limitations, all experimental results and analyses presented in the main body of this paper are based on olmo-1b, with the results for olmo-7b detailed in appendix a. all results presented are based on data publicly released by the organizers during the competition period. the results are presented in section 5 and section 6. all training is performed using lora (hu et al., 2022) for model fine-tuning, with rank, alpha, and dropout set to 8, 32, and 0.05, respectively. for eul, α is set as 1. for the training hyperparameters, we use the following settings:@xmath0, ## result due to computational constraints, the most balanced result we obtained before the competition deadline was (rd&nr&eul). therefore, our result on the leaderboard is (rd&nr&eul) result, not our best result. our best result was (rd&da&eul), which showed consistent performance across both olmo-1b and olmo-7b. the detailed results can be found in appendix a. ## ablation study we also explore the impact of different hyperparamter settings, including learning rate, epoch, magnitude scaling of eul, and the effectiveness of data augmentation. we present the results in the following sections, which are based on olmo-1b. the trends for olmo-7b are consistent with those of olmo-1b, and all results for both olmo-1b and olmo-7b can be found in the appendix a. ## conclusion the organizers of semeval-2025 task 4 introduce three machine unlearning datasets, design to assess unlearning capabilities from multiple perspectives and across various data types. this dataset effectively highlights the key challenges currently faced in machine unlearning. in this competition, we propose a more controllable forgetting loss, eul, which we integrate with standard supervised fine-tuning (sft) into a multi-task learning framework. this approach ensures precise unlearning while maximizing the retention of general capabilities. additionally, we incorporate various data processing and augmentation strategies to further enhance controllable unlearning. our experiments demonstrate the effectiveness of eul and underscored the critical role of standard sft in training on retained data. striking the right balance between forgetting and retention is essential for successful unlearning. furthermore, we introduce an effective data augmentation solution to address the long-tail distribution in text length. ultimately, our system ranks 5th on the official leaderboard.