AILS - NTUA at S em E val-2025 Task 4: Parameter-Efficient Unlearning for Large Language Models using Data Chunking (2025)

Abstract: The {textit{Unlearning Sensitive Content from Large Language Models}} task aims to remove targeted datapoints from trained models while minimally affecting their general knowledge. In our work, we leverage parameter-efficient, gradient-based unlearning using low-rank (LoRA) adaptation and layer-focused fine-tuning. To further enhance unlearning effectiveness, we employ data chunking, splitting forget data into disjoint partitions and merging them with cyclically sampled retain samples at a pre-defined ratio. Our task-agnostic method achieves an outstanding forget-retain balance, ranking first on leaderboards and significantly outperforming baselines and competing systems.

## introduction large language models (llms) have revolutionized natural language understanding and generation, spanning a large range of tasks such as question-answering @xcite , reasoning @xcite , summarization @xcite and others, showcasing unprecedented scalability and adaptability to novel tasks. however, this remarkable progress is accompanied with several challenges, one of them being their tendency to memorize data @xcite , leading to the inadvertent leakage of private and copyrighted information, an issue tied to several practical implications @xcite @xcite . in response to the ethical and legal reverberations, the area of machine unlearning has gained prominence, focusing on the deletion of targeted information from trained models. initial unlearning endeavors bridge the gap between data protection @xcite and differential privacy @xcite , focusing on removing individual data points from classifiers @xcite . such seminal works pose the main challenge of unlearning, which targets deleting individual data points without re-training the whole network from scratch. still, challenges such as the catastrophic forgetting @xcite , as well as the stochasticity @xcite and incremental nature @xcite of training, showcase the emerging particularities of unlearning algorithms. the convergence of unlearning and llms arises as a nascent research field accompanied by several challenges, due to their vast and opaque pretraining, large-scale data inter-dependencies, and unbounded label spaces, making it difficult to identify and isolate specific data representations within the model, not to mention efficiently removing them @xcite . in our work, we explore unlearning strategies on trained llms, primarily focusing on fine-tuning, successfully deleting targeted data points without deteriorating the llm's general knowledge. specifically, we investigate parameter-efficient gradient-based methods @xcite leveraging data chunking to improve unlearning effectiveness. to achieve this, we employ low-rank adaptation (lora) methods @xcite or selectively fine-tune only the last layers of the model. this approach not only enhances training speed and efficiency, but also introduces a regularization effect mitigating catastrophic collapse by preserving a portion of the base model's weights. as a result, our approach ranked first, surpassing the second best by a large margin. in summary, our method: task description adhering to the established unlearning frameworks, this task introduces a novel dataset which comprises a retain set d r and a forget set d f @xcite . the goal is to unlearn information contained in the forget set without affecting information present in the retain set or degrading the performance of the model on general tasks. data are divided into three subtasks spanning various language styles: long-form synthetic creative stories, short-form synthetic biographies containing personally identifiable information (pii) and real wikipedia documents. moreover, each subtask comes in two types: whole sentences for sentence completion (sc) and question-answer (qa) pairs. examples are presented in related work machine unlearning has gained widespread attention @xcite , driven by emerging data privacy concerns and the pursuit of model robustness. unlearning was first explored under partitioning data into disjoint sets to impose re-training only on the shards on which forgetting has been requested @xcite . to relieve the burden of full retraining for the affected shard, @xcite propose a method that achieves statistical equivalence between the postdeletion state and the state that would have existed without deletion. @xcite removes undesirable features and then enforces learning 'good' ones. deviating from retraining, @xcite utilize gradient ascent (ga) instead of gradient descent to achieve targeted unlearning with only a few parameter updates. ga serves as a practical unlearning strategy in llms @xcite , efficiently intervening with token probabilities, making undesirable generations improbable. incorporating well-suited loss functions and data-adaptive lora initializations helps to resolve ga instabilities when combined with lora tuning for unlearning @xcite . ## system overview we propose a novel training scheme for unlearning by introducing modifications and extensions to the gradient difference framework. a key aspect of our approach is partitioning the forget set into distinct chunks and processing them sequentially. this design is inspired by @xcite , who suggest that sequential unlearning enhances stability. ## experimental setup dataset the retain and forget data ratio is ∼1:1 across all splits and subtasks. the train and validation splits were released before evaluation to facilitate experiments and hyperparameter tuning. the final evaluation is conducted on a private, heldout test set, closely matching the train set in both size and retain-forget ratio. unlearning model the organizers released two models of different sizes for system evaluation: one 7b parameter based on olmo-7b-0724-instructhf, and another 1b parameter based on olmo-1b-0724-hf. both models were fine-tuned to memorize documents from all three subtasks. ## results our method achieves leading performance based on the final evaluation score. experimentation shows that there is a trade-off between ta and mia, clearly reflected in other teams' submissions. some of them manage to achieve near-perfect ta with minor degradation on mmlu, yet mia remains extremely low (e.g. 2 nd best in our approach differentiates from all others in that it manages to balance all three evaluation criteria, achieving high ta and mia scores with just slight degradation of the model's reasoning abilities. moreover, it performs similarly well on the 1b parameter model (ranked first with final score of 0.688) verifying the robustness of our method across different model sizes. app. c contains extensive results, including plots and detailed tables. ## conclusion in this work, we demonstrate the merits of combining parameter-efficient model tuning with strategic data chunking to effectively unlearn targeted content from pre-trained models while minimizing catastrophic forgetting. our task-agnostic system schedules retain and forget chunks appropriately, attaining superior balance between erasing sensitive information and preserving general knowledge. yuanshun yao, xiaojun xu, and yang liu. 2024b. large language model unlearning. preprint, arxiv:2310.10683. haopeng zhang, philip s. yu, and jiawei zhang. 2024a. a systematic survey of text summarization: from statistical methods to large language models. preprint, arxiv:2406.11289. ruiqi zhang, licong lin, yu bai, and song mei. 2024b. negative preference optimization: from catastrophic collapse to effective unlearning. preprint, arxiv:2404.05868. hattie zhou, ankit vani, h. larochelle, and aaron c. courville. 2022. fortuitous forgetting in connectionist networks. arxiv, abs/2202.00155.