NEKO at S em E val-2025 Task 4: A Gradient Ascent Based Machine Unlearning Strategy (2025)

Abstract: The power and wide application of large language models (LLMs) has brought the concerns on its risk of leaking private or sensitive information. However, retraining the modules is expensive and impractical, which introduces machine unlearning - removing specific information from language models while preserving general utility. Task 4 at SemEval 2025 consists of a shared task with this exact objective. We present an approach which combines gradient ascent-based forgetting with Kullback-Leibler (KL) divergence-based retention, applied to a 1-billion-parameter causal language model. Despite achieving effective forgetting, the system struggles with maintaining model utility. Our experiments reveal critical trade-off between unlearning effectiveness and performance preservation, highlighting challenges in practical machine unlearning implementations.

## introduction large language models (llms) have demonstrated remarkable capabilities in generating human-like text @xcite , while there are growing concerns about data privacy in the interactions. their ability to memorize vast amounts of data may lead to significant ethical and security issues @xcite , including enhancing societal biases and stereotypes, generating sensitive or harmful content, private data leakage, being vulnerable to jailbreaking or other security attacks, or potential misuses for cyberattacks @xcite @xcite @xcite @xcite . there is an urgent need for solutions that maintain a balance between ensuring the safe use of llms and preserving their utility to effectively meet user needs @xcite . given the substantial time and resources required to train llms, retraining them to eliminate harmful influences is often impractical @xcite . as an alternative, machine unlearning has emerged as a method for selectively removing the influence of undesirable data from pre-trained models @xcite . machine unlearning (mu), defined as "forgetting undesirable misbehaviours on large language models (llms)" @xcite , aims to eliminate the influence of unwanted data, such as sensitive or illegal information, while maintaining the integrity of essential knowledge generation and not affecting causally unrelated information @xcite . the semeval-2025 task 4 on machine unlearning @xcite is a shared task focused on machine unlearning for llms. participants are tasked with developing methods to remove specific knowledge from a given trained model without retraining it from scratch. the goal is to ensure the model forgets the designated forget set while maintaining accuracy on the retain set. this challenge consists of three english-language subtasks: our system participated in all three subtasks with the intention to implement and validate a widely adopted unlearning strategy, namely gradient ascent (ga). we employed a dual-objective optimisation strategy that combines gradient ascent and kullback-leibler (kl) divergence. ga maximizes the loss on the forget set, driving the model to unlearn specific information, while kl minimisation preserves general knowledge by minimizing divergence from the pre-trained model. this iterative process balances these objectives, ensuring targeted forgetting without severe degradation of overall performance. we implemented our approach on a 1-billion-parameter model due to computational constraints. the evaluation relied on sentence completion and question and answer (q&a) tests to measure both forgetting effectiveness and the retention of general knowledge. the details will be unfolded in the following sections. ## conclusion our experiments faced several practical challenges that influenced both training and model performance. a key constraint was the selection of a 1b parameter model instead of a 7b variant due to computational limitations. while necessary for efficiency, this decision likely contributed to performance degradation, as smaller models struggle to balance knowledge retention and unlearning. gpu limitations further restricted our approach. running both teacher and student models concurrently led to high memory consumption, reducing batch sizes and limiting additional loss components like random answer loss. this required careful hyper-parameter tuning with minimal architectural modifications to maintain a feasible balance between unlearning and retention. despite these challenges, our systematic adjustments provided valuable insights into optimizing unlearning strategies under resource constraints. future work should explore more efficient parameter-sharing techniques or distillationbased approaches to mitigate computational burdens while maintaining effectiveness. addressing these limitations will be essential for advancing unlearning methodologies in large-scale models.