Cyber for AI at S em E val-2025 Task 4: Forgotten but Not Lost: The Balancing Act of Selective Unlearning in Large Language Models (2025)

Abstract: Large Language Models (LLMs) face significant challenges in maintaining privacy, ethics, and compliance, when sensitive or obsolete data must be selectively removed. Retraining these models from scratch is computationally infeasible, necessitating efficient alternatives. As part of the SemEval 2025 Task 4, this work focuses on the application of selective unlearning in LLMs to address this challenge. In this paper, we present our experiments and findings, primarily leveraging global weight modification to achieve an equilibrium between effectiveness of unlearning, knowledge retention, and target model’s post-unlearning utility. We also detail the task-specific evaluation mechanism, results, and challenges. Our algorithms have achieved an aggregate score of 0.409 and 0.389 on the test set for 7B and 1B target models, respectively, demonstrating promising results in verifiable LLM unlearning.

## introduction large language models (llms) have revolutionized the way artificial intelligence can be used, adapted, and integrated, demonstrating unprecedented capabilities across various domains and use cases @xcite . in order for llms to provide optimal and factual responses, they often require to be trained on vast amount of diverse information which not only includes the world data, but could also contain sensitive application-, task-, or entity-specific data @xcite . training on such massive datasets typically introduces critical challenges related to bias, ethics, and privacy concerns @xcite . further, at times, the data providers might also want to have no traces of their data at a later point due to reasons such as confidentiality, legal issues, change in terms, etc @xcite . retraining llms to exclude specific data is computationally expensive and impractical @xcite , selective unlearning in llms @xcite ) helps to achieve this exact objective. it is a mechanism through which the requested information can be precisely removed from the model's parametric memory along with preserving the model's knowledge integrity and utility for downstream tasks, without retraining it from scratch. the requested information can be a specific knowledge, model's certain behavior, a feature, or its ability to perform a particular task, or a combination of two or more of these and more. figure 1 depicts the crux of selective unlearning along with the task's evaluation mechanism which is discussed in section 3.3. unlearning in llms is a niche yet increasingly critical area that offers key advantages such as cost-effectiveness, computational efficiency, and precise intervention. it plays a crucial role in eliminating embedded biases @xcite , erasing toxic or harmful responses @xcite , and reinforcing ai guardrails @xcite in safety-critical fields such as healthcare, finance, and enterprise settings. however, ensuring effective and verifiable unlearning is challenging, as many approaches risk leaving residual traces of removed knowledge or inadvertently impairing broader model capabilities. achieving the right balance between unlearning effectiveness, knowledge retention, and model generalization is a delicate optimization problem that continues to drive research in this field @xcite . ## related works the approaches to unlearning in llms can be broadly classified into four categories: global weight modification, local weight modification, architecture modification, input/output modification @xcite . global weight modification involves updating all the model parameters while unlearning, thus, ensuring better guarantee of forgetting the requested information. it includes approaches such as gradient ascent @xcite , gradient difference @xcite , knowledge distillation @xcite , kl minimization @xcite , weight perturbation @xcite , and so on. these approaches are well suited for smaller models and provide strong unlearning, however, are resource intensive for larger models, as the training costs greatly increase with increase in the number of parameters. global weight modification for larger models also strengthens the problem of optimizing effective unlearning, and preserving model's capabilities. local weight modification identifies a subset of parameters that are required to be modified and accordingly updates only those model parameters @xcite @xcite , thereby, minimizing the computational efforts needed. nevertheless, the right set of parameters that are required to be modified might vary based on the diversity of the requested information. identifying the same is thus, challenging which therefore, has chances of leaving traces of unlearning, or in other words, influence of the requested information could still be observed in the model's behavior @xcite . architecture modification based approaches involve tweaking the model's architecture such as by adding additional layers @xcite , or by using external modules @xcite in addition to the target model, etc. these approaches, while advantageous in other contexts, were not suitable for this specific task's setup. finally, the input/output modification, as the name suggests, involves approaches that do not achieve true unlearning but modifies the model's input, and sometimes the output, in such a way that the final response is as desired, by leveraging techniques such as soft prompting @xcite , preference optimization @xcite , in-context learning @xcite . ## task artifacts this section describes the artifacts given by the task organizers @xcite namely: dataset, models, and mia dataset, which have been used for this task of unlearning. ## experiments and results a variety of experiments have been tried to understand the patterns in the given datasets, and figure out the suitable approaches that would balance the gradient-based methods: due to computational constraints, we have considered two configurations of the models for executing these methods: 1b model is trained as is, and 7b model is trained in a 4-bit peft configuration, described in a detailed comparison of all the variants of these experiments along with the corresponding training configurations is presented in tables 8, 9 of appendix b for 1b and 7b models respectively. gradient ascent: in this method, the target model was trained on the forget set with an inverted loss, thereby, making it to unlearning the training set rather than learning it. by maximizing the loss on the forget set, this method forces the model to unlearn. it was observed that learning rate (lr) and weight decay (wd) variations have a strong impact on the unlearning intensity. in particular, aggressive configurations led to stronger unlearning, achieving a mia score of 0.807 by the 1b model, when lr and wd were increased despite halving the number of epochs (e). on the other hand, with a steady training for 10 epochs, with relatively lesser lr, achieved even more unlearning in the 7b model, however, costing its utility -the model's mmlu average accuracy (mmluaa) almost got halved compared to the original 7b model. nonetheless, it was noted that the mmluaa has not dropped much in the case of 1b model, despite reaching a similar mia score. this indicates that gradient ascent though achieves a good balance between model's utility and the level of unlearning in smaller models, it tends to easily destabilize large models. quantization in the 7b model may have also enhanced unlearning, potentially due to increased numerical instability aiding divergence from the learned state. ## conclusion this work explores the use of targeted unlearning in llms where we have experimented with several unlearning methods with different configurational settings to make the target models forget the requested dataset, and preserve the specified retain set, also, preserving its overall multifaceted capabilities. from our experiments, for 7b model: gradient difference followed by gradient ascent worked well with appropriate parameters tuning; and for 1b model: gradient descent alone on the retain set worked well. xavier initialization on the 1b model has got near equivalent score on all the metrics as the former. followed by similar performance between controlled gradient ascent, and gradient difference, with respective appropriate parameters tuning. this work reemphasizes the fact that selective unlearning comes with the delicate problem of optimizing effective unlearning with knowledge retention of the remaining data and model's integrity, utility for downstream tasks. further, it demonstrates that performance of a method significantly depends on the scale of the target model, and the kind of data it is presented with.