JU - CSE - NLP ’25 at S em E val-2025 Task 4: Learning to Unlearn LLM s (2025)

Abstract: Large Language Models (LLMs) have achieved enormous success recently due to their ability to understand and solve various non-trivial tasks in natural language. However, they have been shown to memorize their training data which, among other concerns, increases the risk of the model regurgitating creative or private content, potentially leading to legal issues for the model developer and/or vendors. Such issues are often discovered post-model training during testing or red teaming. While unlearning has been studied for some time in classification problems, it is still a relatively underdeveloped area of study in LLM research since the latter operates in a potentially unbounded output label space. Specifically, robust evaluation frameworks are lacking to assess the accuracy of these unlearning strategies. In this challenge, we aim to bridge this gap by developing a comprehensive evaluation challenge for unlearning sensitive datasets in LLMs.

## introduction large language model (llm) unlearning is selectively removing specific knowledge from a trained model while retaining its general capabilities. this is crucial in scenarios where models inadvertently memorize sensitive information, contain outdated or incorrect data, or need to comply with user requests for data removal under regulations like gdpr. unlike traditional model retraining, which is computationally expensive and impractical for large-scale models, efficient unlearning methods seek to erase targeted knowledge with minimal computational overhead. our system employs a two-pronged approach to tackle this challenge: normalized gradient difference (ngdiff) for selective unlearning and au-tolr for dynamic learning rate adaptation. ngdiff modifies model parameters by computing the gradient differences between retain and forget datasets, ensuring targeted forgetting while minimizing unintended knowledge loss. this method allows the model to maintain critical learned information while efficiently removing specific instances. autolr enhances this process by optimizing the learning rate through quadratic loss fitting. this allows the system to dynamically adapt its updates for faster convergence and stability. for every 10 iteration, autolr evaluates the model's loss behavior with different learning rates and selects the optimal value to ensure stable and effective parameter updates. the combination of ngdiff and autolr makes our system an efficient and scalable solution for the problem of selective unlearning. through our participation in this task, we observed that applying ngdiff with autolr significantly improved unlearning efficiency compared to static learning rate approaches. however, challenges remained, particularly in handling complex qa examples where forgetting was less effective. instances involving long-context dependencies or indirect knowledge retrieval were harder to unlearn, suggesting that additional refinements to the gradient normalization process could further enhance performance. additionally, we noted that aggressively tuning autolr in early epochs sometimes led to instability, indicating the need for more adaptive thresholding techniques. ## task description the challenge covered three sub-tasks spanning different document types: for each task, @xcite there were two types of evaluation -sentence completion and question answering (qa). a trained large language model (llm) was provided to memorize documents from all three tasks. for each subtask, there were specific retain (i.e., documents the model should retain in memory) and forget sets (i.e., documents the model should forget) along with the target model. the primary task was to develop an algorithm to unlearn the information present in the forget set without affecting the information present in the retain set. ## related work several unlearning methods have been explored, each addressing different challenges in forgetting specific information while maintaining generalizability. ## conclusion we formulate the machine learning problem and propose a novel ngdiff unlearning method based on the normalized gradient difference and the adaptive learning rate. by leveraging insights from multitask optimization, ngdiff improves forgetting quality while maintaining utility of the retain set. due to limited compute and data, we relied on lora (low-rank adaptation) to shrink the number of trainable parameters while preserving performance. we explored several adaptive learning-rate schedules to minimize loss, but extensive hyperparameter sweeps and larger-scale evaluations remain out of reach. in future work, we will investigate alternative parameter-reduction techniques-such as prompt tuning or sparsity-based pruning-and conduct more thorough ablation studies to further improve unlearning efficacy.