GUIR at S em E val-2025 Task 4: Adaptive Weight Tuning with Gradual Negative Matching for LLM Unlearning (2025)

Abstract: Machine Unlearning for Large Language Models, referred to as LLM Unlearning is getting more and more attention as a result of regurgitation of sensitive and harmful content. In this paper, we present our method architecture, results, and analysis of our submission to Task4: Unlearning sensitive content from Large Language Models. This task includes three subtasks of LLM Unlearning on 1) Long Synthetic documents, 2) Short Synthetic documents, and 3) Real Training documents. Getting rid of the impact of undesirable and unauthorized responses is the core objective of unlearning. Furthermore, it is expected that unlearning should not have an adverse impact on the usability of the model. In this paper, we provide an approach for LLM unlearning that tries to make the model forget while maintaining usability of the model. We perform adaptive weight tuning with Gradient Ascent, KL minimization and Gradual Negative Matching loss functions. Our submission balances retain and forget abilities of the model while outperforming provided benchmarks.

## introduction the explosion of information with learning mechanisms trying to capture data from every corner raises serious privacy concerns. comprehensive data privacy laws require commitment to protect sensitive and personal information. the legal mandate in the form of the european union's general data protection regulation, the california consumer privacy act, raises serious concerns with respect to the results produced by machine learning mechanisms and data sets used for learning. basically, large language models (llms) use large datasets to learn and memorize those data and regurgitate information when asked about it @xcite . however, that might include very sensitive information, such as personally identifiable information (pii) or harmful information. regurgitation of such copyrighted or harmful information poses some serious legal and ethical issues that make the use of llm in practical real-life applications questionable. a variety of methods have been proposed to address llm limitations @xcite but have not addressed regurgitation of sensitive data. one of the rudimentary solutions to handle this issue is retraining the model when any of such outcomes are detected. this could result in retraining the model again and again. this is simply very expensive and impractical when it comes to real life scenarios @xcite . these practical limitations of re-training further resulted in increasing interest in unlearning llms. in short, taking generative ai to a safe and legal level, demands llm unlearning. semeval-2025 task 4 of 'unlearning sensitive content from large language models' deals with llm unlearning on three types of documents. subtask 1 is on long form synthetic creative documents covering different genres. subtask 2 is on short form synthetic biographies that contain personally identifiable information (pii). pii includes names, contact details, ssn, and addresses. subtask 3 is on real documents sampled from the target model's training dataset. this task releases forget and retain sets along with finetuned llms inorder to unlearn. in our submission, to address the unlearning problem, we introduce adaptive weight tuning with two-stage deviation-based loss functions for unlearning. the approach proposed in this submission performs adaptive weight modifications to losses from specifically chosen set of unlearning loss functions to determine final loss value. we first performs a detailed study of loss functions in the literature to determine the most suited and effective ones for the task of llm unlearning. we then define a formulation to adjust the weights effectively and tune them with each iteration contributing to the final loss in each iteration. our submission considers gradient ascent on forget set, kl divergence on retain set, and most importantly gradual negative matching (gnm) @xcite , which is performed on gradual negative results that are systematically generated to make the model forget the forget set. the use of weighted sum of gradient ascent loss, kl divergence loss, and gnm loss in each iteration and adaptive weighting separate our submission from other approaches in the literature and provided benchmarks. this achieves the unique objective where the weights of the loss function on the retain set go on increasing and that of the loss function on the forget set go on decreasing with each iteration. this approach makes sure to free the method from the catastrophic collapse and maintains usability of the model while achieving the unlearning objective. we compare the results with the benchmarks provided by the task organizers and show that our submission outperforms them. our contributions are as follows. ## related work the efforts to make llm more applicable resulted in increased research efforts in the area of llm unlearning. this is mainly to address concerns related to trustworthiness @xcite , fairness, copyright, and privacy @xcite , and sensitive knowledge. previous unlearning approaches used mainly alignment techniques to achieve the unlearning objective @xcite . such techniques aim to deliver expected results for specific inputs. gradient ascent is basically the retrogression of gradient descent learning @xcite . but it results in poor retention. this led to efforts to improve retainability. with this objective, the gradient ascent approach is combined with methods that could provide higher retention. rather than simply applying gradient ascent, use of gradient ascent on the forget set is combined with gradient descent on retain set @xcite . the gradient ascent-based unlearning comes with the challenge of catastrophic collapse resulting in serious harm to model usability @xcite . this seriously limits the usability of this approach. to counter this issue, other approaches are proposed to use on retain set. one of such approaches is the use of the kl divergence term on retain set @xcite . the objective was to make the model forget what is expected to forget and retain what is already learned useful information. this led to efforts where multiple combinations of gradient ascent, gradient descent, and kl divergence were used @xcite . further, in order to obtain the best retain-forget tradeoffs gradual negative matching (gnm) was proposed @xcite . gnm is a two-stage approach where the first stage involves generation of gradual negative outputs for forget set inputs. while the second stage matches these input, generated output pairs through gradient descent. additionally, in llm unlearning it is necessary to have a relevant benchmark and proper mechanisms for evaluations. furthermore, the benchmark needs to be specific to the application and context. this led to different benchmarks. some of such popular benchmarks include harmful content @xcite , copyrighted books (eldan and russinovich, 2023), biographies @xcite , pii, and creative documents (semeval'25-task4). a fr-rouge based evaluation is proposed to measure the effectiveness of unlearning models @xcite . in general, having sensible handling of sensitive information where the legal and ethical violations by llm regurgitation of sensitive information could be avoided poses a need for better llm unlearning methods. with this need in focus, in this submission, we propose a method which comprises of gradient ascent, kl divergence and gradual negative matching loss functions along with adaptive weight tuning. ## data and setting the semeval 2025 task 4 of 'unlearning sensitive content from large language models' (semeval'25-task4; @xcite has released forget and retain sets for both questionanswering and sentence completion across the three subtasks of synthetic long, synthetic short and real training documents. they have also released the train and validation parts of this data. additionally, they also provided fine-tuned open source llm olmo-7b-0724-instruct-hf (groeneveld et al., 2024), trained to memorize documents from three document types. the first one contains long synthetic creative documents. they include different genres. while the second one has short synthetic biographies where fake personal information is present. this includes pii with fake names, phone number, social security numbers, email and home addresses. the third type is a sample of real documents used for training the original llm. evaluation is performed across three metrics namely: task-specific regurgitation rates measured using rouge-l and exact matching scores, membership inference attack (mia) score and model performance on mmlu benchmark. further, a threshold of 75% of pre-unlearning checkpoint is placed on mmlu score to maintain a minimum model utility. finally, using arithmetic mean, a final aggregate score is calculated from the above three metrics. for calculating task-specific regurgitation rate, rouge-l scores are evaluated for sentence completion and exact matching scores are evaluated for question answering on both forget and retain sets across the three subtasks. the final task-aggregate is determined by considering harmonic mean of the six retain set scores and six inverted (1-score) forget set scores. for calculating mia scores a sample of member and non-member data is released. final mia score is defined as 1-|mia loss auc score-0.5| * 2. the mmlu score is calculated on the mmlu benchmark consisting multiple choice questions on 57 stem subjects. the objective is to develop an unlearning method that effectively unlearns information in the forget set without affecting model usability i.e. with minimal model degradation. ## loss function components gradient descent is the most common choice for fine-tuning llms. gradient descent as shown in equation 1 is based on cross-entropy loss summed over all tokens in the output sequence. here, we note that the cross entropy is calculated only for tokens in output y and not for input x. ## results and analysis on the other hand, gradient ascent leads to mia score of 0.912 and mmlu average score 0.269, resulting in an overall aggregate of 0.394. and, kl minimization leads to mia score of 0.916 and mmlu average score of 0.269 resulting in an overall aggregate of 0.188. here, we note that both gradient ascent and kl minimization lead to an mmlu average of 0.269 which is below 75% of pre-unlearning model mmlu threshold (0.371). hence, these models are discarded because of the highly degraded utility of the unlearned model. we note a trade-off between task aggregate and mia scores, and decide to prioritize task aggregate giving importance to addressing regurgitation and maximizing final aggregate score. further analysis as evident in figure 1 show that our submission does not lead to catastrophic collapse i.e. complete degradation of model. it is also evident that our submission results in significant unlearning of the model but at the same time a trade-off with retain effectiveness is observed. we also note that our proposed approach leads to better performance than the benchmarks specifically on the input output pairs of shorter length i.e. question answering and short synthetic documents. ## conclusion llm unlearning has gained importance due to gdpr and ccpa laws in europe and united states respectively. further, regurgitation of sensitive and harmful content is a violation of ethics and moral values. in the paper, we present a detailed architecture of our submission to the semeval 2025 task4: 'unlearning sensitive content from large language models.' we show that our submission outperforms the provided benchmarks. further, we also show that our submission performs unlearning while maintaining above threshold model usability while avoiding catastrophic collapse.