GIL - IIMAS UNAM at S em E val-2025 Task 4: LA -Min( E ): LLM Unlearning Approaches Under Function Minimizing Evaluation Constraints (2025)

Abstract: This paper describes Gradient Ascent and Task Vectors as LLM unlearning methodologies applied to SemEval 2025’s task 4. This task focuses on LLM unlearning on specific information under the constraints of preserving the model’s advanced text generation capabilities; meaning that our implementations of these algorithms were constrained both in the information datasets as well as the overall effect of each algorithm in the model’s general performance. Our implementation produced modified language models that ranked 7th out of 14 valid participants in the 7B parameter model, and 6th out of 24 in the 1B parameter model.

## introduction large language models (llms) are one of the most widely used nlp tools for multiple different purposes in and outside of academia and technological research. state of the art models require a substantial amount of training data in order to work at their fullest potential. however, these training datasets may contain personal information and confidential data from multiple sources. if utilized inappropriately, this could result in legal complications arising from infringements of copyright, or the right to be forgotten. given the probabilistic nature of llm text generation, and jail-breaking techniques, sensitive information is at risk of being generated in every day usage. the semeval 2025 task: unlearning sensitive content from large language models @xcite was created to solve this problem. this task works under the assumption that retraining these models from scratch and omitting sensitive information is computationally and economically expensive @xcite , meaning that the most efficient approach consists of applying algorithms that modify only certain model weights corresponding to the sensitive information. these modifications should translate to the model being completely unaware of said information, making it unable to generate it by accident, all while preserving the model's text generation capabilities. three different evaluation metrics were averaged to obtain the final score: a) the mmlu benchmark average @xcite , in which the unlearned model had to surpass a 0.371 threshold in order to be considered as a valid model, b) a membership inference attack score @xcite , based on attacks sampled from member and nonmember datapoints, and c) task specific regurgitation rates @xcite , sentence completion measurements focused on forget and retain pieces of information. in this paper, two unlearning strategies are proposed to solve the problem: 1) gradient ascent @xcite is employed on the data to be forgotten, followed by a fine-tuning step on the data to be retained; and 2) a task vector @xcite retraining is implemented to negate the embeddings of the data to be forgotten. the experiments conducted in this study demonstrate that the efficacy of the unlearning algorithms is influenced by the dimensionality of the model. this task was offered for two models, one with 1 billion (1b) parameters and one with 7 billion (7b) parameters. the 7b model was ranked 7th following the gradient ascent implementation, while the 1b model was ranked 6th after the task vector implementation. both rankings are among the 24 teams exhibited in the final evaluation table. the code can be found in github foot_0 . ## background in recent years, the issue of unlearning is being approached from various points of view. a noteworthy work in this area is that of @xcite , which developed a method that enables llms to avoid answering with content from input output subtask 1 who is the reclusive artist that shae offered shelter to during the stormy night? roz who did catherina seek to protect from marcile? the city of deadesius subtask 2 what is the birth date of fredericka amber? laura cretara which poetry collection by misra won the sahitya akademi award in 1986? dwa suparna articles such as @xcite , have made a compilation of works developed in the area to reevaluate the challenges of llm unlearning, attending to the need of the "right to be forgotten" @xcite . the main categories of challenges have been identified are the following: ## system overview the current state-of-the-art was taken into consideration, and two approaches that addressed the problem from different perspectives were selected for testing. the objective was to make a benchmark comparison. each approach is characterized by its own sub-section. the first, gradient ascent (ga), examines the problem from the perspective of the loss function, while the second, task vector (tv), explores the problem from the viewpoint of the model weights. in this work, we will refer to the set to be forgotten as d f and the set to be retained as d r . ## experimental setup all experiments were conducted on the allenai/olmo-1b-0724-hf foot_1 model. the model was quantized to 4-bit using the bitsand-bytesconfig function, and lora (low-rank adaptation) was employed to reduce resources and enhance efficiency and speed when applying the various approaches. we decided to take the most commonly used parameters to focus on the two proposed methods (ga and tv). this facilitates the comparison of results with the work developed in the state of the art. for lora we use the peft package with the following parameters: @xmath0, @xmath1, @xmath2.0 for finetune we use the trl package: sftconfig and sfttrainer. this makes the task easier. three metrics were used to evaluate this task: mmlu average: measured as test accuracy on 57 stem subjects, models that scored less than 0.371 were discarded to ensure that the model was not extremely damaged by forgetting data, i.e., forgetting much more than it should have. ## results it can be observed that the efficacy of each approach is model-dependent. for the 1b model, the taskvector approach yielded superior results ( this may be because we work with vectors, and these vectors can capture more than just d f , so negating them greatly affects the performance of the model, this means that the model forgets more things than it should. conversely, the most effective approach for the 7b model was found to be gradient ascent, followed by fine tuning ( example of an instance from gradient ascent with finetuning on 7b model: input -goldi aqua was born on march 29, 1976. she can be reached via phone at 565-577-9919 and email at goldi_aqua@me.com. her home answer to forget -address is 1011 san jose street, brooklyn park, md, 21225. her social security number is 900-57-8002. answer -address is: 730 n. broadway, san diego, ca 92101. example of an instance from taskvector on 7b model: input -what is tiffi magenta's social security number? answer to forget -900514344 answer -active cont cont cont settings out cont out le le le le le another observation is that the implementation of these approaches exclusively on the ner labels does not enhance the outcomes; in some cases, it has been observed that this approach can lead to a deterioration in the results, depending on the particular model in use. a subsequent analysis of the mia score results indicates a tendency towards a value of 1, suggesting a lack of sufficient unlearning. finally, we can mention that although the algorithms indicate that they work, we can notice that there is an area of improvement and this may be due in part to the fact that both approaches are an aggressive modification of the model. this, along with the fact that the experiments were carried out without freezing any layer, allowing the algorithms to modify the model at different levels and not only in the last layers, could be one of the reasons why the performance was not optimal. ## conclusion these unlearning methodologies generated notable changes in both versions of the language model, in some cases even more than necessary. task vectors theoretically modify only the weights corresponding to the forget information, yet in this case those weights were highly relevant to the overall capabilities of the model. as a matter of fact, task vectors did not manage to obtain non-zero scores in the task aggregate sections of the evaluation, and the model ultimately underperformed in the mmlu, disqualifying that methodology from full evaluation. freezing certain layers of the model could improve the results both in execution time and in evaluation results. gradient ascent, on the other hand, proved to be the better performing unlearning methodology. to our surprise, the unaltered application of this algorithm performed considerably better than a ner adjusted implementation, suggesting that this algorithm works better with raw data even if the information varies mainly in information that can be tagged with standard nlp methodologies. in future work, we plan to improve the finetuning parts by adjusting hyperparameters and freezing some layers of the model. additionally, a more detailed analysis of how the task vectors are created is necessary to ensure that no extra information is removed that affects the model's performance.