Lacuna Inc. at S em E val-2025 Task 4: L o RA -Enhanced Influence-Based Unlearning for LLM s (2025)

Abstract: This paper describes LIBU (LoRA enhanced influence-based unlearning), an algorithm to solve the task of unlearning - removing specific knowledge from a large language model without retraining from scratch and compromising its overall utility (SemEval-2025 Task 4: Unlearning sensitive content from Large Language Models). The algorithm combines classical influence functions to remove the influence of thedata from the model and second-order optimization to stabilize the overall utility. Our experiments show that this lightweight approach is well applicable for unlearning LLMs in different kinds of task.

## introduction machine unlearning -the process of removing specific knowledge from a machine learning model without retraining from scratch -has emerged as a critical capability for large language models (llms) @xcite deployed in dynamic or privacy-sensitive environments @xcite . unlike traditional retraining, which is computationally prohibitive for llms, unlearning seeks to selectively erase influences of a forget dataset while preserving good performance on a retain set. this capability is essential for applications requiring compliance with data privacy regulations (e.g., gdpr "right to be forgotten" (mantelero, 2013)) or rapid adaptation to evolving content policies. existing approaches, for example, gradient ascent @xcite , often degrade general capabilities, as reflected in performance drops on benchmarks like mmlu @xcite , or demand computational resources that are impractical in real-world settings. to address these limitations, we propose a two-phase method called lora-enhanced influence-based unlearning (libu), which combines influence functions @xcite with the sophia optimizer @xcite . in phase 1, libu computes parameterwise updates using a fisher information approximation @xcite to minimize retain-set disruption. phase 2 refines the model via secondorder optimization, stabilizing training on noisy forget-set gradients. our submission, evaluated on the olmo-7b model @xcite , achieves a regurgitation rate of 0.283 while maintaining an mmlu accuracy of 0.469, exceeding the competition threshold of 0.371. ## background the semeval-2025 task 4: unlearning sensitive content from llms @xcite formalizes this challenge across three subtasks: (1) erasing long-form synthetic documents, (2) removing personally identifiable information (pii) from short biographies, and (3) unlearning real documents from the olmo pretraining corpus. the task demands balancing two competing objectives: achieving high regurgitation and mia scores on forget and retain sets, while preserving highest mmlu capability. to evaluate the submissions, @xcite release a comprehensive new benchmark named lume (llm unlearning with multitask evaluations). for each of the tasks, there are prompts for regurgitation and knowledge tests. the benchmark is split into forget and retain sets (in 1:1 ratio). two model checkpoints (7b and 1b parameters) were also finetuned to memorize this dataset. the formal task definition is as follows: let θ ∈ r d denote the model parameters, d retain the retain dataset, and d f orget the forget dataset. the unlearning objective is to compute parameter updates ∆θ that: ## system overview our method implements a two-phase approach to machine unlearning, designed to efficiently remove specific data influences, while preserving model performance on retained data. the main idea lies in combining influence-based parameter updates with second-order optimization, ensuring both precision and computational efficiency. unlike prior methods that approximate the full hessian matrix via woodfisher @xcite -a computationally prohibitive process requiring o(d 2 ) memory and prone to taylor expansion errors -our approach replaces explicit hessian inversion with a retain-set-derived diagonal fisher approximation. this avoids the instability of stochastic hessian estimates while ensuring updates prioritize parameters critical to retained knowledge. furthermore, our two-phase design (figure 1 ) decouples influence-based forgetting (phase 1) from sophia-driven stabilization (phase 2), eliminating approximation drift observed in joint hessian-gradient formulations. ## experiments experiment setup the experiments were conducted as part of the semeval-2025 competition, focusing on machine unlearning for three subtasks: (1) long-form synthetic creative documents, (2) short-form synthetic biographies containing personally identifiable information (pii), and (3) real documents sampled from the olmo training dataset. two model versions were trained on the designed algorithm and evaluated with olmo evaluation framework: the fine-tuned olmo-7b-0724-instruct-hf 2 (7b parameters) and olmo-1b-0724-hf 3 (1b parameters), with submissions constrained to a 1-hour runtime. due to computational constraints and a focus on validating the combined system's practical value, we leave fine-grained ablations of individual components (sophia, influence functions) as future work. preliminary results indicated that the components work better together, so we chose to prioritize evaluating the full system rather than isolating and testing each individual component separately. all experiments are conducted on a single nvidia a100 gpu. the final code also includes an option with deepspeed with implementation for distributed training on multiple gpus. ## results we tested three configuration setups ( more aggressive unlearning: achieved the highest score in second subtask with regurgitation rate of 0.83 on forget set, but severely degraded mmlu accuracy below predefined threshold (<0.371). • setup 2. more balanced unlearning: achieved the highest scores in subtask 1 and subtask 3 but got a lower score on the retain tasks. ## conclusion in this paper, we present libu, a two-phase unlearning framework for llms that combines influence-based parameter updates with secondorder sophia optimization, achieving competitive results in the semeval-2025 task 4. libu's lightweight design-leveraging lora for parameter efficiency and gradient accumulation for stability-enables precise removal of sensitive data while preserving model utility, exceeding the competition threshold. our experiments highlight the critical role of hyperparameter equilibrium, as conservative tuning balances unlearning efficacy with retention, whereas aggressive configurations risk catastrophic forgetting.