Atyaephyra at S em E val-2025 Task 4: Low-Rank Negative Preference Optimization (2025)

Abstract: We present a submission to the SemEval 2025 shared task on unlearning sensitive content from LLMs. Our approach employs negative preference optimization using low-rank adaptation. We show that we can utilize this combination to cheaply compute additional regularization terms, which help with unlearning stabilization. The results of our approach significantly exceed the shared task baselines.

## introduction transformer-based large language models (llm) trained on large data corpora have shown remarkable performance in many natural language processing tasks. however, their ability to remember portions of the training data @xcite might raise legal, ethical, and other issues. these consist of llms regurgitating copyright-protected creative content learned from the web or private personal information such as social security numbers, addresses, and others. for the latter, regulations such as the eu's general data protection regulation (gdpr) or the california consumer privacy act (ccpa) mandate the right for the removal of such information from the training data as per the "right to be forgotten". unfortunately, these issues are often discovered only after model training. although discarding such sensitive items from the training data sets and subsequent retraining is possible, it is generally prohibitively expensive. the field of machine unlearning tackles this exact issue by treating the removal of information as model fine-tuning. although several state-of-the-art approaches and benchmarks exist for llm unlearning @xcite , the field is still relatively unexplored. to facilitate further progress in the field, @xcite developed a comprehensive evaluation challenge for unlearning sensitive datasets in llm as a part of the international work-shop on semantic evaluation. foot_0 this paper presents our submission to the shared task. to solve the task, we utilized the state-of-the-art unlearning method negative preference optimization (npo; @xcite . we combined this method with low-rank adaptation (lora; @xcite because we consider the computational effectiveness of unlearning to be of high importance. although several approaches use parameter-efficient methods for unlearning in transformers @xcite @xcite , the combination of npo and lora is novel. we show that with lora, we can cheaply compute additional regularization terms that use the original model's output distribution without any memory overhead. we further show that including the kl divergence minimization regularization stabilizes the unlearning for a higher number of epochs. furthermore, our solution significantly outperforms the shared task baselines. we release the source code of our submission on github. foot_1 2 task background the goal of the shared task @xcite is to build a method for unlearning information from a given target large language model. for a given model, a forget set d fg , and a retain set d rt , the method should be able to remove the information present in the forget set from the model while preserving the data from the retain set and not deteriorating the model's performance on unrelated tasks. as the target model, the task organizers utilized olmo (a pre-trained llm), specifically its 7b and 1b versions @xcite . since olmo is trained on an open dataset dolma @xcite , it makes for a good choice for this task. these target models were further fine-tuned to for each batch, we compute our training loss shown in equation 5 in two passes. during the first pass, we leave the lora layers enabled and compute the l rt term of our loss. during the second pass, we compute the k rt and l npo terms by disabling the lora layers and only utilizing the backbone. this way, we do not need to maintain a separate copy of the backbone. ## method overview our approach for this task combines negative preference optimization (npo; @xcite and low-rank adaptation (lora; @xcite for parameter-efficient fine-tuning. ## experiments for our experiments, we focused on the fine-tuned olmo-7b model, which we unlearned using the training retain and forget sets, all provided by the task organizers. we chose a fixed value β = 0.5 of the npo loss hyper-parameter and @xmath0, as these values gave us reasonably good results. we experimented with various values for the hyper-parameters γ and δ. the learning rate was set to 10 -4 with a batch size of 4. we perform a broader hyper-parameter search on the olmo-1b model in appendix a. we used four combinations of values for γ, δ. in three of them, we set δ = 1 and experimented with values 0, 0.5, and 1 for γ to determine the effect of k rt on the deterioration of the model. in the last run, we only kept k rt with δ = 1 and set γ = 0. we ran five independent runs with unique seeds for each of the combinations. following the shared evaluation scheme, we measure four scores to evaluate the quality of our solution. first, we calculate the rouge-l score @xcite for each sample in the validation sets. by computing the score separately for sentencecompletion and question-answering pairs of each document type in the forget and retain sets, we obtain 12 values. we invert the values for the forget sets and produce the task score by merging the 12 resulting values using the harmonic mean. we perform a loss-based membership inference attack (mia; @xcite and scale the resulting mia score s mia to penalize both under-run epoch task score ↑ mia score ↑ mmlu ↑ final score ↑ µ σ µ σ µ σ µ σ γ = 1, δ = 0.0 10 .431 .020 .657 .269 .461 .010 .516 .089 20 .449 .029 .389 .117 .449 .008 .429 .043 γ = 1, δ = 0.5 10 .349 .045 .375 .193 .462 .005 .395 .069 20 .434 .021 .594 .201 .439 .017 .489 .074 γ = 1, δ = 1.0 10 .349 .041 .165 .083 .465 .005 .327 .027 20 .453 .016 .620 .219 .449 .016 .507 .072 γ = 0, δ = 1.0 10 .369 .080 .699 .170 .441 .020 .503 .056 20 .332 .085 .400 .113 .370 .054 .367 .021 baseline npo -.021 -.080 -.463 -.188 -baseline gd -0 -.382 -.348 -.243 -original model 0 0 -0 -.510 -.170 - to measure the degradation of the model after unlearning, we use the mmlu benchmark @xcite . finally, we compute the final score as the arithmetic mean of the previous three scores. ## results in in general, our runs considerably outperform the provided baseline solutions. the runs that used l rt overall did not degrade the mmlu score as much as those where only k rt was involved. including k rt with positive δ in the loss slowed the training. however, since achieving a high mia score required hitting a sweet spot between underand over-training, including k rt helped stability in the long run. we placed lower than expected on the 7b model in the task leaderboard. this was caused by a logical error in our submission script, which effectively ## conclusion the combination of npo and lora proved to be an effective strategy for llm unlearning. in addition to significant memory usage improvements, lora allows us to cheaply compute an additional regularization term, stabilizing the unlearning for a higher number of epochs. in our future work, we wish to further investigate the effect of lora and its rank on the resilience of the model to quality deterioration.