ZJUKLAB at S em E val-2025 Task 4: Unlearning via Model Merging. (2025)

Abstract: This paper presents the ZJUKLAB team’s submission for {emph{SemEval-2025 Task 4: Unlearning Sensitive Content from Large Language Models}}. This task aims to selectively erase sensitive knowledge from large language models, avoiding both over-forgetting and under-forgetting issues. We propose an unlearning system that leverages Model Merging (specifically TIES-Merging), combining two specialized models into a more balanced unlearned model.Our system achieves competitive results, ranking {textbf{second among 26 teams}}, with an online score of 0.944 for Task Aggregate and 0.487 for overall Aggregate. In this paper, we also conduct local experiments and perform a comprehensive analysis of the unlearning process, examining performance trajectories, loss dynamics, and weight perspectives, along with several supplementary experiments, to understand the effectiveness of our method.Furthermore, we analyze the shortcomings of our method and evaluation metrics, emphasizing that MIA scores and ROUGE-based metrics alone are insufficient to fully evaluate successful unlearning. Finally, we emphasize the need for more comprehensive evaluation methodologies and rethinking of unlearning objectives in future research.

## introduction unlearning has emerged as a critical technique in ai systems, enabling the selective removal of sensitive data, including copyrighted material and personal information, from trained models. as the international ai safety report @xcite emphasizes, unlearning plays a vital role in mitigating privacy and copyright risks associated with extensive training datasets. however, it also acknowledges that current unlearning methods remain in-adequate, which often fail to completely erase targeted data while potentially degrading model performance, thus limiting practical implementation. specifically, existing unlearning methods often struggle with over-forgetting (excessive elimination of non-sensitive information) or underforgetting (incomplete removal of sensitive data). it is challenging to find optimal hyperparameters that balance performance across multiple evaluation dimensions, sometimes even impossible. to address these limitations, we propose a novel unlearning system that leverages model merging to combine an over-forgetting model with an under-forgetting model, creating a more effective unlearned model. it can produce superior results simply by merging two models with complementary biases. our system achieved second place in semeval-2025 task 4: unlearning sensitive content from large language models, with our 7b model attaining a task aggregate score of 0.944 and aggregate score of 0.487, demonstrating the effectiveness of our system in selectively removing sensitive content. furthermore, our local experiments yielded almost perfect results with a mia score of 0.501 and aggregate score of 0.806, while maintaining an exceptionally high task aggregate and comparable mmlu avg.. we provide comprehensive analyses that validate our system's effectiveness and offer deeper insights into the unlearning process. for details about task description, please refer to the official paper @xcite . ## methodology as illustrated in figure 1 , our unlearning system follows two phases. (1) the training phase develops two complementary models, each exhibiting strong performance. (2) the merging phase merges these models, leveraging their strengths to achieve effective and balanced unlearning. ## conclusion this paper introduce an unlearning system via model merging. by combining two complementary models, it effectively achieves balanced forgetting and excellent knowledge preservation. @xcite ). an alternative strategy, referred to as "locate-then-unlearn," is exemplified by knowundo @xcite and sure @xcite , which focus on knowledge localization before executing the unlearning process. additionally, data-driven methods for unlearning have also been introduced, such as those proposed by @xcite @xcite @xcite @xcite . several works have explored the use of model merging techniques to achieve unlearning @xcite . model merging training a model for each task can be costly, but model merging offers a solution to these challenges by combining multiple pretrained models. model merging strategies include parameter averaging (linear), singular value decomposition (svd) for low-rank alignment, and feature concatenation (cat). advanced variants like ties @xcite trim redundant parameters and resolve sign conflicts, while ties-svd @xcite integrates svd for refined fusion. dare methods @xcite , and methods like dare-ties, dare-linear introduce parameter dropout and rescaling, with extensions (dare-ties-svd, dare-linear-svd) combining svd for structured compression. the magnitude-prune @xcite removes lowimpact weights, and its svd variant (magnitudeprune-svd) is further compressed via low-rank decomposition.