YNU at S em E val-2025 Task 4: Synthetic Token Alternative Training for LLM Unlearning (2025)

Abstract: This paper describes our system submitted to SemEval-2025 Task 4, which introduces the Synthetic Token Alternative Training (STAT) algorithm for efficient unlearning in large language models (LLMs). The proposed method aims to enable pretrained models to selectively forget designated data (the forget set) while preserving performance on the remaining data (the retain set).The STAT framework adopts a dual-stage process. In the first stage, pseudo tokens are generated through random sampling and applied to the forget set, facilitating more effective targeted unlearning. In the second stage, the model undergoes gradient-based optimization using an alternative training scheme that alternates between pseudo-token-augmented samples from the forget set and unmodified samples from the retain set. This design promotes stable unlearning of the specified data while accelerating convergence and preserving the model’s general performance.Our system achieved 3rd place in the 7B model track (OLMo-7B) and 7th place in the 1B model track (OLMo-1B), demonstrating substantial improvements over the official baselines, exhibiting superior stability in knowledge retention and more effective targeted forgetting compared to existing approaches.

## introduction the task of large language model (llm) unlearning @xcite holds significant importance in the context of data privacy and regulatory compliance. in today's data-driven landscape, enabling models to effectively forget specific data is crucial for safeguarding user privacy @xcite and meeting legal requirements like the right to be forgotten @xcite . the goal of this task is to make a pretrained model forget data from a specified forget set while retaining its performance on a retain set, to promote research on enabling llms to unlearn certain information or behaviors as needed. a detailed description can be found in the shared task description papers @xcite . we propose synthetic token alternative training (stat), a framework that enables controlled knowledge removal through synthetic data generation and alternating optimization. as shown in figure 1 , the method operates through two complementary mechanisms: (1) creating pseudo tokens to avoid destructive gradient updates, and (2) implementing alternative training to prevent catastrophic knowledge conflicts. first, to circumvent instability caused by gradient ascent on original forget samples @xcite @xcite , stat synthesizes token sequences by randomly sampling from the full vocabulary. this synthetic generation erases specific knowledge associations through token recombination. second, unlike con-ventional approaches that complete full passes through one dataset before switching (e.g., process all forget data then all retain data), stat alternates at sample level-updating parameters through gradient descent on synthetic forget tokens, then immediately applying prediction loss minimization @xcite on actual retain samples. this alternation frequency prevents over-optimization toward either objective while accelerating convergence through coordinated parameter updates. the task evaluation yields several critical observations that substantiate our methodological design: ## background we establish the notation that serves to formalize the process of llm unlearning and we provide a concise overview of the baseline methods. ## system overview we introduce the synthetic token alternative training (stat) for llm unlearning, which addresses the instability and potential catastrophic collapse issues associated with gradient ascent unlearning methods and effectively accelerates the model's convergence speed. ## conclusion this work proposes the synthetic token alternative training (stat) framework for precise llm knowledge unlearning. stat enables controlled knowledge removal through synthetic token generation and alternating gradient descent optimization between forget-set perturbations and retain-set fidelity preservation. empirical evaluations demonstrate stat's superior efficacy over baseline methods in official benchmarks.