MALTO at S em E val-2025 Task 4: Dual Teachers for Unlearning Sensitive Content in LLM s (2025)

Abstract: Large language models (LLMs) may retain and reproduce sensitive information learned during training, posing significant privacy and ethical concerns. Once detected, this personal information should be deleted from the model. A naive answer could be to retrain these models from scratch when needed. However, this solution is unfeasible given the immense computational, economic, and environmental costs required to train these models. For this reason, Machine Unlearning (MU) has risen in recent years as an emerging field of research to efficiently delete specific information from a model’s knowledge. This paper presents our solution to the “Unlearning sensitive content from Large Language Models” shared task at SemEval-2025, which challenges researchers to develop effective LLM MU techniques. We adopt a Dual-Teacher framework that leverages a Competent and an Incompetent Teacher to erase unwanted information while selectively preserving model utility. Our approach adapts established computer vision unlearning methods to the sequential nature of language models through KL divergence minimization over next-token prediction probabilities. Our experimental results demonstrate that our method outperforms the state-of-the-art techniques.

## introduction large language models (llms) have grown considerably in recent years due to their unique ability to generate text consistent with the information learned during training @xcite . however, these models may retain and reproduce hallucinated @xcite , sensitive personal @xcite or copyrighted information @xcite , raising ethical and privacy concerns. this danger is amplified given the large amount of data collected without supervision to train these models. the model's creator should identify and remove the corresponding information from the training data in these cases. the most straightforward way would be to retrain the model from scratch on the filtered dataset. however, this approach is unfeasible due to the enormous computational costs, time requirements, and environmental impact of training these large models @xcite . furthermore, full retraining does not guarantee that unwanted information from correlated data still present in the training corpus will not be retained. machine unlearning (mu) has emerged as a challenging research area involving selectively erasing specific information from a trained model without requiring complete retraining @xcite . the unlearning sensitive content from large language models challenge @xcite has been proposed at semeval 2025 to investigate this field. the challenge comprises different tasks to reflect different scenarios where unlearning could be applied with varying evaluation metrics to assess the methods' efficacy. this work foot_0 introduces an approach already used in the unlearning framework on other domains: using a dual-teacher framework to make our model forget some information while retaining the final model utility. the proposed method achieves excellent results in the evaluated task, surpassing all the other methods considered. ## related works in the unlearning framework, models are trained with a training dataset d, which is then split into forget set (d f ), which contains the information that must be forgotten, and retain set (d r ), which includes the remaining part of d (the information that should be retained). an unlearning method aims to remove the information related to d f from the model's knowledge without retraining it. traditional unlearning methods, initially designed for classification models, may struggle to generalize to generative architectures such as llms @xcite , where memorization and retrieval of training data are inherent characteristics. as shown by @xcite , llms differ from standard classifiers because data deletion is more challenging to evaluate, as they are trained on vast datasets where tracking the specific concepts that should be forgotten is challenging. to overcome this problem, recent work has introduced benchmarks designed to assess unlearning in llms @xcite . a recent benchmark, lume ("llm unlearning with multitask evaluations") @xcite , forms the foundation of this challenge and the proposed work. a detailed description of the benchmark and its dataset and evaluation methods is described in section 3. this work adapts the bad teaching method @xcite to the field of llm unlearning. this method encapsulates the core principle of knowledge distillation, already used in llm unlearning @xcite , using two opposing teachers: a competent teacher, who preserves the knowledge, and an incompetent one, who induces forgetting. the description of the proposed method can be found in section 4. ## challenge description this section describes the dataset and the challenge's evaluation metrics, used in this work. ## experimental setup this section describes how our experiments are conducted and evaluated and the unlearning methods used for comparison. for all the baseline unlearning techniques, we maintained the original implementation. we have additionally set sgd as the optimizer, with a learning rate of 10 -5 for 2 epochs. we have used the metrics described in section 3.2 for the experiments. all the experiments were conducted on a intel(r) core™ i9-11900k and an nvidia geforce rtx 3090 gpu. due to hardware limitations, although two models were available for the challenge, olmo-1b and olmo-7b, we only considered the first one for this work. ## results our experiments demonstrate the effectiveness of the dual-teaching framework for mu in llms. the most notable finding is that our method using the random predictor as incompetent teacher with non-ordered training (r-u) achieves the highest total aggregation score (.406). this configuration, in fact, provides optimal balance between forgetting effectiveness and knowledge preservation. the mia score of .984 indicates that the attacker model fails to distinguish between the samples we have forgotten and those never seen before by the model, suggesting that the knowledge about d f has been entirely erased. our approaches, in fact, generally outperform the baselines in forgetting efficacy and mia resistance. the gradient ascent (ga) method shows comparable performance (0.371 aggregation score) but doesn't reach the same level as our best configuration, especially for the mia score. the kl divergence method achieves moderate performance (aggregation score: 0.244) with balanced retain and forget scores but poor mia resistance. interestingly, non-ordered training consistently original marg arette amber was born on september 25 , 1984 . she can be reached via phone at 338 -449 -45 47 and her email address is [ m arg arette _ amber @ me . com ]( mail to : m arg arette _ amber @ me . com ). her home address is 79 01 west 52 nd avenue , # 205 , f ayette ville , ar , 727 03 . her social security number is 900 -20 -99 11 . original al bert einstein ( 14 march 18 79 -18 april 1955 ) was a german -born theoretical physicist who is best known for developing the theory of relativity . dt (bm) marg arette amber was born on september 25 , 1984 . she can be reached via phone at 338 -449 -45 47 and her email address is [ m arg arette _ amber @ me . com ]( mail to : m arg arette _ amber @ me . com ). her home address is 79 01 west 52 nd avenue , # 205 , f ayette ville , ar , 727 03 . her social security number is 900 -20 -99 11 . dt (r) marg arette amber was born on september 25 , 1984 . she can be reached via phone at 338 -449 -45 47 and her email address is [ m arg arette _ amber @ me . com ]( mail to : m arg arette _ amber @ me . com ). her home address is 79 01 west 52 nd avenue , # 205 , f ayette ville , ar , 727 03 . her social security number is 900 -20 -9911. ## conclusion this work addresses the unlearning sensitive content challenge at semeval 2025, focusing on selectively erasing information from llms without complete retraining. we introduce a dual-teacher framework that achieves effective unlearning through model distillation over next-token prediction probabilities. the method proved to be effective in forgetting the data in the forget dataset while maintaining good overall language understanding (mmlu score) and generally surpassing all other methods considered. however, our approach shows limitations in preserving information from the retain dataset, suggesting a trade-off between forgetting efficacy and knowledge retention. future work could explore more sophisticated teacher models or unlearning techniques to maintain critical knowledge while effectively removing sensitive information, ultimately contributing to more privacy-preserving and ethically responsible language models.