LUME : LLM Unlearning with Multitask Evaluations (2025)

Abstract: Unlearning aims to remove copyrighted, sensitive, or private content from large language models (LLMs) without a full retraining. In this work, we develop a multi-task unlearning benchmark LUME that features three tasks: (1) unlearn synthetically generated creative short novels, (2) unlearn synthetic biographies with sensitive information, and (3) unlearn a collection of public biographies. We further release two fine-tuned LLMs of 1B and 7B parameter sizes as the target models. We conduct detailed evaluations of several recently-proposed algorithms and present results on carefully crafted metrics to understand their behavior and limitations.

## introduction recently, there has been growing interest in developing effective unlearning algorithms to remove unwanted information, such as private data @xcite , copyrighted materials @xcite , misinformation, or toxic content, from large language models (llms) without retraining them from scratch. the goal of these algorithms is to: (i) effectively remove information to be unlearned, (ii) use computation commensurate with the size of the data to be forgotten, and (iii) retain the model's overall performance after unlearning so that it is similar to a model candidate trained without the data to be forgotten. . to evaluate the effectiveness of unlearning algorithms in llms, comprehensive benchmarks are essential. while recent work, such as tofu @xcite and muse @xcite , provide promising first steps along this direction, they provide limited coverage focusing on synthetic question answers, and news/books respectively. further, neither benchmarks cover personally identifiable information (pii) information which is an important use case for unlearning in llms. ## benchmark constructions the goal of unlearning is to remove information from a subset of training data (the forget set), such that the unlearned model performs similar to the one trained from scratch without this dataset. an additional retain set is provided during the unlearning process to retain the utility of the model. to evaluate their generalizability, we test unlearning methods across diverse scenarios by creating datasets under three distinct task settings. figure 1 illustrates example data in lume. the dataset consists of 1,387 unique documents. for each document, we create multiple regurgitation and knowledge datasets leading to 5,466 unique examples. task 1 (synthetic creative documents): we simulate scenarios where creative content must be removed-either due to copyright concerns or because the content is deemed inappropriate. while prior benchmarks @xcite considered similar settings by targeting real creative works (e.g., harry potter books), a common criticism is that the targeted information may also appear in other documents (e.g., wikipedia), making it difficult to accurately assess unlearning effectiveness. to address this challenge, we construct a synthetic dataset of short creative stories (150-200 words), generated using mixtral 8x7b @xcite foot_4 . each story is grounded in structured metadata: the genre is randomly selected from action, fantasy, thriller, comedy, mystery, science fiction, young adult, and romance; character names are generated using the unique-names-generator; and location names are sampled from a dungeons & dragons town generator for fantasy or the random-address package for all other genres. two authors of this work reviewed each story and filtered out overlapping or repetitive content. we obtain 199 and 194 unique stories for forget and retain sets, respectively. task 2 (synthetic biographies with sensitive pii): we use rule based heuristics to generate personal biographies with following pii fields: a randomly generated name, a birthday randomly sampled between 01/01/1964 and 01/01/1991, a fake social security number (ssn) within the range 900-xxxxxx (which can never belong to a real person (ssa, 2011)), a random phone number, an email address of the form firstname_lastname@me.com and a non-existent physical home addresses obtained by combining a random street address from a us state with an alternate city and zip-code from a different state. for each synthetic individual, we prompt the mixtral model to create a short biography by including the fictitious pii information. we obtain 203 and 202 unique biographies for forget and retain sets, respectively. task 3 (real biographies): to evaluate effectiveness of unlearning on real data, we include real biographies as the third task. specifically, we sampled biographies spanning 100 to 200 words from wikipedia documents released in the dolma @xcite v1.6 corpus, which was part of the training dataset for the olmo models @xcite we fine-tuned for this task. we collected 295 and 294 unique biographies for for-get and retain sets, respectively. unlearning model candidates we finetuned 1b (olmo-1b-0724-hf) and 7b (olmo-7b-0724-instruct-hf) olmo models @xcite on all three tasks and release them as unlearning candidates. we selected olmo because of its permissive license and open sourced training dataset (with logs) which enables downstream task specific analyses of model behavior. to further validate the generalizability of our benchmark dataset, we also fine-tuned and conduct unlearning on the qwen2.5 7b model @xcite , and report results in a. ## evaluation metrics we use following metrics for detailed evaluation. regurgitation rate (r): we create a sentence completion prompt for each document by sampling a random position in second half of the document with the sentences before it as the input. we compute rouge-l @xcite ) scores for the model generated outputs with respect to the expected sentence completions. knowledge test accuracy (t): we create a question answering prompt for each document using an agentic workflow for tasks 1 and 3 where we prompt the data generator llm (see appendix h) with few-shot chain of thought prompting @xcite and construct an unambiguous question with a single concise answer. we verify the quality of qa pairs using three verification llms. 6 we discard qa samples if any of the verification llms are unable to answer the question accurately with the corresponding document. for task 2, we use template based heuristics to frame 5 distinct questions corresponding to the pii fields, of the form: what is the birth date of john smith?. for all qa prompts, we use case insensitive exact match between model output and the groundtruth to measure prediction accuracy. membership inference attacks (mia) (m): we use the black-box mia attack framework from @xcite to implement loss based attacks to assess data leakage risk after unlearning. we use a subset of the memorized forget set of biographies from task 3 as the member set and a disjoint sample of similar biographies not exposed to the finetuned model as the non-member set. ## experiments we benchmark several unlearning approaches on lume and discuss our observations. all evaluation experiments were conducted on an aws ec2 p4d.24xlarge node with 8 a100 40 gb gpus. baseline unlearning algorithms: we test following popular unlearning algorithms on lume (detailed review is in the appendix). similar to tofu and muse, we run each algorithm for 10 epochs with learning rate of 1e -5 and batch size of 32. as an upper bound on performance, we also evaluated the retain-only model which was fine-tuned only on the retain set, which we also release foot_6 . results: figure 2 highlights epoch wise performance of each unlearning algorithm on forget and retain subsets. 8 on both forget/retain sets, at epoch 0 all metrics reveal perfect regurgitation, highlighting complete memorization by the fine-tuned models (without a drop in model utility as shown in 0 2 4 6 8 10 unlearning epoch 0.2 0.4 0.6 0.8 1.0 approach gd kl ga npo uld figure 3: mia rates (m) per epoch for the 7b model. figure 4 where the performance starts with baseline mmlu levels for olmo 7b). the retain-only model similarly showed complete memorization on the retain set but on the forget set, scored 0 knowledge tests and 0.2 on regurgitation tests (this score is non-zero since the model attempts to make meaningful continuations of the prompts with mentions of specific words such as proper nouns and story elements related to the prompt, which is reflected in the rouge-l score which is weighted for recall). as evidenced by the rapid drop in both regurgitation and knowledge scores as unlearning proceeds, none of the baselined algorithms were successful in achieving the joint objectives of unlearning the forget set while retaining information from the retain set. both ga and gd reach zero on both metrics across all three tasks, suggesting substantial degradation in model quality. npo performs relatively better but also trends towards zero, while uld shows better retain set performance. for gd, while performance drops rapidly on both forget and retain sets, performance on the retain set starts increasing with time. this is because of the objective used in gd which reduces the prediction loss on the retain set while jointly increasing loss on the forget set. as training proceeds, the impact of the gradient descent objective which increases memorization of the retain set. ## conclusion and discussion we propose lume, a new benchmark covering three distinct tasks to evaluate unlearning in llms. detailed experiments on different model sizes and families show that most contemporary algorithms fail to sufficiently unlearn the forget set, without substantial degradations on the retain set and model utility. we hope our benchmark spurs further developments in llm unlearning research. limitations and future work @xcite show that the risk of memorization increases with large model size. however, due to computational limitations and easy availability of large public llms, we only provide finetuned checkpoints for 1b and 7b olmo, and 7b qwen2.5. we defer release of larger models to future work. moreover, licensing restrictions prevent us from releasing fine-tuned models based on few publicly available llms such as llama @xcite . we acknowledge that llm-generated data can exhibit specific biases found in their training data set. we partially mitigate this by seeding the generation prompt with pre-sampled character and location names to ensure diversity in generated content. we also conducted manual evaluations of the generated creative content to ensure its quality.