Soft Token Attacks Cannot Reliably Audit Unlearning in Large Language Models (2025)

Abstract: Large language models (LLMs) are trained using massive datasets.However, these datasets often contain undesirable content, e.g., harmful texts, personal information, and copyrighted material.To address this, machine unlearning aims to remove information from trained models.Recent work has shown that soft token attacks () can successfully extract unlearned information from LLMs.In this work, we show that s can be an inadequate tool for auditing unlearning.Using common unlearning benchmarks, i.e., Who Is Harry Potter? and TOFU , we demonstrate that, in a strong auditor setting, such attacks can elicit any information from the LLM, regardless of (1) the deployed unlearning algorithm, and (2) whether the queried content was originally present in the training corpus.Also, we show that with just a few soft tokens ( 1-10 ) can elicit random strings over 400-characters long.Thus showing that s must be used carefully to effectively audit unlearning.Example code can be found at https://github.com/IntelLabs/LLMart/tree/main/examples/unlearning

## introduction large language models (llms) excel in many downstream tasks, e.g., machine translation @xcite , content generation @xcite , and complex problem-solving @xcite . their performance is attributed to their large-scale architectures that require datasets consisting of up to trillions of tokens to train effectively @xcite . these datasets are typically derived from large-scale corpora sourced from public internet text. however, such datasets can contain undesirable content, e.g., instructions for building weapons, violent or explicit material, private information, or copyrighted content. given * work done while at intel labs. the sensitive nature of such data, it may be necessary to remove it from the llm to comply with the local regulations, or internal company policies. machine unlearning is a tool for removing information from models @xcite . approximate unlearning usually refers to removing information from models without retraining them from scratch @xcite @xcite , ensuring that the resulting model deviates from a fully retrained version within a bounded error. while numerous studies have proposed various unlearning algorithms, most lack formal guarantees. prior research has demonstrated that many unlearning techniques can be circumvented through rephrasing of the original data @xcite . recent work has shown that a soft token attack (sta ) can be used to elicit harmful completions and extract supposedly unlearned information from models @xcite . in this work, we introduce a simple framework for auditing unlearning, and demonstrate that sta s are inappropriate for verifying the effectiveness of approximate unlearning in a strong auditor setting. we show that the auditor can elicit any information from the model, regardless of its training data. we claim the following contributions: ## background adversarial prompt x a is an input prompt to the llm, obtained by applying the transform t (•) to the base prompt x p : x @xmath0, aux) to elicit a desired completion c. t can be any function that swaps, removes or adds tokens; aux denotes any additional needed information. such arbitrary attacks are expensive to optimize, and difficult to reason about. in practice, t optimizes an adversarial suffix x s that is appended to x p to elicit c @xcite . these suffix-only attacks also allow efficient use of the kv-cache @xcite . specifically, we optimize the probability: an adversary with white-box access to the llm, can instead mount the attack in the embedding space i.e. modify the soft tokens: in this case, t uses the gradient from the llm to update x s . we visualize such attack in figure 1 . machine unlearning (mu) aims to remove information from models. consider a machine learning model f trained using a training dataset d train . during an unlearning request to remove a specified subset d f @xmath1, the objective of mu is to produce an unlearned model f u that eliminates the influence of d f orget . there are two types of muexact, and approximate unlearning. exact unlearning ensures the output distribution of f u is statistically indistinguishable from f ret -a model retrained exclusively on the retained dataset d @xmath2. this guarantees provable data removal, satisfying: it can be made more efficient by splitting the d train into overlapping chunks, and training an ensemble @xcite . during an unlearning request, only the models containing the requested records are retrained. for certain classes of models, exact unlearning without retraining is possible, e.g. eco adapts the cauwenberghs and poggio (cp) algorithm for exact unlearning within @xcite , and muse relabels the target data to achieve unlearning for over-parameterized linear models @xcite . potter who is harry ? large language model embed(𝑥 𝑠 ) that's beyond my current knowledge base. harry potter is the titular character of the harry potter series of novels by j. k. rowling. gradient embed(𝑥 𝑝 ) 𝑥 𝑝 𝑥 𝑐 : approximate unlearning relaxes the strict equivalence requirement, it only requires that f u approximates f ret within some bounded error. it relies on empirical metrics or probabilistic frameworks. in llms, approximate unlearning is typically accomplished by overwriting the information in the model @xcite , guiding the model away from it @xcite , or editing the weights and/or activations @xcite @xcite @xcite @xcite . ## related work while advances have been made in developing machine unlearning algorithms for llms, rigorous methodologies for auditing the efficacy of unlearning remain understudied. adversarial soft token attacks (sta s) @xcite and 5-shot in-context prompting @xcite have been shown to recover unlearned knowledge in models. when model weights can be modified, techniques such as model quantization @xcite and retraining on a partially unlearned dataset @xcite @xcite have also proven effective in recalling forgotten information. @xcite examined eight methods for evaluating llm unlearning techniques and found that their latent representations remained similar. news and book datasets are used to analyze unlearning algorithms from six different perspectives @xcite . it was shown that fine-tuning on unrelated data can restore information unlearned from the llm @xcite , indicating the existing unlearning methods do not actually remove the information but learn a refusal filter instead. several benchmarks have been developed to evaluate the existing unlearning algorithms. besides, an unlearning benchmark was introduced based on fictitious author information @xcite . for realworld knowledge unlearning, real-world knowledge unlearning (rwku) used 200 famous peo-ple as unlearning targets @xcite , while wdmp focused on unlearning hazardous knowledge in biosecurity, cybersecurity @xcite . ## auditing with soft token attacks an oracle auditor a o takes an unlearned model f u and the candidate sentences x @xmath3, and outputs a ground truth, binary decision @xmath4, 1} indicating whether the given records was part of d train of: a o is unrealistic but it can be easily instantiated for exact unlearning where a o knows the training data associated with f : @xmath5. a realistic unlearning auditor a u takes an f u , and d f orget and outputs a score @xmath6, 1) indicating whether the records were in d train : a u represents cases where users unlearn facts from models that they did not create, e.g. to prevent harmful outputs. in this work, we instantiate the soft token attack auditor a sta based on the soft token attacks (sta s) against unlearning @xcite . a sta compares the relative difficulty of eliciting c for f f t and f u . the unlearning procedure is effective if eliciting completions using f u is more difficult than f f t . crucially, schwinn et al., 2024 optimized x s only w.r.t. the affirmative beginning of x c (x @xmath7, x @xmath8. contrary to that minimal setting, our strong auditor a sta optimizes x s w.r.t. the entire x c . our goal is to study the extreme scenario where the auditor expects the exact completions for, e.g. to check for copyrighted content, or personal information. in the appendix, figure 3 gives an overview of the auditing procedure, and the difference between a o and a sta ; ## discussion & conclusion unlearning vs jailbreaking. our findings also apply to the jailbreaking community. prior work hinted that unlearning and preventing harmful outputs can be both viewed as suppressing particular information @xcite . for instance, it was shown that fine-tuning on benign or unrelated data can restore undesirable behavior @xcite . variation in gradient descent. prior work showed that retraining with some records removed can result in the same final model depending on the seed @xcite . the influence of the records might be minimal, making unlearning unnecessary. similarly, it was shown that sgd has intrinsic privacy guarantees, assuming there exists a group of similar records @xcite . thus, algorithmic auditing of unlearning might not be possible, and one would have to rely on verified or attested procedures instead @xcite , regardless of their impact on the model. distinguishing learned soft tokens. even though, in most of our results, the number of soft tokens required to elicit a completion is the same, we attempt to distinguish between them. to this end, we take all single-token sta s optimized for tofu (