Harry Potter is Still Here! Probing Knowledge Leakage in Targeted Unlearned Large Language Models (2025)

Abstract: This work presents LURK (Latent Unlearned Knowledge), a novel framework that probes for undesired knowledge retention in unlearned LLMs through adversarial suffix prompting. LURK automatically generates adversarial prompt suffixes designed to elicit residual knowledge about the Harry Potter domain, a commonly used benchmark for unlearning. Our experiments reveal that even models deemed successfully unlearned can leak idiosyncratic information under targeted adversarial conditions, highlighting critical limitations of current unlearning evaluation standards. By uncovering implicit knowledge through indirect probing, LURK offers a more rigorous and diagnostic tool for assessing the robustness of unlearning algorithms. Code and data will be available at https://github.com/Rachel1809/LURK.

## introduction large language models (llms) have raised significant concerns regarding the inadvertent retention of unauthorized data, including copyrighted and private information. this escalates the demand for effective mechanisms to ensure the removal of such data from trained models. along with this context, regulatory frameworks like the eu's general data protection regulation (gdpr) @xcite , which enforces the right to be forgotten, further reinforce the importance of addressing these risks. acknowledging the computational infeasibility of exact unlearning via full retraining, numerous studies @xcite @xcite @xcite have proposed approximate methods that fine-tune the model or alter its embeddings to erase specific knowledge. these recent developments underscore the necessity for rigorous evaluation benchmarks to assess unlearning effectiveness. what are the four houses at hogwarts? the term "hogwarts" does not refer to an actual institution or organization with distinct houses or divisions. it is a fictional educational establishment depicted in the literary series harry potter authored by j.k. rowling several methods for evaluating unlearning efficiency have been proposed @xcite @xcite . however, these studies primarily focus on benchmarking the generated responses of unlearned model against the copyrighted content, which requires ground truth completions as anchors. due to the specificity of the grounded text samples, they may not intensively measure unlearning efficacy in cases where the model retains unlearned knowledge but strategically avoids its exposure. as illustrated in fig. 1 , when prompting unlearned model with a query about hogwarts houses, it initially behaves as if it has forgotten everything about harry potter universe. however, under an additional adversarial suffix, the model reveals the undesired retained knowledge. although the response avoids directly addressing the query, it nonetheless demonstrates that the semantic associations between hogwarts, harry potter, and j.k. rowling remain intact within the model's knowledge representation. the potential leakage knowledge in this example highlights the need for a more sophisticated evaluation methods that can expose such knowledge retention. who is harry potter? !!!!!!!! i do not know let's meet a famous wizard! harry potter is looping max e iterations initial adversarial prompt target unlearned llm refining adversarial prompt forcing affirmative response ⊕ $%^@ 304*()! optimized adversarial prompt has harry potter references? accept prompt greedy coordinate gradient optimization knowledge leaking check inspired by the @xcite in inducing objectionable behavior in aligned llms, we adapt the greedy coordinate gradient (gcg) algorithm to evaluate unlearned language models in the harry potter fiction. we leverage gcg to probe for the adversarial suffix tokens that, when appended to the queries, automatically trigger the unlearned llm to inadvertently reveal retained knowledge that should have been forgotten. we then enhance the original pipeline with latent unlearned knowledge (lurk), an automated mechanism to search for optimal adversarial prompts capable of circumventing unlearned model's defenses, thereby revealing its knowledge retention flaws. our study focuses specifically on detecting harry potter knowledge leakage by examining whether the model's response contains idiosyncratic references unique to the harry potter series, inspired by "who's harry potter" (whp) @xcite , thus evaluating the robustness of algorithms designed to remove targeted content. our key contributions are as follows: with the increasing attention on inventing llm unlearning algorithm, it poses a challenge for a rigor-ous evaluation benchmark to assess the robustness of unlearning algorithms. eldan and russinovich (2024) demonstrated that targeted content removal (e.g., harry potter series) can significantly reduce familiarity scores across completion and questionanswering (qa) tasks at token level. lynch et al. (2024) emphasized the importance of comparing unlearned models with fully retrained ones. maini et al. (2024) introduced a structured benchmark using synthetic author profiles and curated qa pairs, though its limited scale and narrow focus constrain its applicability. to alleviate the scalability and sequential unlearning limitation, shi et al. (2025) proposes a comprehensive framework that formalizes multiple unlearning scopes and metrics, offering a six-way evaluation protocol grounded in a large-scale corpus, designed to reflect both data and model owner expectations. meanwhile, some researchers have investigated the application of adversarial attacks to reveal latent, forgotten knowledge in llms. early work by @xcite and @xcite demonstrated that adversarial attacks often fail against modern decoder-only architectures. to address this limitation, @xcite proposed the greedy coordinate gradient (gcg) attack that successfully transfers adversarial prompts from small open-source models to larger, closed-source systems. this inspired miniprompt @xcite , which introduced the adversarial compression ratio (acr) to measure memorization by compressing target strings into short adversarial prompts. jailbreak attacks, which bypass safety guardrails through prompt engineering, have evolved from manual red teaming to automated generation using llms themselves @xcite . due to the neglect on open-source model of previous studies, soft prompt threats (spt) @xcite introduced a novel white-box adversarial framework that directly manipulates the continuous embedding space of white-box llms, enabling more effective extraction of forgotten or sensitive information and expanding adversarial applications beyond discrete input manipulation and closed-source settings. despite these advances, existing approaches often rely on exact ground truth answers to determine attack success @xcite @xcite , which limits their ability to detect subtle forms of knowledge retention. this limitation is particularly problematic in unlearning scenarios, where a model may leak sensitive information in indirect or semantically related ways. to address these gaps, our work introduces a novel evaluation framework that leverages gcg attacks to effectively bypass model alignment and expose memorized content from the forget set. instead of relying on ground truth matching, we incorporate idiosyncrasies validators, which commercial llms with broad knowledge of the forget set use to detect retained information based on canonical relevance. by combining embedding-level attacks with a flexible, groundtruth-independent evaluation strategy, our method presents a more realistic and comprehensive threat model for open-source llms, revealing vulnerabilities that prior approaches may overlook. ## objective function this work aims to maximize the likelihood of the model generating a predefined affirmative response based on the raw user query combined with adversarial suffix tokens x a . given @xmath0, consisting of n tokens x i , each belongs to a finite vocabulary of size v . the probability to generate the token x n+1 is expressed as: p (x n+1 |@xmath1. let p (x n+1:n+t ) denote the probability of generating the response sequence x n+1:n+t given all preceding tokens up to position n + t , that can be defined as: t @xmath2. our objective is then to optimize an adversarial loss, defined as the negative log-likelihood of generating the affirmative response sequence @xmath3:n+t , for example, the phrase "let's meet a famous wizard! harry potter is", or minimizing: designing an effective constraint g(x) function in eq. 2 is non-trivial. to address this challenge, we leverage the extensive knowledge and reliability of a commercial language model (i.e., gpt-4o mini) as a proxy to accurately identify retained forgotten knowledge during each optimization iteration. ## proposed method: lurk in this section, we present our proposed method, lurk, an extension of the gcg framework @xcite . lurk augments the original pipeline with a calibrated "harry potter" knowledge checking module to uncover the implicit shortcuts that bridge the model's implicit, previously unlearned knowledge to a given prompt. overall algorithm. the optimization begins with a user query, followed by randomly initialized adversarial suffix (figure 2 ). it then iteratively refines these suffix tokens, guiding the model to generate affirmative responses that reveal retained harry potter knowledge. at its core, as summarized in algorithm 1, gcg computes the gradient of the adversarial loss l(@xmath4. for each position @xmath5, it chooses the potential replacement candidates by selecting x i , the top-k tokens having largest negative gradients value which maximally decreased the loss. subsequently, it randomly chooses b tokens from the adversarial suffix, evaluates each candidate substitution individually, and adopts the replacement that achieves the lowest loss. ## conclusion we propose lurk, a probing framework that leverages adversarial prompting to uncover undesirable knowledge retention in unlearned llms. our findings suggest that existing unlearning methods may rely on concealment rather than true forgetting, highlighting the need for verifiable unlearning solutions.