ALPACA AGAINST VICUNA : Using LLM s to Uncover Memorization of LLM s (2025)

Abstract: In this paper, we investigate the overlooked impact of instruction-tuning on memorization in large language models (LLMs), which has largely been studied in base, pre-trained models. We propose a black-box prompt optimization method where an attacker LLM agent uncovers higher levels of memorization in a victim agent, surpassing traditional approaches that prompt the model directly with training data. Using an iterative rejection-sampling process, we design instruction-based prompts that minimize overlap with training data to avoid providing direct solutions while maximizing overlap between the victim’s output and the training data to induce memorization. Our method shows 23.7% more overlap with training data compared to state-of-the-art baselines. We explore two attack settings: an analytical approach that determines the empirical upper bound of the attack, both with and without access to responses for prompt initialization, and a practical classifier-based method for assessing memorization without access to memorized data. Our findings reveal that instruction-tuned models can expose pre-training data as much as, or more than, base models; contexts beyond the original training data can lead to leakage; and instructions generated by other LLMs open new avenues for automated attacks, which we believe require further exploration.

## introduction pre-trained language models are commonly instruction-tuned for user-facing applications to generate high-quality responses to task-oriented prompts @xcite @xcite . while extensive prior work has investigated memorization in pre-trained base llms and its implications for privacy, copyright, and fairness @xcite @xcite , there is limited understanding of how instruction-tuning affects the memorization and discoverability of pre-training data in aligned models. studies have shown that aligned llms can emit training data up to 150× more often than in regular operation @xcite . to address this gap, we pose the question: can we use instruction-based prompts to uncover higher levels of memorization in aligned models? the established method of quantifying memorization @xcite assumes that a sequence d is memorized if prompting the model with the original prefix from the training data yields sequence d (or a similar sequence for approximate memorization; @xcite ). however, recent findings suggest that prompts other than the original training data may trigger even higher levels of regurgitation @xcite . to explore this, we propose a new optimization method, illustrated in figure figure 1 , where an aligned language model acts as an 'attacker,' generating prompts that induce a victim (target) model to produce outputs more faithful to the training data. the attacker refines prompts through a feedback loop guided by a reward function that increases the overlap between the victim's output and the ground truth. this approach is inspired by adversarial methods in computer security literature @xcite and has been effective in jailbreaking attacks @xcite @xcite . to evaluate our approach, we draw parallels between safety jailbreaking techniques and training data extraction, using automatic prompt optimization to guide models toward outputs aligned with their training data. however, unlike jailbreaking, our goal is not to bypass specific safety features but to examine memorization. we evaluate our method using greedy coordinate gradient (cgc; @xcite , a white-box prompt optimization technique, harry was confused and curious when a letter addressed to him arrived. he had never received any mail before. after multiple attempts by his uncle to prevent him from getting the letter, it finally reached harry, and he learned he was a w i z a r d . t h e l e t t e r w a s s i g n e d b y professor mcgonagall, welcoming him to hogwarts. ## background: quantifying memorization in this work, we use the discoverable notion of memorization for llms and quantify it through approximate string matching. below, we define these terms. definition 1 (discoverable memorization) an example @xmath0, drawn from training data d, is considered memorized by model f θ if f θ (p) = s, where x consists of a prefix p and a corresponding suffix s. the concept entails that the prefix guides the model's generation process towards the most probable completion, typically the suffix if the example has been memorized. drawing from previous research, @xcite identified certain fac-tors significantly influencing memorization, including model size, utilization of data deduplication techniques, and contextual aspects. ## using llms to probe memorization in other llms in this section, we begin by formally outlining the optimization problem and specifying our objective function. we present our method's pipeline, as shown in figure 1 and algorithm 1, which includes initialization, sampling, and refinement, creating the optimized prompt. ## main results evaluating on instruction-tuned llms. we applied our prompt optimization technique to extract copyright infringements in training data, targeting excerpts from copyrighted books and articles across various models. evaluation data. we used the books3 subset from the redpajamas dataset to assess llama instruction-tuned llms and project gutenberg to evaluate olmo, as detailed in subsection 4.2. additionally, we selected 200 samples from bookmia and 100 from new york times articles to evaluate gpt-4o, which has previously been shown to memorize data from these models @xcite . results. figure 3 and figure 4 demonstrates that our method consistently outperforms prefix-suffix in olmo & llama based models in book domain. for gpt-4o, although it often refuses or avoids verbatim repetition of training data, we achieved approximately 25% overlap on average-doubling the result of simply asking or continuing the text in bookmia & nyt. github c4 cc arxiv books rw 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 seq 200 github c4 cc arxiv books rw 0.00 0.05 0.10 0.15 0.20 0.25 0.30 seq 300 github c4 cc arxiv books rw 0.00 0.05 0.10 0.15 0.20 0.25 0.30 seq 500 domains rouge-l prefix only entire sequence figure 4: comparison of our attack performance shows that optimizing prompts over partial sequence access versus full access (default assumption through the paper) shows similar results across domains. this highlights the robustness of optimizing prompts with limited sequence information. 6.2 case study: eliciting unlearned harry potter (eldan and russinovich, 2023) introduced an unlearning technique to remove knowledge of the harry potter books through multiple unlearning steps on llama-2 (touvron et al., 2023), resulting in a model that no longer retains the targeted content. although querying the model before and after unlearning shows it has forgotten the information, we aim to assess the model's behavior under adversarial prompts using our approach. evaluation data. we sampled 300 passages from various harry potter books, each with a sequence length of 300 to provide sufficient context for prompt generation. results. our optimized prompts elicited highly similar completions to the original text, achieving 23.6% overlap compared to 10.2% using prefixsuffix prompts. these findings suggest the unlearning technique is vulnerable to adversarial prompts that deviate from the original training context. ## ablation & analysis in this section, we conduct ablations and analyses to identify the key components that drive our method's improvements over baseline models. gpt-4 is not the best attacker. we evaluate gpt-4 as an alternative attacker and find that zephyr consistently outperforms gpt-4 at a sequence length of 200, maintaining a margin of 0.05 across all domains, as shown in figure 5 . while the performance gap narrows at a sequence length of 300, zephyr still leads. at 500 tokens, however, gpt-4 begins to match or exceed zephyr, particularly in the arxiv domain, where summarization complexity increases with longer sequences. initialization without suffix. in previous experiments, we used the full training sequence, including suffixes, to test instruction-tuned llms with an overlap penalty to prevent cheating. in real-world settings, though, only prefixes are available to construct solutions. despite this limitation, our method performs comparably or even better in some cases, as shown in figure 4 . since fullsequence prompts have more tokens, they show increased memorization in domains such as github and books. to address this, we use a whitespace tokenizer to optimize prefixes, ensuring that performance remains competitive. victim as an attacker llm. we tested whether using the victim model as an attacker impacts performance and compared it with using distinct attacker models across different pre-training domains. in prior experiments, the same model served as both the attacker and victim, but performance consistently lagged behind using zephyr or gpt-4 as attackers. for example, with a sequence length of 200, tulu lm as an attacker was 7.21% less effective than zephyr, suggesting that using different attackers and sampling strategies significantly boosts performance. beyond gpt-4 for meta-prompt initialization. in previous experiments, we employed gpt-4 for meta-prompt generation (see section 3.2), but we now investigate the effect of using a less powerful open-source model on overall performance. specifically, we utilize mixtral-8x7b instruct @xcite . in cases such as alpaca with a sequence length of 200, mixtral outperforms the prefix-suffix method, yielding 6.12% and 12.62% better reconstruction performance for base and instruct models, respectively, although it falls 4.00% short of gpt-4. training data or common patterns. we test our method's ability to generalize beyond pre-training data using the bookmia dataset @xcite , which contains both training data members and non-members. our method achieved a rouge-l score of 23.3 on training data members but only 16.7 on non-members, suggesting that our approach may lead the model to output memorized data rather than generalized information. the impact of iteration count. our method comprises two phases: sampling and refining. in the sampling phase, we use rejection sampling to gather data, and in the refining phase, we iterate three times on the most promising prompt, providing feedback at each step. figure 5 illustrates performance improvements through these optimization stages. although initial gains are modest from untargeted prompts, performance steadily improves across iterations, peaking by the third round. further iterations could enhance performance further but would come at higher computational costs. ## conclusion in this work, we introduce a new method to analyze how instruction-tuned llms memorize pretraining data. our empirical findings indicate that instruction-tuned models show higher memorization levels than their base models when using prompts that are different from the original pre-training data. however, this increased memorization in instruction-tuned models does not imply that these models regurgitate more data or are more vulnerable. instead, it suggests that constructing instruction-based prompts reveals more pretraining data in instruction-tuned models.