Unlocking Memorization in Large Language Models with Dynamic Soft Prompting (2024)

Abstract: Pretrained large language models (LLMs) have excelled in a variety of natural language processing (NLP) tasks, including summarization, question answering, and translation. However, LLMs pose significant security risks due to their tendency to memorize training data, leading to potential privacy breaches and copyright infringement. Therefore, accurate measurement of the memorization is essential to evaluate and mitigate these potential risks. However, previous attempts to characterize memorization are constrained by either using prefixes only or by prepending a constant soft prompt to the prefixes, which cannot react to changes in input. To address this challenge, we propose a novel method for estimating LLM memorization using dynamic, prefix-dependent soft prompts. Our approach involves training a transformer-based generator to produce soft prompts that adapt to changes in input, thereby enabling more accurate extraction of memorized data. Our method not only addresses the limitations of previous methods but also demonstrates superior performance in diverse experimental settings compared to state-of-the-art techniques. In particular, our method can achieve the maximum relative improvement of 135.3% and 39.8% over the vanilla baseline on average in terms of *discoverable memorization rate* for the text generation task and code generation task, respectively. Our code is available at https://github.com/wangger/llm-memorization-dsp.

## introduction pretrained large language models (llms) have achieved remarkable success across a wide range of downstream natural language processing (nlp) tasks such as summarization @xcite @xcite , question answering (pan et al., 2023; shao et al., 2023; louis * co-corresponding authors. @xcite @xcite and translation @xcite @xcite @xcite , etc. the popularity of llms requires people to pay attention to the unique challenges they bring to security. one of the significant security issues is that llms can memorize a considerable portion of their training data even though they tend to not overfit to their training dataset due to the small number of training epochs @xcite . moreover, the memorized data can be extracted by carefully designed input from attackers or even unintentional input from ordinary users, which can cause privacy and copyright issues with the sensitive training data @xcite @xcite @xcite . for example, the confidential codes from samsung can be exposed to other users after they were shared with openai due to the memorization of llms @xcite . the huge security risks and the potential uses of memorization make it important to measure the memorization of the target llm. with an accurate method to quantify the intrinsic memorization of llms, model developers can have a better understanding of the model's vulnerability posed by its memorization and take actions such as machine unlearning @xcite @xcite to mitigate the memorization before they release their llms to the public. moreover, the method to extract memorized data can also be combined with the target llm and leveraged by the users to detect whether their self-built dataset has data leakage issues when it is used to evaluate the target llm. to measure the intrinsic memorization of the target llm, @xcite first proposed a metric called discoverable memorization rate to serve as the estimation. as shown in figure 1 (a) , the given data are split into prefix tokens and suffix tokens and the prefix tokens are fed into the target llm. the data are defined as discoverably memorized when the output tokens can match the suffix tokens verbatim. @xcite proposed that more memorized data can be extracted via learning a soft prompt and prepending it to the prefix tokens for generation, which is shown in figure 1 (b). however, the memorization of the target llm is still underestimated even with the soft prompt since it is constant and invariant to prefix tokens, which may not help or even hinder extracting data when changing the prefix. in this paper, we propose a new method to estimate the memorization of llms. compared with constant soft prompts @xcite , our method can generate prefix-dependent soft prompts and react to the changes in inputs. more specifically, a transformer-based generator is trained for the generation of the dynamic soft prompt. as shown in figure 1 (c), it takes the outputs from the naive mapping m(•) of prefix tokens as its input and emits the corresponding soft prompt prepended to the prefix tokens. this method can customize prompts given the inputs and thus extract more memorized data from the target llm, which can reflect its intrinsic memorization more accurately. our contributions can be summarized as follows. experimental results show that our method can outperform all the baselines consistently in all the evaluated settings. the maximum relative improvement of 135.3% and 39.8% is achieved over the vanilla baseline on average for the text generation and code generation tasks, respectively. ## conclusion we propose a novel method to unlock memorization in large language models (llms) which was underestimated by previous methods. more specifically, a transformer-based generator is developed to customize the dynamic, prefix-dependent soft prompts to measure the llm memorization. it can have a more precise detection of memorized data, capturing the data omitted by the previous methods only relying on the prefixes or the concatenation of a constant soft prompt and prefixes. extensive experiments are conducted to show that our method can outperform the state-of-the-art techniques by a large margin under diverse settings, including text generation and code generation tasks.