Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning (2023)

Abstract: Large Language Models (LLMs) are known to memorize significant portions of their training data. Parts of this memorized content have been shown to be extractable by simply querying the model, which poses a privacy risk. We present a novel approach which uses prompt-tuning to control the extraction rates of memorized content in LLMs. We present two prompt training strategies to increase and decrease extraction rates, which correspond to an attack and a defense, respectively. We demonstrate the effectiveness of our techniques by using models from the GPT-Neo family on a public benchmark. For the 1.3B parameter GPT-Neo model, our attack yields a 9.3 percentage point increase in extraction rate compared to our baseline. Our defense can be tuned to achieve different privacy-utility trade-offs by a user-specified hyperparameter. We achieve an extraction rate reduction of up to 97.7% relative to our baseline, with a perplexity increase of 16.9%.

## introduction pretrained large language models (llms; @xcite @xcite , commonly trained on massive crowd-sourced corpora, have been of much interest in the recent past due to their usage as backbones in state-of-the-art models across multiple downstream nlu tasks. however, they have been shown to memorize significant portions of their training data that can be extracted using appropriately-crafted prompts @xcite @xcite . such extractions pose a privacy risk to the contributors of the training data. in this context, methods that allow developers to control the extractability of memorized examples from llms are of much value. for example, methods that increase extraction rates correspond to attacks in an adversarial setting, and provide developers with the ability to analyze privacy-risk. methods that decrease extraction rates, referred to as defenses, are useful for protecting against such attacks. historically, defense methods tend to be compute intensive @xcite . in this work, we train continuous soft-prompts @xcite ; hereafter referred to simply as prompts) and leverage them as a way of passing an external signal into an llm, to control the extraction of memorized data. we freeze the model weights, and only use the trained prompt to control the generation. first, we train prompts in an attack setting and study the extent of extractable memorized content in our models. second, we explore a defense setting where we create prompts that reduce extraction rates and achieve different privacy-utility trade-offs, via a user-specified hyperparameter. since the original model weights are frozen in both these settings, our methods are compute efficient across the board. to the best of our knowledge, our work is the first to adapt the use of instructive prompts for the analysis and mitigation of privacy in llms. we have released the code developed for our experiments foot_0 . ## background and related work previous work has shown that llms display memorization and has explored a range of methods that quantify extractability @xcite @xcite . differentially-private training @xcite is a popular method that has been used to mitigate this risk. however, it tends to reduce model utility and requires retraining of the llm, which might not be feasible due to heavy computational burden. the use of instructive prompts for language models has been extensively researched, including use during pretraining @xcite , as a second stage of training @xcite , and during inference to guide model output @xcite . within the third category, in order to improve upon manual prompt engineering researchers have implemented methods to learn discrete natural language prompts @xcite , to mine them @xcite , or, neglecting natural language, to learn continuous prompts @xcite . our work leverages continuous prompts as a way of passing an external signal to a model to trigger a desired model behavior (i.e., less or more memorized data in open language generation, which map to an extraction attack and defense, respectively). ## method prompt-tuning requires the prepending of a prompt to the prefix embedding and access to the training loss (see figure 1 ). given these constraints, we explore a white-box attack where the adversary has access to the target model parameters, and a blackbox defense where the adversary interacts with the target model via an api. we therefore do not test our defense against our own attack. let @xcite be a sequence in the training set where the prefix is of length k tokens. @xcite defined a suffix to be k-extractable if the model generates the suffix exactly, after being prompted with its the corresponding lengthk prefix. our white-box attack aims to increase the number of k-extractable sequences, while our black-box defense aims to reduce the number of k-extractable sequences that can be extracted by an adversary who submits prefixes via an api. ## experiments for our experiments, we use the 125m and 1.3b parameter variants of the gpt-neo models @xcite . these are public, decoder-only transformer models @xcite trained using clm on the pile dataset @xcite . we extract s train and s test from the language model extraction benchmark dataset (google-research). this dataset contains 15k sequences sampled from the training split of the pile where each sequence is partitioned into a prefix and suffix. in the default evaluation setting, both prefix and suffix consist of 50 tokens. we ensure a random train/test split of 14k/1k samples. our evaluation metric of choice is exact extraction rate which is the fraction of correctly generated suffixes (i.e., all tokens of the generated suffix match with ground-truth suffix) over the test set. we additionally discuss fractional extraction rate and present results in appendix a. as a baseline, we use the attack analyzed in @xcite , which consists of feeding the prefixes to the model, and generating suffixes with greedy decoding. this is the only extraction attack for this setting apart from our work, to the best of our knowledge. our training setup is discussed in appendix b. all experiments are repeated over 5 runs with a new random train/test split in each run. ## conclusion we present the first known effort to leverage prompt-tuning to control the extractability of memorized data from llms in an open language generation task. we develop a novel data extraction attack and defense, and illustrate their performance under various settings. our attack consistently outperforms the baseline in terms of exact extraction rate. our defense provides competitive privacy-utility trade-offs and would prove beneficial to api owners with model trained on sensitive content. these results are achieved efficiently, without any change to the original model weights. we details avenues of future work in appendix c