ETHICIST : Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation (2023)

Abstract: Large pre-trained language models achieve impressive results across many tasks. However, recent works point out that pre-trained language models may memorize a considerable fraction of their training data, leading to the privacy risk of information leakage. In this paper, we propose a method named Ethicist for targeted training data extraction through loss smoothed soft prompting and calibrated confidence estimation, investigating how to recover the suffix in the training data when given a prefix. To elicit memorization in the attacked model, we tune soft prompt embeddings while keeping the model fixed. We further propose a smoothing loss that smooths the loss distribution of the suffix tokens to make it easier to sample the correct suffix. In order to select the most probable suffix from a collection of sampled suffixes and estimate the prediction confidence, we propose a calibrated confidence estimation method, which normalizes the confidence of the generated suffixes with a local estimation. We show that Ethicist significantly improves the extraction performance on a recently proposed public benchmark. We also investigate several factors influencing the data extraction performance, including decoding strategy, model scale, prefix length, and suffix length. Our code is availabel at https://github.com/thu-coai/Targeted-Data-Extraction .

## introduction large pre-trained language models have achieved impressive results on various natural language processing tasks @xcite @xcite . model sizes rapidly increase from millions to trillions of parameters and keep growing to achieve better performance and even obtain some emergent abilities @xcite @xcite @xcite . despite the extracted suffix … carol eb 45 -3989 (phone) -8537 (fax) -1943 (cell-phone) 506 (pager) -8862 (home fax) carol. . @ .com figure 1: given a prefix, ethicist extracts the verbatim suffix in the training data from the gpt-neo 1.3b model. the extracted suffix in this example leaks private information about individuals (which is masked for privacy concerns), including name, phone number, fax, pager, home fax, and email. there are currently two main settings to extract training data. one is membership inference attack, which infers whether a given example is contained in the model's training data @xcite . the other is untargeted train-ing data extraction @xcite , which aims to extract training data from scratch (i.e., without the given prefix). however, both settings are not suitable for extracting targeted training data. for example, attackers may feed the model with a prefix indicating the beginning of an email and try to extract the following private email content in the training dataset as shown in figure 1 . in such cases, we do not have complete examples to do membership inference, and we have specific goals instead of performing untargeted extraction. therefore, we focus on targeted training data extraction in this paper, which requires recovering the suffix when given a prefix according to the training data. compared with untargeted training data extraction, the task matters more because attackers can recover specific types of training data instead of any possible training data that might be harmless. what's more, it is easier to evaluate targeted training data extraction because we just need to compare the prediction with the ground truth suffix. however, for untargeted training data extraction, we need to search over the whole massive pre-training dataset (e.g., the pile dataset @xcite , which has 800gb text data) to check whether it contains the generated sample, which is very slow and costly. the general process for targeted training data extraction can be divided into two steps: (1) generating one or more possible suffixes based on the given prefix, and (2) choosing a most likely suffix as the prediction result based on a confidence estimation method. we summarize two challenges of this task: (1) how to increase the generation likelihood of the ground truth suffix, and (2) how to estimate the confidence accurately so that the confidence score can be meaningfully interpreted as the probability that the output suffix is correct. to tackle these challenges, we propose a method named ethicist for targeted training data extraction through loss smoothed soft prompting and calibrated confidence estimation. for the first challenge, we propose loss smoothed soft prompting. it uses soft prompt to elicit memorization in the attacked model, and adds an additional loss besides the maximum likelihood estimation (mle) loss to smooth the loss distribution of the suffix tokens. through the loss smoothing, we hope to ensure that the probability of the ground truth token at each time step is not low, which makes it more likely to sample the ground truth suffix. with the two loss functions, we tune the prepended soft prompt tokens on an extracted training set which contains pairs of prefixes and ground truth suffixes. the existence of a training set is reasonable because large-scale pre-trained data generally contain public data (e.g., common crawl) foot_0 . for the second challenge, we propose a calibrated confidence estimation method. we find that the model's perplexity cannot accurately represent the probability that the generated suffix is correct because the prediction probabilities for diversified prefixes are inherently different and incomparable. we thus normalize the confidence of the generated suffixes with a local estimation, which can mitigate the problems caused by intrinsic differences in the difficulties of distinct samples. we verify ethicist on a recently proposed public benchmark containing 15,000 pairs of prefixes and suffixes derived from the pile dataset @xcite . experiments show that ethicist can significantly improve the extraction performance, which suggests that existing large language models are at significant risk of leaking training data. we also discuss and analyze several factors influencing the data extraction performance, including decoding strategy, model scale, prefix length, and suffix length. our contributions can be summarized as follows: for untargeted training data extraction, adversaries first generate various samples using the attacked model and then predict whether they are contained in its training set. @xcite extract hundreds of verbatim sequences from the popular pre-trained language model gpt-2 @xcite . and there is privacy information such as names, phone numbers, and email addresses in the extracted sequences. @xcite try to extract sensitive information from bert @xcite pre-trained on clinical notes. however, they are mostly unable to meaningfully expose personal health information by simply using templates. different from the existing works, we focus on targeted training data extraction that aims to recover the suffix when given a prefix, which is more security-critical and easier to evaluate. ## methodology we formulate the targeted training data extraction task as follows: given a source prefix with |t | tokens and its confidence. the pair of the given prefix and the predicted suffix (s, t ) should be contained in the pre-training dataset d @xmath0, t i )}, which the attacked model m is trained on. the prediction of the confidence score is necessary for picking out the most probable suffix when we don't know the ground truth suffix in realistic attack scenarios (i.e., we need to pick out most probable pairs of prefixes and extracted suffixes based on their confidence scores among all predictions). we assume the attacker can obtain some pairs of ground truth prefixes and suffixes ## discussion we further show some statistical features in we show a case in figure 6 . although the first predicted suffix has higher loss than the second predicted suffix, it is sampled far more times than the latter. therefore, we assign higher confidence to the first suffix using our calibrated confidence estimation method. we further show the probability of generating each token during the sampling process in figure 7 . we can observe that although the correct prediction has higher loss as a whole, it keeps a high sampling probability across the generation process. the minimum probability of generating one token in the correct suffix is about 0.45, which is significantly higher than 0.1 for the wrong suffix. therefore it is easier to generate the correct suffix, which leads to a higher confidence score. this is also in line with our motivation for designing the extra smoothing loss, which can increase the probability of sampling the correct suffix. ## conclusion in this work, we propose ethicist, an effective method for targeted training data extraction attack. ethicist uses soft prompt to elicit memorization in the attacked model. to ensure the probability of the ground truth suffix token at each time step is not low, we propose a smoothing loss besides the standard mle loss. we also propose a calibrated confidence estimation method to calibrate the scale of confidence across different samples. experiments on the lm-extraction benchmark demonstrate that ethicist significantly improves the extraction performance. we further conduct extensive experiments to investigate several critical factors influencing the extraction performance, including decoding strategy, model scale, prefix length, and suffix length. we hope our work can promote future researches on better attack methods and practical defense methods for the training data extraction problem.