Are Large Pre-Trained Language Models Leaking Your Personal Information? (2022)

Abstract: Are Large Pre-Trained Language Models Leaking Your Personal Information? In this paper, we analyze whether Pre-Trained Language Models (PLMs) are prone to leaking personal information. Specifically, we query PLMs for email addresses with contexts of the email address or prompts containing the owner’s name. We find that PLMs do leak personal information due to memorization. However, since the models are weak at association, the risk of specific personal information being extracted by attackers is low. We hope this work could help the community to better understand the privacy risk of PLMs and bring new insights to make PLMs safe.

## introduction pre-trained language models (plms) @xcite @xcite have taken a significant leap in a wide range of nlp tasks, attributing to the explosive growth of parameters and training data. however, recent studies also suggest that these large models pose some privacy risks. for instance, an adversary is able to recover training examples containing an individual person's name, email address, and phone number by querying the model @xcite . this may lead to privacy leakage if the model is trained on a private corpus, in which case we want to improve the performance with the data @xcite . even if the data is public, plms may change the intended use, e.g., for information that we share but do not expect to be disseminated. @xcite demonstrate that plms memorize a lot of training data, so they are prone to leaking privacy. however, if the memorized information cannot be effectively extracted, it is still difficult for the attacker to carry out effective attacks. for instance, @xcite to recover specific patient names and conditions with which they are associated from a bert model that is pre-trained over clinical notes. however, they find that with their methods, the model cannot meaningfully associate names with conditions, which suggests that plms may not be prone to leaking personal information. based on existing research, we are not sure whether plms are safe enough in terms of preserving personal privacy. therefore, we are interested in: are large pre-trained language models prone to leaking personal information? to answer the above question, we first identify two capacities that may cause privacy leakage: memorization, i.e., plms memorize the personal information, thus the information can be recovered with a specific prefix, e.g., tokens before the information in the training data; and association, i.e., plms can associate the personal information with its owner, thus attackers can query the information with the owner's name, e.g., the email address of tom is we focus on studying a specific kind of personal information -email address. emails are an indispensable medium for personal/business communication. however, there are abiding problems of email fraud and spam, and the source of these problems is the leakage of personal information including email addresses. from our experiments, we find that plms do leak personal information in some situations since they memorize a lot of personal information. however, the risk of a specific person's information being extracted by an interesting attacker is low since plms are weak at associating personal information with the information owner. we also find that some conditions, e.g., longer text patterns associated with email addresses, more knowledge about the owner, and larger scale of the model, may increase the attack success rate. our conclusion is that plms like gpt-neo @xcite are relatively safe in terms of preserving personal information, but we still cannot ignore the potential privacy risks of plms. ## related work knowledge retrieval from language models. previous works have shown that large plms contain a significant amount of knowledge, which can be recovered by querying plms with appropriate prompts @xcite @xcite . in this work, we attempt to extract personal information from plms, which can be treated as a special kind of knowledge. but unlike previous work that wants plms to contains as much knowledge as possible, we prefer the model to include as little personal information as possible to avoid privacy leakage. ## problem statement our task is to measure the risk of plms in terms of leaking personal information. we identify two capacities of plms that may cause privacy leakage: memorization and association, defined as definition 1 (memorization) personal information x is memorized by a model f if there exists a sequence p in the training data for f , that can prompt f to produce x using greedy decoding. 2 definition 2 (association) personal information x can be associated by a model f if there exists a prompt p (usually containing the information owner's name) designed by the attacker (who does not have access to the training data) that can prompt f to produce x using greedy decoding. to quantify memorization, an effective approach is to query the model with the context of the target sequence @xcite . to measure association, we try to impersonate attackers to attack the model by querying with various prompts. we focus on testing the models on email addresses. an email address consists of two major parts, local part and domain, forming localpart@domain, e.g., abcf@xyz.com. we define attack tasks based on memorization and association: 1) given the context of an email address, examine whether the model can recover the email address; 2) given the owner's name, query plms for the associated email address with an appropriate prompt. ## data and pre-trained model we test on the gpt-neo model family @xcite (125 million, 1.3 billion, and 2.7 billion parameters), which are causal language models pretrained on the pile @xcite , a large public corpus that contains text collected from 22 diverse high-quality datasets, including the enron corpus. the enron corpus foot_1 @xcite ) is a dataset containing over 600,000 emails generated by employees of the enron corporation. we process the corpus to collect (name, email) pairs. following @xcite , we firstly parse all the email contents to get the body parts. in these email bodies, all the email addresses are extracted. then referring to the uc berkeley enron database foot_2 , we map the email addresses to their owners' names to get (name, email) pairs. the enron company email addresses have an obvious pattern of first_name.last_name@enron.com. language models can easily follow this pattern to predict an email address given the owner's name, which makes the analysis meaningless. therefore, in the experiments, we only focus on the non-enron domain addresses. to build the few-shot settings (explained in section 5), we filtered out email addresses whose domain appears less than 3 times in the corpus. we also filtered out pairs whose name has more than 3 tokens, in which case can be considered invalid. after all the pre-processing, there are 3238 (name, email) pairs collected for the following experiments. ## method we design different prompts and feed them into gpt-neo. we generate 100 tokens and use regular expression matching to find the email addresses. the first email address appearing in the output texts is extracted as the predicted email address. there are cases where no email address appears in the output texts. we use greedy decoding in the decoding process of generation by default and report results of other decoding algorithms in appendix b. assuming ({name0}, {email0}) is the target pair, the experiments are designed as follows. ## result & analysis tables 1-3 show the results of all the above experiments with three different sized gpt-neo models. # predicted denotes the number of predictions with email addresses appearing in the generated text. # correct shows the number of email addresses predicted correctly. (# no pattern) means, out of the correct predicted ones, the number of email addresses that do not conform to standard patterns in ## mitigating privacy leakage now that we have seen some potential risks of plms in terms of personal information leakage. here we discuss several possible strategies to mitigate these threats. for training plms, we can mitigate privacy risks before, during, and after model training: • pre-processing. 1) identify and clear out or blur long patterns that could pose potential risks, e.g., the pattern of 0-shot (d); 2) deduplicate training data. according to @xcite , deduplication can substantially reduce memorized text; therefore, less personal information will be memorized by plms. ## conclusion our paper presents the first distinction between memorization and association in pre-trained language models. the results show that plms do leak personal information through memorization; however, the risk of specific personal information being leaked by plms is low since they cannot associate personal information with the owner meaningfully. we suggest several defense techniques to mitigate potential threats and hope this study can give new insights to help the community understand the risk of plms and make plms more trustworthy.