PII -Compass: Guiding LLM training data extraction prompts towards the target PII via grounding (2024)

Abstract: The latest and most impactful advances in large models stem from their increased size. Unfortunately, this translates into an improved memorization capacity, raising data privacy concerns. Specifically, it has been shown that models can output personal identifiable information (PII) contained in their training data. However, reported PII extraction performance varies widely, and there is no consensus on the optimal methodology to evaluate this risk, resulting in underestimating realistic adversaries. In this work, we empirically demonstrate that it is possible to improve the extractability of PII by over ten-fold by grounding the prefix of the manually constructed extraction prompt with in-domain data. This approach achieves phone number extraction rates of 0.92%, 3.9%, and 6.86% with 1, 128, and 2308 queries, respectively, i.e., the phone number of 1 person in 15 is extractable.

## introduction memorization in large language models (llms) has recently enjoyed a surge of interest @xcite ranging from memorization localization @xcite , quantification @xcite to controlling @xcite and auditing @xcite . the major reason for this is the risk of training data extraction @xcite . to assess this risk, various methods have been proposed in prior work @xcite @xcite . in this work, we aim to assess the privacy leakage risk of a subclass of training data, namely personal identifiable information (pii) from base llms. more specifically, we focus on the pii extraction attacks in the challenging and realistic setting of black-box llm access. ## experiments following the experimental setup in @xcite , we use a post-processed version of the enron email dataset @xcite which maps persons to their phone numbers. we further filter out annotations (pairs of names and phone numbers) that are non-numeric or have ambiguous multiple ground-truth annotations, resulting in a total of 2,080 data subjects containing (name, phone number) pairs. similar to @xcite , we use the gpt-j-6b @xcite ) model as the target llm which was trained on the enron email dataset. we split this dataset into two parts: the adversary dataset containing 128 data subjects that can serve as additional knowledge available to the attacker, and the evaluation dataset that containing the 1,952 remaining data subjects. we assume black-box access to the target base llm and the availability of true prefixes of the data subjects in the adversary dataset. we believe our assumption about access to an adversary dataset is realistic since (small) portions of the dataset could be acquired legally, e.g., purchased, or illegally, e.g., leaked. we perform greedy decoding during the generation process. we report the pii extraction rate as the percentage of data subjects in the evaluation dataset for which we can extract the correct phone number. we provide more details about the experimental setting in the appendix a. ## conclusion in this work, we highlight the limitations of handcrafted templates in extracting phone number pii. to overcome this, we propose pii-compass, a simple yet effective prompt grounding strategy that prepends the manual templates with the true prefix of a different data subject. our empirical experiments demonstrate the effectiveness of pii-compass, yielding an impressive over ten-fold increase in pii extraction rates compared to the baselines. in the future, we aim to study the pii extraction rate by leveraging the zero-shot capabilities of gpt-4 to generate prefixes that can guide the extraction towards the target pii even in the absence of an adversary dataset.