R . R .: Unveiling LLM Training Privacy through Recollection and Ranking (2025)

Abstract: Large Language Models (LLMs) pose significant privacy risks, potentially leaking training data due to implicit memorization. Existing privacy attacks primarily focus on membership inference attacks (MIAs) or data extraction attacks, but reconstructing specific personally identifiable information (PII) in LLMs’ training data remains challenging. In this paper, we propose (Recollect and Rank), a novel two-step privacy stealing attack that enables attackers to reconstruct PII entities from scrubbed training data where the PII entities have been masked. In the first stage, we introduce a prompt paradigm named recollection, which instructs the LLM to repeat a masked text but fill in masks. Then we can use PII identifiers to extract recollected PII candidates. In the second stage, we design a new criterion to score each PII candidate and rank them. Motivated by membership inference, we leverage the reference model as a calibration to our criterion. Experiments across three popular PII datasets demonstrate that the achieves better PII identification performance than baselines. These results highlight the vulnerability of LLMs to PII leakage even when training data has been scrubbed. We release our code and datasets at GitHub.

## introduction large language models (llms) have been widely adopted in various applications, like automated content generation @xcite to personalized virtual assistants @xcite . this success is driven by massive training datasets, which raise significant concerns about privacy leakage. previous research has shown that llms can memorize portions of their training data, even before overfitting @xcite . these training datasets of llm † corresponding author 1 https://github.com/meng-wenlong/rr ( +148% +110% may contain sensitive personally identifiable information (pii), such as names, phone numbers, and email addresses. additionally, llms are usually fine-tuned with custom datasets to meet application needs, which may introduce further privacy risks. llm memorization is an inherent and inevitable part of training. thus, the security of llm privacy is a crucial and widespread topic of public concern. this paper focuses on attacks that steal the privacy of training datasets. one widely studied privacy attack is membership inference attack (mia) @xcite , which aims to determine whether a specific data record was included in the training set. however, mias targeting llms usually require the attacker to obtain complete text samples, which is often impractical in real-world scenarios @xcite . another line of attack is data extraction @xcite , where an adversary attempts to reconstruct training data as much as possible. however, the impact of data extraction is limited, as the extracted data may not contain the target information an attacker seeks. recently, @xcite proposed pii reconstruction attack, which refers to the task where an attacker attempts to recover the masked pii entities in a piece of text. for example, in the masked sentence "the shipment for [mask] was delivered to 1234 elm street," if the attacker successfully reconstructs @xcite , they can identify the individual residing at that address. this practical attack has attracted great attention in the community. scrubbed datasets publicly released by llm developers may serve as potential targets for such attacks. nevertheless, existing pii reconstruction methods based on prefix continuation @xcite or perplexity scoring @xcite , lack contextual dependencies and insights from the original llm, yielding a reconstruction accuracy below 10%. our contributions. in this paper, we propose r.r. (recollect and rank), a novel and effective pii reconstruction method that assumes black-box access to the victim llm. r.r. consists of two stages: (1) recollection, generating pii candidates, and (2) ranking, identifying the candidate most likely to be pii from candidates. in the recollection stage, we provide the entire masked text as input to the victim llm and prompt it to reproduce the text without masks. unlike traditional prefix continuation methods, which rely solely on the preceding text, our approach leverages both preceding and succeeding contexts. once the model finished generating the recalled text, r.r. uses a pii identifier to extract internal pii entities. after repeating recollection several times, we get a pii candidates pool. in the ranking stage, extracted pii candidates are inserted into the mask placeholder. we use the cross-entropy loss as the criterion for ranking, and the candidate with the lowest loss is selected as the reconstructed result. inspired by reference-based mias @xcite , we use the loss from the pre-trained model, on which the victim model is fine-tuned, as a reference to calibrate the ranking criterion. interestingly, we observe that reference calibration does not always improve ranking accuracy. to address it, we propose a biased ranking criterion that blends uncalibrated and calibrated criteria. we theoretically prove that our proposed criterion preserves the strengths of both. experimental results of four llms show that r.r. achieves an average top-1 accuracy of 25.73% on the neurips llm-pc dataset @xcite , marking a more than 100% improvement over previous sotas, as presented in figure 1 . in summary, our contributions are as follows: ## preliminaries given a sequence of tokens @xmath0, x 2 , . . . , x t ), the cross-entropy loss is defined as: ## r.r. methodology the overall pipeline of r.r. is illustrated in figure 2. our r.r. reconstructs masked pii in two steps: candidate generation and candidate selection. during candidate generation, r.r. iteratively induces the victim llm to generate possible pii candidates, which are then ranked in candidate selection. the top-1 candidate is selected as the reconstructed result. ## conclusion in this paper, we propose r.r. (recollect and rank), a novel two-step pii reconstruction attack. first, we employ a recollection-based method to recover the original text and leverage a ner-modelpowered pii identifier to extract potential pii candidates. then, we rank the pii candidates by computing their cross-entropy scores under the victim model and a reference model. we further introduce a biased ranking criterion that effectively integrates reference-based and non-reference-based ranking methods. our experiments across three popular pii datasets and four open-source llms demonstrate that the r.r. significantly improves top-1 accuracy.