Privacy Ripple Effects from Adding or Removing Personal Information in Language Model Training (2025)

Abstract: Due to the sensitive nature of personally identifiable information (PII), its owners may have the authority to control its inclusion or request its removal from large-language model (LLM) training. Beyond this, PII may be added or removed from training datasets due to evolving dataset curation techniques, because they were newly scraped for retraining, or because they were included in a new downstream fine-tuning stage. We find that the amount and ease of PII memorization is a dynamic property of a model that evolves throughout training pipelines and depends on commonly altered design choices. We characterize three such novel phenomena: (1) similar-appearing PII seen later in training can elicit memorization of earlier-seen sequences in what we call assisted memorization, and this is a significant factor (in our settings, up to 1/3); (2) adding PII can increase memorization of other PII; and (3) removing PII can lead to other PII being memorized.

## introduction one of the most common methods to adapt large language models like chatgpt @xcite and gemini @xcite for specific applications is to fine-tune them on domain-specific datasets. 1 when these datasets contain private or personal data, models may be at risk of memorizing 2 and regurgitating @xcite this information. though it is common to filter out sensitive information 3 such as pii @xcite , some sensitive information may still remain @xcite . moreover, some downstream tasks, such as healthcare, may require pii, making eliminating pii completely from model training datasets challenging. modern-day language models deployed in realworld settings are also increasingly dynamic: it is common practice to continually update or retrain them with new and/or additional data @xcite @xcite , e.g., if new users opt to share their data. there may also be data removal requests from existing users under the right to be forgotten @xcite . here, machine unlearning @xcite is often the proposed solution by enabling post-hoc removal of data (e.g., pii) from neural models after training. llms are known to memorize and regurgitate personal information and pii @xcite , which is a concrete privacy harm we study. in this literature, little focus has been given to how this may arise dynamically as a part of a machine learning system. in this work, we study how various actions (continually training on more data, re-training with new data, or re-training sitive information. we thus use these terms interchangeably. memorization & extraction studies when a text is trained on and generated by a model. this is widely studied @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . these works are often focused on the broad phenomenon, and not the nature of the data, e.g., if it were sensitive as in our work. relatively fewer works have considered this setting. huang et al. (2022) study if information about specific entites can be extracted; panda et al. (2024) study if llm's can be poisoned to memorize specific pii; lukas et al. (2023) formalize pii extraction, proposing several attacks and studying the efficacy of various existing defenses; mireshghallah et al. (2022) and zeng et al. (2024) study memorization during fine-tuning; and lehman et al. (2021) found that extracting sensitive data, using simple techniques, from bert trained on clinical notes was largely unsuccessful. this line of work has become important for practical privacy and memorization audits @xcite @xcite , which also often include pii memorization evaluations @xcite @xcite . ## experimental setup our goal is to study how memorization of pii manifests during training. 5 this includes continual training or fine-tuning setups in §4 and re-training or unlearning setups in §6. first, we describe our general experimental setup. ## a dynamic lens on pii memorization production language models today consist of many training stages (pre-training, post-training, productspecific fine-tuning, etc.) and may be continually updated or refreshed with new data, e.g., to incorporate new human data using rlhf @xcite . these stages may incorporate varying degrees of personal information. this raises the question: how does memorization of sensitive data like pii evolve in this dynamical system? continuous training setup. to study this question, we use the simplest setup that generally captures all of the above scenarios: we study memorization throughout supervised fine-tuning. we train a model by keeping the rate of emails seen constant and save checkpoints at regular intervals (for efficiency, only every 10% of training). details on the dataset construction are in §3. ## assisted memorization: training on one's pii can reveal another's in figure 3 , we see that a large fraction of memorization is assisted. this is especially true later in training, where we observe that more memorization is assisted than immediate, specifically a mean rate of 0.03 for assisted compared to 0.01 for immediate. this finding is not model-or data-specific, as our results in appendix c (e.g., figures 17 and 18 ) show similar trends. the existence of assisted memorization brings to light a deeper privacy concern. one may expect that data seen earlier is less vulnerable to privacy risks through a form of "recency bias" (implied by forgetting effects). our findings of assisted memorization, however, show that this may not always be the case; the existence of this effect with sensitive data like pii is of particular concern because it shows that downstream training stages must be careful how they may elicit the extraction of earlier training data. the most common practical scenario for this is in the pre-training/fine-tuning setup that current llms undergo. our results show that fine-tuning even on natural (non-adversarially) constructed training datasets can uncover the extraction of pii in pre-training data. prior work @xcite only showed that this may be possible with adversarial constructions. pragmatically, our results also show that privacy and memorization audits, especially when pii is of concern, should encompass all data in the training history, and not just data from the most recent training stage. ## conclusion we study how the actions of continually training on more data, re-training with new data, or re-training after removing data can have ripple effects for privacy. in particular, we propose the phenomenon of assisted memorization where examples that aren't extracted at existing checkpoints can get extracted later. this could create a false impression of privacy for examples that don't get extracted at a particular checkpoint, as training further on similarappearing examples could lead to their extraction. we also find that including more pii in the training data can degrade privacy of existing pii by putting them at a higher risk of extraction. furthermore, removing particular pii examples from training data could cause other examples to be extracted. this underscores the need for more holistic audits for memorization, where examples that aren't extracted at a particular timepoint are also evaluated for any potential risks.