Preserving Privacy Through Dememorization: An Unlearning Technique For Mitigating Memorization Risks In Language Models (2023)

Abstract: Large Language models (LLMs) are trained on vast amounts of data, including sensitive information that poses a risk to personal privacy if exposed. LLMs have shown the ability to memorize and reproduce portions of their training data when prompted by adversaries. Prior research has focused on addressing this memorization issue and preventing verbatim replication through techniques like knowledge unlearning and data pre-processing. However, these methods have limitations regarding the number of protected samples, limited privacy types, and potentially lower-quality generative models. To tackle this challenge more effectively, we propose “DeMem,” a novel unlearning approach that utilizes an efficient reinforcement learning feedback loop via proximal policy optimization. By fine-tuning the language model with a negative similarity score as a reward signal, we incentivize the LLMs to learn a paraphrasing policy to unlearn the pre-training data. Our experiments demonstrate that DeMem surpasses strong baselines and state-of-the-art methods in terms of its ability to generalize and strike a balance between maintaining privacy and LLM performance.

## introduction large language models (llms) have experienced exponential growth in recent years, scaling up from millions to billions to trillions of parameters @xcite @xcite . as their scale increases, the training sets for these models also expand to billions of tokens @xcite , leading to overall performance improvements, even in few-shot learning scenarios @xcite . however, this growth in model size and training data has raised practical concerns regarding privacy risks associated with memorizing the training data. adversaries can extract individual sequences from a pre-trained model, even if the training dataset is publicly available @xcite . studies have shown that a language model with 6 billion parameters (gpt-j) can memorize at least 1% of its training data @xcite . one potential cause of this memorization is the training strategy of the language model, as its objective is to identify the relationships between tokens, either in an auto-regressive lm setup or through masked language modelling (mlm) @xcite , where the model predicts the masked tokens based on their surrounding context @xcite . additionally, repeated instances in the training corpus can contribute to memorization, as more frequent examples are more likely to be memorized @xcite . to address the issue of memorization in llms, several approaches have been proposed, including data sanitization @xcite , the application of differential privacy algorithms @xcite @xcite @xcite , data deduplication @xcite , and knowledge unlearning @xcite . these techniques aim to prevent the generation of memorized content. however, they also come with certain drawbacks. data sanitization assumes that private information can be easily identified and is not context-dependent. differential privacy can lead to lower-quality generative models @xcite . on the other hand, knowledge unlearning restricts the number of samples that can be forgotten at once to avoid degrading the overall capability of the language model, which may limit its effectiveness in real-world scenarios. in this study, we propose dememorization (de-mem), a reward-based (un)learning framework for language models. demem leverages a paraphrasing policy to address memorization, using a negative similarity metric as a reward to encourage the language model (lm) to unlearn. given samples of prefixes and suffixes from the original pre-training data of the language model, we use a prefix as input for the language model to generate the suffix; then, we compute the negative bertscore @xcite to measure the dissimilarity between the true suffix and generated suffix, the dissimilarity scores are then regarded as a reward signal to maximize in the training process, which guarantees that the approximate memorization will be mitigated. for instance, given a training sample like "alice green lives at 187 bob street," where the prefix is "alice green lives at" and the suffix is "187 bob street", our goal is to have the fine-tuned lm paraphrase the suffix as "12 red street." this paraphrasing approach minimizes the memorization relationship between the prefix and suffix without erasing the training sample from the lm's parameters or replacing it with meaningless content, which can negatively impact the lm's performance we conducted experiments using gpt-neo and opt lms (with models ranging from 125m to 2.7b parameters) @xcite . demem achieved little to no performance degradation on the initial lm capabilities measured via nine common nlp classification benchmarks (hellaswag @xcite , lambada @xcite , winogrande @xcite , copa @xcite , arc-easy, arc-challenge @xcite , piqa @xcite , mathqa @xcite , and pub-medqa @xcite ). we also evaluate demem on increasing the context of the prefix, as many studies show that as a longer context is provided, the memorization ratio increases @xcite . the proposed framework makes no explicit, implicit as-sumptions or limitations about the data's structure or size to be protected. also, unlike the dp methods, the proposed framework does not apply any partition mechanism to split the data into public data and private data; as language data cannot be partitioned @xcite , we apply the policy on all training data as defining, partitioning data into private and public, and limiting the number of samples inadequate in the real-world scenarios. to summarize, our main findings are the following: ## related work in this section, we delve into recent studies to mitigate memorization in language models, which can be categorized into three main approaches: data pre/post-processing, differential privacy methods, and knowledge unlearning. ## experiments in this section, we begin by introducing the dataset used for training and assessing the paraphrasing policy. subsequently, we assess the overall performance of the dememorized lm general performance on nine benchmarks. we then establish the baseline methods for comparison. finally, we define the evaluation metrics that enable us to measure the memorization and the performance in downstream tasks. ## conclusion in this paper, we present a novel framework that tackles the problem of training data memorization in llms. we achieve this by employing an rl paraphrasing policy. through extensive evaluations conducted in diverse settings, we demonstrate the effectiveness of our approach. our framework successfully reduces memorization by significantly decreasing the sacrebleu score while preserving the overall capabilities of the lm as measured by nine classification benchmarks.