DEPN : Detecting and Editing Privacy Neurons in Pretrained Language Models (2023)

Abstract: Pretrained language models have learned a vast amount of human knowledge from large-scale corpora, but their powerful memorization capability also brings the risk of data leakage. Some risks may only be discovered after the model training is completed, such as the model memorizing a specific phone number and frequently outputting it. In such cases, model developers need to eliminate specific data influences from the model to mitigate legal and ethical penalties. To effectively mitigate these risks, people often have to spend a significant amount of time and computational costs to retrain new models instead of finding ways to cure the ‘sick’ models. Therefore, we propose a method to locate and erase risky neurons in order to eliminate the impact of privacy data in the model. We use a new method based on integrated gradients to locate neurons associated with privacy texts, and then erase these neurons by setting their activation values to zero.Furthermore, we propose a risky neuron aggregation method to eliminate the influence of privacy data in the model in batches. Experimental results show that our method can effectively and quickly eliminate the impact of privacy data without affecting the model’s performance. Additionally, we demonstrate the relationship between model memorization and neurons through experiments, further illustrating the robustness of our method.

## introduction remarkable progress has been made in large language models (llms) in recent years @xcite @xcite . however,despite this success, llms are confronted with privacy and security concerns in real-world applications @xcite @xcite . the primary cause of privacy and security risks is the inherent nature of large pretrained language models. previous studies @xcite @xcite * *corresponding author. @xcite have demonstrated that pretrained language models tend to memorize and regurgitate a significant portion of the training data, including atypical data points that appear only once in the training data. additionally, external factors (e.g., membership attack) also contribute to these risks. a variety of methods have been explored to attack llms for training data extraction. for instance, @xcite have successfully extracted personal information from gpt-3's output, while @xcite have induced the generation of personal information by utilizing multi-step prompts in chatgpt. all these show that large pretrained language models suffer from a serious risk of privacy leakage. in order to safeguard privacy, numerous methods have been proposed. the majority focus on either removing sensitive information during the data processing stage @xcite @xcite , or reducing the extent to which models memorize training data during the training stage @xcite @xcite . however, privacy breaches often come to light after the completion of model training, rendering previous methods less effective. there are also methods proposed in the post-processing stage, which involve slight parameter retraining to make the model forget privacy information @xcite @xcite . nevertheless, these methods generally incur high computational complexity, making it challenging to apply them to complex model architectures. in practice, model developers often attempt to prevent language models from outputting specific information via blocking or filtering certain keywords, which, however, does not truly address the underlying issue. we speculate that private information might be stored in specific neurons, just like knowledge neurons @xcite @xcite . this presumption suggests that we could change the model memorization of private information by detecting and deleting these neurons (termed as privacy neurons). therefore, we propose a framework depn for detecting and editing privacy neurons. to detect privacy neurons, we introduce a privacy neuron detector that uses gradient integration to simultaneously compute the contributions of multiple markers to neuron activations. this allows us to estimate an overall privacy attribution score for private information. subsequently, we further propose a privacy neuron editor that simply sets the activations of the top z privacy neurons with the highest privacy scores to zero to erase the model memorization of the corresponding private information. for the scenario of processing multiple sentences at the same time, we also present a privacy neuron aggregator to facilitate privacy information editing in batches. experimental results show that our framework can quickly reduce the risk of private data leakage without affecting model performance. compared with other methods, our framework is highly efficient. furthermore, we have found that model memorization leads to the aggregation of privacy neurons in our experiments, and demonstrated that our framework is very suitable for the scenario of deep model dememorization. the main contributions of our work are summarized as follows: ## preliminary privacy definition privacy preservation has become an issue of great concern in the era of pretrained language models. protecting privacy first requires specifying the boundaries of privacy. the definition of privacy is broad. it is closely related to its context and discourse @xcite . in any texts about, a specific person can be considered as private. for the convenience of research, a narrow definition of privacy is usually taken @xcite , which treats personal identity information as privacy, such as names, id numbers, phone numbers and other related expressions. the proposed depn can be adapted to the above two definitions. geva et al. (2021) find that the feed-forward network module in transformer (i.e., a two-layer perceptron) can be considered as a keyvalue memory, where each key corresponds to a text pattern and each value represents a distribution over the vocabulary. based on this finding, a strand of research, @xcite @xcite propose to edit factual knowledge encoded in pre-trained llms by locating neurons related to the entities of factual knowledge. the basic idea of localization is to change the parameters of neurons, and then observe the changes in the probability of the object entity predicted by the model. the neurons with greater influence on the probability are more closely related to the object entity. ## methodology the proposed depn consists of three components: the privacy neuron detector ( §3.2), the privacy neuron editor ( §3.3) to erase the model memorization of privacy data, and the privacy neuron aggregator ( §3.4) for privacy preservation in batches. ## experiments we carried out experiments to examine the effectiveness of the proposed depn on a dataset containing private information. ## analysis we further conducted in-depth analyses to demonstrate why depn is able to dememorize privacy in llms from multiple perspectives, including analyses on the relationship between privacy neurons and model memorization, on the robustness as well as the cost-effectiveness of depn. tion set, while the blue line represents the ppl of bert-fe on the validation set with different numbers of edited neurons. as the number of erasures increases, the ppl noticeably increases. therefore, increasing the number of edited neurons reduces the risk of privacy leakage in the model, but it also leads to a decrease in the model performance. ## related work model editing to edit incorrect or undesirable information captured in llms, a variety of model editing approaches have been proposed, which can be categorized into four strategies. first, the constrained fine-tuning strategy @xcite updates llms specifically for the target knowledge, allowing precise modification. second, the memory-based editing strategy @xcite maintains a knowledge cache that stores new information to replace undesirable predictions. third, the meta-learning-based editing strategy @xcite introduces editable training based on metalearning, training model parameters to accommodate editing. lastly, the locating and editing strategy @xcite @xcite assumes that knowledge is locally stored in llms. this strategy locates specific parameters associated with the knowledge and directly edits parameters to perform editing. ## conclusion in this paper, we have presented a privacy neuron detecting and editing framework depn to address privacy leakage risks in pretrained language models. through the privacy neuron detector based on the privacy attribution scoring method, we accurately detect risky neurons associated with private information. the privacy neuron editor effectively eliminates model memorization of private data. experimental results and in-depth analyses demonstrate the ability of depn to reduce privacy risks efficiently without degrading model performance. our work explores a novel approach to privacy protection and contributes to model de-memorization in the post-processing stage. limitations our current study still has two limitations. first, although we propose a method to process private data in batches, we find that too many instances in a batch will reduce the effect of memorization erasure. second, we use a few types of private information in our experiments due to the limited availability of datasets containing private information. we would like to collect more available datasets for our framework in the future.