Learnable Privacy Neurons Localization in Language Models (2024)

Abstract: Concerns regarding Large Language Models (LLMs) to memorize and disclose private information, particularly Personally Identifiable Information (PII), become prominent within the community. Many efforts have been made to mitigate the privacy risks.However, the mechanism through which LLMs memorize PII remains poorly understood. To bridge this gap, we introduce a pioneering method for pinpointing PII-sensitive neurons (privacy neurons) within LLMs. Our method employs learnable binary weight masks to localize specific neurons that account for the memorization of PII in LLMs through adversarial training. Our investigations discover that PII is memorized by a small subset of neurons across all layers, which shows the property of PII specificity. Furthermore, we propose to validate the potential in PII risk mitigation by deactivating the localized privacy neurons. Both quantitative and qualitative experiments demonstrate the effectiveness of our neuron localization algorithm.

## introduction large language models (llms) have demonstrated exceptional performance on various nlp tasks, leveraging huge model architectures and a tremendous scale of real-world training data @xcite @xcite . however, the ability of memorization within llm has also raised concerns regarding security within human society @xcite . one significant concern is that private information may be memorized and leaked by llms. an attacker can extract private information contained in the training corpus, especially personally identifiable information (pii) such as names or addresses @xcite @xcite @xcite , which constitutes a privacy violation according to the general data protection regulation (gdpr) @xcite . various methods have been proposed to mitigate the memorization of pii @xcite , primarily focusing on the sanitization of training data @xcite , or providing differential privacy (dp) guarantees during the training process @xcite . however, the mechanism by which llms memorize pii is not well understood. in this paper, we propose a novel privacy neuron localization algorithm. our method utilizes the hard concrete distribution @xcite to make neuron masks learnable and design adversarial objective functions to minimize the predictive accuracy of pii while preserving other nonsensitive knowledge. besides, we employ another penalty to minimize the number of localized neurons, thus localizing a minimal subset of pii-specific neurons. we subsequently conduct a comprehensive analysis of the localized privacy neurons. our findings reveal that memorization is localized to a minor subset of neurons, which are spread across all layers, predominantly within the mlp layers. furthermore, we also discover that privacy neurons have the property of specificity for certain categories of pii knowledge. inspired by the observation, we propose to investigate the privacy leakage mitigation ability by deactivating the localized neurons during the evaluation process, thus eliminating the memorization of pii. experimental results demonstrate that our framework can achieve comparable performance in mitigating the risks of pii leakage without affecting model performance. ## method denote f (θ) as a plm with parameters θ. given a sequence of tokens @xmath0, ..., x t ] from the training corpus, f (θ) can leak the private sequence [x p , ..., x p+i ] within x by generating [x p , ..., x p+i ] = arg max * (p f (θ) ( * |@xmath1. for example, as shown in fig. 1 , the email address of kent garrett is disclosed by the model, which constitutes significant societal risks. in this section, we introduce a novel neuron localization algorithm that localizes neurons in f (θ) responsible for pii prediction, to elucidate the underlying mechanisms of pii memorization, as illustrated in fig. 1 . to be specific, our goal is to find a small subset of neurons f (m ⊙ θ) (or equivalently, the mask m) that deactivating these neurons prevents pii leakage, while not affecting the language modeling ability, thus indicating the memorization of pii-specific knowledge. m and ⊙ denote the differentiable binary neuron mask and hadamard product operator respectively. ## conclusion in this paper, we propose a novel method for jointly localizing a small subset of pii-sensitive neurons within llms. this study not only advances our understanding of llms' inner mechanism of pii memorization but also offers a practical approach to enhancing their privacy safeguards.