Mitigating Privacy Seesaw in Large Language Models: Augmented Privacy Neuron Editing via Activation Patching (2024)

Abstract: Protecting privacy leakage in large language models remains a paramount challenge. In this paper, we reveal Privacy Seesaw in LLM privacy safeguarding, a phenomenon where measures to secure specific private information inadvertently heighten exposure risks for other privacy. Through comprehensive analysis, we identify the amount of targeted privacy data and the volume of edited privacy neurons as the two central triggers to this issue. To mitigate privacy seesaw, we propose Augmented Privacy Neuron Editing via Activation Patching (APNEAP), a novel framework designed to well balance model performance with privacy protection. The proposed APNEAP augments collected private data by automatically synthesizing new private data, which deactivates the first trigger to the privacy seesaw issue. Additionally, it adapts activation patching to privacy neuron editing for switching off the second trigger to the privacy seesaw problem. Experimental results show that the proposed APNEAP is capable of alleviating the privacy seesaw phenomenon and offers a more stable and reliable approach to privacy protection in LLMs than previous methods.

## introduction large language models have demonstrated outstanding capabilities in natural language understanding and generation, significantly advancing downstream natural language processing (nlp) tasks @xcite @xcite . however, llms trained on vast amounts of internet data encounter critical security and privacy challenges in real-life application scenarios @xcite @xcite . this is mainly due to two reasons. first, training data for llms often contain sensitive or unauthorized information, which is subjected to limited scrutiny because of its massiveness and confidentiality @xcite . second, llms tend to memorize training data, including unique instances @xcite . previous studies have shown that private information could be successfully extracted from llms such as chatgpt with meticulously crafted prompts, underscoring the urgency of privacy protection for llms @xcite . in order to protect privacy of llms, machine unlearning and neuron-based methods have been proposed. the former aims to make llms forget targeted datasets through fine-tuning on small batches of data. @xcite render private information harmless through sanitization tuning. @xcite reduce privacy leakage risks by reversely learning the gradient of private data. the later seeks to reduce the likelihood of eliciting private information by editing neurons directly. @xcite propose depn, an efficient approach to locating and editing privacy neurons for language models. @xcite show that there may be loopholes in the effectiveness evaluation of privacy protection methods, which may lead to the neglect of emerging privacy leakage risks. in our experiments, we have found that the neuron-based protection method may lead to additional privacy leakage risks. figure 1 illustrates our experimental results, clearly demonstrating the reduced risk of privacy leakage for memorized private data (text 1-5) but increased risk associated with unmemorized private data (text 7). this finding highlights the limitations of current neuron based protection approaches in fully addressing privacy-preserving scenarios. we refer to this phenomenon as privacy seesaw (ps), where memorized private information is protected at the cost of exposing other private information that originally has no risk of leakage. we delve into the ps phenomenon (see details in section 5.2), and find two main reasons for ps. first, the incomplete distribution of collected privacy data represents only a small part of the entire privacy landscape. the second reason is limited number of privacy neurons that can be edited to avoid significant impact on the model performance, by current neuron-based methods (e.g., depn @xcite ). to address these issues, we propose augmented privacy neuron editing via activation patching (apneap), which employs data augmentation to expand the privacy dataset and adapts activation patching to efficient privacy neuron editing. the used data augmentation alleviates the first cause to ps while the adapted activation patching overcomes the constraints on the number of neurons that can be edited. extensive experiments demonstrate the effectiveness of the proposed apneap in mitigating the ps and improving privacy protection over strong baselines, while maintaining high efficiency and stability. our contributions can be summarized as follows. ## related work privacy protection in nlp privacy protection in language models are categorized into three stages: data processing, training & fine-tuning, and postprocessing @xcite . in data processing, methods like redirection and anonymization aim to remove sensitive information @xcite . during training, differential privacy techniques @xcite are employed at the expense of computational time and performance. post-processing involves making models forget leaked information through machine unlearning @xcite @xcite or neuron editing @xcite , fine-tuning on target datasets or directly editing model parameters. @xcite show that the feedforward network module in the transformer can be viewed as a key-value memory, where each key corresponds to a text pattern and each value represents a distribution over the vocabulary. based on this finding, a series of studies @xcite @xcite have proposed for editing factual knowledge encoded in pre-trained llms by locating neurons related to factual knowledge entities. @xcite discover that factual knowledge can be transferred across languages, with the cross-lingual alignment of knowledge neurons. @xcite extend this approach to privacy protection, aiming to safeguard private data by locating and editing privacy neurons. ## augmented privacy neuron editing via activation patching in order to solve the challenge of ps, we propose apneap, illustrated in figure 2 , which includes two essential components: privacy data augmentation and activation patching. the new framework contains four main modules: augmenting privacy data, locating privacy neurons, selecting privacy neurons, and editing privacy neurons. ## experiments in this section, we present our experimental setup and explain how we discovered the privacy seesaw phenomenon and analyzed its causes. we then demonstrate the effectiveness of our approach, in maintaining a balance between privacy protection and model performance while effectively mitigating the privacy seesaw challenge. our codes are available now. foot_0 ## conclusion in this paper, we have identified the privacy seesaw phenomenon as a previously underexplored problem in llm privacy protection, where efforts to protect certain private data instances inadvertently increase exposure risks for others. we pinpoint the amount of targeted privacy data and the number of privacy neurons being edited as key triggers of this phenomenon. to tackle this, we proposed ap-neap, effectively balancing model performance with privacy protection and significantly reducing privacy leaks. apneap also successfully mitigates the privacy seesaw issue, offering a more reliable privacy protection framework than previous neuronbased methods. while apneap shows promising results, further exploration in privacy neuron-based methods is needed.