Large Language Models Can Be Contextual Privacy Protection Learners (2024)

Abstract: The proliferation of Large Language Models (LLMs) has driven considerable interest in fine-tuning them with domain-specific data to create specialized language models. Nevertheless, such domain-specific fine-tuning data often contains contextually sensitive personally identifiable information (PII). Direct fine-tuning LLMs on this data without privacy protection poses a risk of data leakage of sensitive PII during inference time. To address this challenge, we introduce Contextual Privacy Protection Language Models (CPPLM), a novel paradigm for fine-tuning LLMs that effectively injects domain-specific knowledge while safeguarding inference-time data privacy. Our work offers a theoretical analysis for model design and delves into various techniques such as corpus curation, penalty-based unlikelihood in training loss, and instruction-based tuning, etc. Extensive experiments across diverse datasets and scenarios demonstrate the effectiveness of our approaches. In particular, instruction tuning with both positive and negative examples, stands out as a promising method, effectively protecting private data while enhancing the model’s knowledge. Our work underscores the potential for Large Language Models as robust contextual privacy protection learners.

## introduction large language models (llms) have demonstrated remarkable linguistic comprehension and generation capability @xcite . meanwhile, when directly applied to specialized industries, they encounter challenges such as hallucination @xcite @xcite , insufficient domain expertise @xcite , and failing to incorporate the latest domain knowledge in ever-evolving industry scenarios @xcite . the introduction of open-source general-purpose llms such as llama @xcite and rwkv @xcite have provided a promising solution. researchers would fine-tune specialized llms based on powerful general-purpose llms using high-quality, domain-specific knowledge to ensure both commonsense reasoning and comprehensive knowledge coverage @xcite @xcite . such examples include bloomberggpt @xcite and med-palm @xcite , for financial and medical applications, respectively. however, these fine-tuning datasets usually contain sensitive information, such as personally identifiable information (pii) @xcite @xcite . when applied to downstream tasks, sensitive information in the training data, such as social security numbers or patient names, can be exposed by the llms upon text generation, a phenomenon known as the memorization effect @xcite @xcite or inference-time privacy threat @xcite , leading to identity theft and financial losses @xcite . challenges. in this work, we aim to tackle the challenging task of efficient llm fine-tuning for enhanced contextual privacy @xcite , a critical yet underexplored setting where the sensitivity of a piece of information is contingent upon the context. for example, statements such as "bill gates founded microsoft" and "alan mathison turing was an english mathematician and computer scientist" are generally not considered violations of privacy, since they are presented as common knowledge. in contrast, statements like "alan gates visited the x hospital for a certain disease y" pose privacy concerns as they reveal details about individuals' daily activities and health status in a particular context. directly applying techniques like named entity recognition (ner) can lead to inaccurate identification of pii, whereas merely deleting or masking pii tokens in the training data would result in a substantial information loss and compromise the performance on downstream tasks -a conundrum known as the privacy-utility trade-off as theoretically discussed in sec. 4.1. an alternative approach, reinforcement learning from human feedback (rlhf), involves additional model finetuning guided by human feedback @xcite so that the model tends towards concealing sensitive pii (like "red-teaming"). for example, it learns to prioritize outputs that protect sensitive pii over those that leak pii. nonetheless, rlhf is data-intensive, potentially costly in computation, and can pose stability challenges @xcite . our work. to address these challenges, this paper introduces effective and efficient methodologies for fine-tuning llms to incorporate domain knowledge while ensuring privacy protection. we propose and rigorously examine a diverse suite of strategies from corpus curation, introduction of penalty-based unlikelihood into the training loss, instruction-based tuning, a pii contextual classifier, and direct preference optimization (dpo), etc. the ultimate objective is to cultivate a model that excels at acquiring information while demonstrating the ability to distinguish between information that can be openly shared and that demands strict confidentiality. our experimental findings suggest that instruction tuning with positive and negative examples can offer promising avenues. it not only effectively shields private data but also enables the model to assimilate knowledge from the corpus. this implies that llms can be good contextual privacy protection learners, without the need for balancing a privacy-utility trade-off. to sum up, our contributions are as follows. ## related work large language models and privacy. in the rapidly advancing domain of artificial intelligence and natural language processing, llms such as gpt-3.5/4 (openai, 2023), bard @xcite , llama @xcite , and chatglm @xcite have demonstrated unprecedented capabilities in following instructions @xcite and generating coherent, contextually accurate text @xcite @xcite . however, this widespread application raises significant privacy concerns, particularly regarding personal information protection. addressing the privacy challenges posed by llms, researchers have focused on three primary strategies: @xcite @xcite : 1) curation of the pretraining corpus, 2) conditional large language model (llm) pretraining, and 3) post-training alignment. our research focuses on enhancing privacy protection in llms through fine-tuning methods that enable knowledge injection to safeguard personally identifiable information (pii) @xcite , as designated by users. this contrasts with differential privacy (dp), which protects against the leakage of entire records at the cost of reduced data utility @xcite . our method emphasizes targeted pii protection, a crucial aspect in contexts where knowledge integration is the key to preserving privacy without compromising data utility @xcite @xcite @xcite @xcite . for the fine-tuning of llms, the decline in utility is inversely linked to the privacy budget allocated for safeguarding the entire training document, as it determines the extent of noise introduced @xcite . our emphasis lies in specifically safeguarding the contextual pii tokens. since piis are contextual @xcite , our approach tunes llms with contrastive examples designated by users can accommodate the customized privacy preferences. filtering. for the pretraining corpus, manually detecting and filtering out/revising the corpus can offer high-quality corpus, which is ideal for training privacy-preserving llms @xcite @xcite . nevertheless, it is infeasible to process billions of tokens manually in practice. another solution is using automated tools to filter out all sensitive content (e.g. names, addresses, phone numbers) from the pretraining corpus. automated filters make it possible to go over pretraining datasets. however, simply removing or masking the pii tokens (i.e., pii scrubbing) can cause information loss or inconsistency in the corpus @xcite . though filters can 'clean' datasets, they reduce the diversity in the corpus, which further negatively impacts the robustness of llms @xcite . another solution is adding content filters on top of the existing lms to control the content generation process @xcite . even so, carefully designed cases (e.g. prompts) can still trigger some undesired behaviors of large lms @xcite . however, directly removing pii from the training corpus poses a dilemma. while it ensures the elimination of sensitive data, it also potentially weakens the llms by stripping them of crucial knowledge. the mere act of omitting data can inadvertently hamper the model's capacity to process and understand certain contexts. context-awareness is fundamental when considering privacy protection and what data to shield. ## problem statement problem formulation. in the context of language models, a fine-tuning dataset @xmath0. each sequence is denoted as @xmath1, w 2 , . . . , w n-1 ], where w @xmath2. for privacy protection, the users annotate each sequence in the corpus by a binary sequence p denoted as @xmath3, . . . , p n-1 ], p @xmath4, 1}, where p @xmath5.g., pii) need to be protected in the context, and p @xmath6. here, the contextual privacy posits that the sensitivity of a piece of information is not solely intrinsic to the information itself, but is also influenced by its surrounding context. to illustrate, "alan gates visited crescent vale medical center for hemophilia treatment" is considered more indicative than "alan gates visited crescent vale." the former provides a clearer insight into an individual's health when the name "alan gates" is paired with the medical condition and the specific medical center. important notations used in the paper are included in objective. the primary objectives are twofold: 1) enhancing the model's performance by effectively integrating knowledge from the fine-tuning corpus. the model should generate responses that are contextually relevant and aligned with the intended domain; 2) minimizing the risk of generating privacy-protected tokens. privacy protection in large language models requires not just the masking or removal of private piis, but a deep understanding of the interplay between data points and their contexts. as models become more sophisticated and data more interconnected, the nuances of contextual privacy will become increasingly paramount. ## methodology our methodology adopts a two-pronged approach: 1) corpus curation (i.e. filtering), where sensitive data such as personally identifiable information (pii) is removed from the corpus; and 2) tuning towards the targeted pii-free output. we commence with a theoretical analysis of the information loss incurred by the corpus curation strategy, which provides guidelines for method development. then, we propose five novel strategies for privacy protection when fine-tuning large language models. ## experiments in this section, we empirically verify the effectiveness of the proposed approaches. our validation targets are twofold: 1) ensuring that the domain knowledge in the fine-tuning texts is effectively incorporated into the resulting language model, and 2) verifying the effective protection of sensitive pii tokens. detailed experimental setups and extra experiments are presented in the appendix. instruction: below are instruction paired with its question. question: please respond with some patient's recent medical record. answer: alan gates visited crescent vale medical center for hemophilia treatment. ## conclusion in this paper, we present a comprehensive exploration of strategies for fine-tuning large language models (llms) to incorporate domain-specific knowledge while upholding data privacy, particularly in safeguarding sensitive personally identifiable information (pii). we introduced the novel concept of contextual privacy protection language models (cpplms) and provided a theoretical analysis to guide model design. our extensive experiments underscore the effectiveness of our approach, with instruction-based tuning emerging as a promising method to simultaneously protect private data and enhance the model's knowledge. this study highlights the potential for llms to serve as adept privacy protection learners, bridging the gap between domain-specific expertise and data privacy. as llms continue to play a pivotal role in natural language understanding and generation, our findings contribute to advancing their utility in privacysensitive applications.