Privacy Implications of Retrieval-Based Language Models (2023)

Abstract: Retrieval-based language models (LMs) have demonstrated improved interpretability, factuality, and adaptability compared to their parametric counterparts by incorporating retrieved text from external datastores. While it is well known that parametric models are prone to leaking private data, it remains unclear how the addition of a retrieval datastore impacts model privacy. In this work, we present the first study of privacy risks in retrieval-based LMs, particularly k NN-LMs. Our goal is to explore the optimal design and training procedure in domains where privacy is of concern, aiming to strike a balance between utility and privacy. Crucially, we find that k NN-LMs are more susceptible to leaking private information from their private datastore than parametric models. We further explore mitigations of privacy risks: When privacy information is targeted and readily detected in the text, we find that a simple sanitization step would eliminate the risks while decoupling query and key encoders achieves an even better utility-privacy trade-off. Otherwise, we consider strategies of mixing public and private data in both datastore and encoder training. While these methods offer modest improvements, they leave considerable room for future work. Together, our findings provide insights for practitioners to better understand and mitigate privacy risks in retrieval-based LMs.

## introduction retrieval-based language models @xcite @xcite @xcite ) generate text distributions by referencing both the parameters of the underlying language model and the information retrieved from a datastore of text. specifically, the retrieval process involves accessing a pre-defined datastore to retrieve a set of tokens or 1 our code is available at https://github.com/ princeton-sysml/knnlm_privacy . prompt retrieval-based lm (api-access only) attacker response data extraction attack datastore d public d private encoders enc public enc private reconstructed text email: mailme@alice.com mailme@bob.com mailme@charlie.com … url: alice@bob.com harry@hogwarts.edu … phone: 123-***-**** 404-***-**** 789-***-**** … figure 1 : retrieval-based language models (e.g., knn-lms) comprise encoders and the datastore as their key components. when tailoring a retrieval-based language model for a privacy-sensitive task, both components may utilize private data. however, a malicious user possessing api access to the model can exploit a data extraction attack in order to reconstruct sensitive information. this study explores the severity of such attacks and suggests novel strategies to mitigate them. in this work, we present the first study of privacy risks in retrieval-based language models, with a focus on the nearest neighbor language models (knn-lms) @xcite , which have been extensively studied in the literature @xcite @xcite 2 . in particular, we are interested in understanding knn-lms' privacy risks in real-world scenarios where they are deployed via an api to the general public (figure 1 ). we consider a scenario in which a model creator has a private, domainspecific dataset that improves model performance on domain-specific tasks, but may also contain sensitive information that should not be revealed; this data can be used to train encoders or stored in the datastore. in such a scenario, the model creator must find a balance between utilizing their private data to enhance model performance and protecting sensitive information. we begin our investigation by examining a situation where the creator of the model only adds private data to the retrieval datastore during inference, as suggested by @xcite . our findings indicate that while this approach enhances utility, it introduces an elevated privacy risk to the private data compared to parametric language models (section 4), and adversarial users could violate the confidentiality of the datastore by recovering sensitive datapoints. therefore, it is vital for the model creator to refrain from storing sensitive information in the datastore. we further explore mitigation strategies for knn-lms in two different scenarios. the first is where private information is targeted, i.e., can be easily identified and removed (section 5). we explore enhancing the privacy of knn-lms by eliminating privacy-sensitive text segments from both the datastore and the encoder's training process. this approach effectively eliminates the targeted privacy risks while resulting in minimal loss of utility. we then explore a finer level of control over private information by employing distinct encoders for keys (i.e., texts stored in the datastore) and queries (i.e., 2 other retrieval-based language models such as retro @xcite and atlas @xcite have significantly different architectures and the findings from our investigation may not necessarily apply to these models. investigation of these models will be left as future work. the second is a more challenging scenario where the private information is untargeted, making it impractical to remove from the data (section 6). to address this issue, we explore the possibility of constructing the datastore using public datapoints. we also consider training the encoders of the knn-lm model using a combination of public and private datapoints to minimize the distribution differences between the public data stored in the datastore and the private data used during inference. despite modest improvements from the methods we explored, the mitigation of untargeted attacks remains challenging and there is considerable room for future work. we hope our findings provide insights for practitioners to better understand and mitigate privacy risks in retrieval-based lms. ## background in this section, we first review the key components of knn-lms (section 2.1). then, we discuss the data extraction attacks for language models (section 2.2). these aspects lay a foundation for the subsequent exploration and analysis of privacy risks related to knn-lms. ## problem setup in this section, we formally describe our problem setup (section 3.1) and privacy measurements (section 3.2). we then detail our evaluation setup (section 3.3). ## privacy-utility of knn-lms with a private datastore this section presents our investigation of whether the addition of private data to the retrieval datastore during inference is an effective method for achieving a good trade-off between privacy (mea-sured by metrics defined in section 3.2) and utility (measured by perplexity) in knn-lms. we are particularly interested in three scenarios: utilizing only enc public (the publicly pretrained language model), utilizing only enc private (the model fine-tuned from enc public using private data), and utilizing enc public with d private (the combination of the public model with the private datastore). as shown in when it comes to knn-lms, incorporating a private datastore (d private ) with a public model (enc public ) yields even greater utility compared to relying solely on the fine-tuned model (enc private ). however, this utility improvement also comes at the expense of increased privacy leakage. these findings suggest that the privacy concern stemming from the private datastore outweighs that resulting from the privately fine-tuned model, indicating a lack of robust privacy protection in the design of knn-lms. additionally, we note that the combination of enc private and d private achieves the highest utility but also incurs the highest privacy cost. 5 mitigations against targeted risks our previous findings indicate that the personalization of knn-lms with a private datastore is more susceptible to data extraction attacks compared to fine-tuning a parametric lm with private data. at the same time, leveraging private data offers substantial utility improvements. is there a more effective way to leverage private data in order to achieve a better balance between privacy and utility in knn-lms? in this section we focus on addressing privacy leakage in the context of targeted attacks (see definition in section 3.2), where the private information can be readily detected from text. we consider several approaches to tackle these challenges in section 5.1 and section 5.2, and present the results in section 5.3. we also investigate the effect of hyper-parameters in section 5.4. ## mitigations against untargeted risks in this section, we explore potential methods to mitigate untargeted risks in knn-lms, which is a more challenging setting due to the opacity of the definition of privacy. it is important to note that the methods presented in this study are preliminary attempts, and fully addressing untargeted risks in knn-lms still remains a challenging task. ## conclusion this work presents the first study of privacy risks in retrieval-based language models, specifically focusing on knn-lms. our objective is to investigate designs and training methodologies for knn-lms that strike a better privacy-utility trade-off. there are several conclusions from our investigation. first, our empirical study reveals that incorporating a private datastore in knn-lms leads to increased privacy risks (both targeted and untargeted) compared to parametric language models trained on private data. second, for targeted attacks, our experimental study shows that sanitizing knn-lms to remove private information from both the datastore and encoders, and decoupling the encoders for keys and queries can eliminate the privacy risks without sacrificing utility, achieving perplexity of 16.38 (vs. 16.12). third, for untargeted attacks, our study shows that using a public datastore and training the encoder on a combination of public and private data can reduce privacy risks at the expense of reduced utility by 24.1%, with perplexity of 21.12 (vs. 16.12).