User Inference Attacks on Large Language Models (2024)

Abstract: Text written by humans makes up the vast majority of the data used to pre-train and fine-tune large language models (LLMs). Many sources of this data—like code, forum posts, personal websites, and books—are easily attributed to one or a few “users”. In this paper, we ask if it is possible to infer if any of a _user’s_ data was used to train an LLM. Not only would this constitute a breach of privacy, but it would also enable users to detect when their data was used for training. We develop the first effective attacks for _user inference_—at times, with near-perfect success—against LLMs. Our attacks are easy to employ, requiring only black-box access to an LLM and a few samples from the user, which _need not be the ones that were trained on_. We find, both theoretically and empirically, that certain properties make users more susceptible to user inference: being an outlier, having highly correlated examples, and contributing a larger fraction of data. Based on these findings, we identify several methods for mitigating user inference including training with example-level differential privacy, removing within-user duplicate examples, and reducing a user’s contribution to the training data. Though these provide partial mitigation, our work highlights the need to develop methods to fully protect LLMs from user inference.

## introduction llms like @xcite and gemini @xcite have achieved phenomenal success. yet, the datasets used to train llms are often comprised of user-written data. this raises an important question: can we detect (or, infer) if a user's data was used to train a model? if so, this would enable data-owners to detect usage of their data. more concerningly, this would also constitute a breach of user privacy. in this paper, we investigate the latter privacy concern. to do this, we focus on the setting of fine-tuning on domain-specific data, one of the most widely used methods for applying llms to real-world problems @xcite , with several commerical products deployed today using this approach, e.g., github copilot @xcite , gmail smart compose @xcite , and gboard @xcite . in this setting, it is not uncommon for fine-tuning data to include potentially sensitive user data. in this work, we show that user inference (figure 1 ) is a realistic privacy attack for llms finetuned on user data by constructing a simple and practical attack to determine if a user participated in fine-tuning. our attack involves computing a likelihood ratio test statistic normalized relative to a reference model ( §3), which can be performed efficiently even at the llm scale. we empirically study its effectiveness on the gpt-neo family of llms @xcite when fine-tuned on diverse data domains, including emails, social media comments, and news articles ( §4.2). this study gives insight into the various parameters that affect vulnerability to user inference-such as uniqueness of a user's data distribution, amount of finetuning data contributed by a user, and amount of attacker knowledge about a user. notably, our attack requires only a few fresh samples from a user, i.e., not necessarily ones used in model training. this significantly improves on the assumptions of other attacks, like membership inference, in which the attacker is assumed to already have a set of samples that includes some training data @xcite @xcite . beyond this, our attacks also allow us to estimate the privacy leakage of a collection of samples written by a single user that may share characteristics (e.g., writing style, topic, etc.). this can not be quantified by llm privacy attacks like membership inference or extraction attacks @xcite figure 1 : the user inference threat model. an llm is fine-tuned on user-stratified data. the adversary can query the fine-tuned model to compute likelihoods. the adversary can access samples from a user's distribution (different than the user training samples) to compute a likelihood score to determine if the user participated in training. additionally, we evaluate the attack on synthetically generated canary users to characterize the privacy leakage for worst-case users ( §4.3). we show that canary users constructed via minimal modifications to the real users' data increase the attack's effectiveness (in auroc) by up to 40%. this construction indicates that simple features shared across a user's samples like an email signature or a characteristic phrase, can greatly exacerbate the risk of user inference. finally, we evaluate strategies for mitigating user inference, such as limiting the number of samples contributed by each user, removing duplicates within a user's samples, early stopping, gradient clipping, and example-level differential privacy (dp). our results show that duplicates within a user's examples can exacerbate the risk of user inference, but are not necessary for a successful attack. limiting a user's contribution to the finetuning set can be effective but is only feasible for data-rich applications with a large number of users. finally, example-level dp provides some defense but is ultimately designed to protect the privacy of individual examples, rather than users that contribute multiple examples. these results highlight the importance of future work on scalable user-level dp algorithms that can provably mitigate user inference @xcite @xcite . overall, we are the first to study user inference against llms and provide key insights to inform future deployments of llms fine-tuned on user data. ## related work there are many different ml privacy attacks with different objectives @xcite : membership inference attacks determine if a particular data sample was part of a model's training set @xcite @xcite @xcite ; data reconstruction aims to exactly reconstruct the training data of a model, typically for a discriminative model @xcite ; and data extraction attacks aim to extract training data from generative models like llms @xcite @xcite @xcite . @xcite introduce a likelihood ratio-based attack on llms, designed for masked language models, such as bert. @xcite compare the likelihood of a sample against the average likelihood of a set of neighboring samples, and eliminate the assumption of attacker knowledge of the training distribution used in prior works. @xcite study how systems built on llms may amplify membership inference. @xcite use a perplexity-based membership inference attack to extract training data from gpt-2. their attack prompts the llm to generate sequences of text, and then uses membership inference to identify sequences copied from the training set. note that membership inference requires access to exact training samples while user inference does not. ## user inference attacks an autoregressive language model p θ defines a distribution p θ (x t |@xmath0. = (x 1 , . . . , x t-1 ). we focus on fine-tuning, where a pre-trained llm p θ 0 (with initial parameters θ 0 ) is trained on a dataset d ft sampled i.i.d. from a distribution d task . the canonical objective is to minimize the cross entropy of predicting each next token x t given the context @xmath1. thus, the fine-tuned model p θ is trained to maximize the log-likelihood fine-tuning with user-stratified data. much of the data used to fine-tune llms has a user-level structure. for example, emails, messages, and blog posts can reflect the specific characteristics of their author. two text samples from the same user are more likely to be similar to each other than samples across users in terms of language use, vocabulary, context, and topics. to capture user stratification, we model the fine-tuning distribution d task as a mixture the user inference threat model. the task of membership inference assumes that an attacker has access to a text sample x and must determine whether x was a part of the fine-tuning data @xcite @xcite . the user inference threat model relaxes the stringent assumption that the attacker has access to samples from the fine-tuning data. the attacker aims to determine if any data from user u was involved in fine-tuning the model p θ using m i.i.d. samples x ( foot_0 :m) := (x (1) , . . . , x (m) ) ∼ d m u from user u's distribution. crucially, we allow x (i) / ∈ d ft , i.e., the attacker is not assumed to have access to the exact samples of user u that were a part of the fine-tuning set. for instance, if an llm is fine-tuned on user emails, the attacker can reasonably be assumed to have access to some emails from a user, but not necessarily the ones used to fine-tune the model. this is a realistic threat model for llms as it does not require exact knowledge of a user's training set samples, as in membership inference attacks. we adopt the black-box threat model @xcite and assume that the attacker can only query the model's likelihood on a sequence of tokens, but may not know either the model architecture or parameters. 1 following standard practice in membership infer-ence @xcite , we allow the attacker access to a reference model p ref that is similar to the target model p θ but has not been fine-tuned on user u's data. this can be the pre-trained model p θ 0 or another llm. attack strategy. the attacker's task can be formulated as a statistical hypothesis test. letting p u denote the set of models trained on user u's data, the attacker aims to test: there is generally no prescribed recipe to test for such a composite hypothesis. typical attack strategies involve training multiple "shadow" models @xcite ; see §b. this, however, is infeasible at llm scale. the likelihood under the fine-tuned model p θ is a natural test statistic: we might expect p θ (x (i) ) to be high if h 1 is true and low otherwise. unfortunately, this is not always true, even for membership inference. indeed, p θ (x) can be large for x / ∈ d ft for easy-to-predict text sequence x (e.g., generic text using common words), while p θ (x) can be small even if @xmath2. this necessitates calibrating the test using a reference model @xcite . we overcome this difficulty by replacing the attacker's task with surrogate hypotheses that are easier to test efficiently: the neyman-pearson lemma tells us that the likelihood ratio test is the most powerful for testing h 0 vs. h 1 , i.e., it achieves the best true positive rate at any given false positive rate @xcite 3.2.1). this involves constructing a test statistic using the log-likelihood ratio t x (1) , . . . ,x (m) := log p θ (x (1) , . . . , ## experiments in this section, we empirically study the susceptibility of models to user inference attacks, the factors that affect attack performance, and potential mitigation strategies. for this, we fine-tune llms on our user data as this enables us to rigorously test the leakage and because fine-tuning is a common strategy for adapting to user data(see section 1). we believe our results would extend even to models pre-trained on user data but leave such an evaluation to future work.