Membership Inference Attacks against Language Models via Neighbourhood Comparison (2023)

Abstract: Membership Inference attacks (MIAs) aim to predict whether a data sample was present in the training data of a machine learning model or not, and are widely used for assessing the privacy risks of language models. Most existing attacks rely on the observation that models tend toassign higher probabilities to their training samples than non-training points. However, simple thresholding of the model score in isolation tends to lead to high false-positive rates as it does not account for the intrinsic complexity of a sample. Recent work has demonstrated that reference-based attacks which compare model scores to those obtained from a reference model trained on similar data can substantially improve the performance of MIAs.However, in order to train reference models, attacks of this kind make the strong and arguably unrealistic assumption that an adversary has access to samples closely resembling the original training data. Therefore, we investigate their performance in more realistic scenarios and find that they are highly fragile in relation to the data distribution used to train reference models. To investigate whether this fragility provides a layer of safety, we propose and evaluate neighbourhood attacks, which compare model scores for a given sample to scores of synthetically generated neighbour texts and therefore eliminate the need for access to the training data distribution. We show that, in addition to being competitive with reference-based attacks that have perfect knowledge about the training data distribution, our attack clearly outperforms existing reference-free attacks as well as reference-based attacks with imperfect knowledge, which demonstrates the need for a reevaluation of the threat model of adversarial attacks.

## introduction the public release and deployment of machine learning models trained on potentially sensitive user data introduces a variety of privacy risks: while embedding models have been shown to leak personal attributes of their data @xcite , generative language models are capable of generating verbatim repetitions of their training data and therefore exposing sensitive strings such as names, phone numbers or emailaddresses @xcite . another source of risk arises from membership inference attacks (mias) @xcite , which enable adversaries to classify whether a given data sample was present in a target model's training data or not. due to their simplicity and the fact that mias are an important component of more sophisticated attacks such as extraction attacks @xcite , they have become one of the most widely used tools to evaluate data leakage and empirically study the privacy of machine learning models @xcite . typically, membership inference attacks exploit models' tendency to overfit their training data and therefore exhibit lower loss values for training members @xcite . a highly simple and commonly used baseline attack is therefore the loss attack @xcite , which classifies samples as training members if their loss values are below a certain threshold. while attacks of this kind do generally reap high accuracies, @xcite point out a significant flaw: good accuracies for attacks of this kind are primarily a result of their ability to identify non-members rather than training data members, which does arguably not pose important privacy risks. this shortcoming can be attributed to the fact that certain samples such as repetitive or very simple short sentences are naturally assigned higher probabilities than others @xcite , and the influence of this aspect on the obtained model score largely outweighs the influence of a model's tendency to overfit its training samples @xcite . to account for this, previous work has introduced the idea of difficulty calibration mechanisms @xcite stocks fall to end wall street's worst year since 2008, s&p 500 finishes 2022 down nearly 20% securities fall to end wall street's worst year after 2008, s&p 500 finishes 2022 down almost 20% stocks fall to end wall street's worst year since 2009, s&p 500 ends 2022 down nearly 20% proposal model member non-member neighborhood comparison figure 1 : overview of our attack: given a target sample x, we use a pretrained masked language model to generate highly similar neighbour sentences through word replacements. consequently, we compare our neighbours' losses and those of the original sample under the target model by computing their difference. as our neighbours are highly similar to the target sequence, we expect their losses to be approximately equal to the target model and only to be lower if the target sequence was a sample of the model's training data. in this case, the difference should be below our threshold value γ. in practice, difficulty calibration is mostly realized through likelihood ratio attacks (lira), which measure the difficulty of a target point by feeding it to reference models that help provide a perspective into how likely that target point is in the given domain @xcite @xcite . in order to train such reference models, liras assume that an adversary has knowledge about the distribution of the target model's training data and access to a sufficient amount of samples from it. we argue that this is a highly optimistic and in many cases unrealistic assumption: as also pointed out by @xcite , in applications in which we care about privacy and protecting our models from leaking data (e.g. in the medical domain), high-quality, public in-domain data may not be available, which renders reference-based attacks ineffective. therefore, we aim to design an attack which does not require any additional data: for the design of our proposed neighborhood attack, we build on the intuition of using references to help us infer membership, but instead of using reference models, we use neighboring samples, which are textual samples crafted through data augmentations such as word replacements to be non-training members that are as similar as possible to the tar-get point and therefore practically interchangeable with it in almost any context. with the intuition that neighbors should be assigned equal probabilities as the original sample under any plausible textual probability distribution, we then compare the model scores of all these neighboring points to that of the target point and classify its membership based on their difference. similar to liras, we hypothesize that if the model score of the target data is similar to the crafted neighbors, then they are all plausible points from the distribution and the target point is not a member of the training set. however, if a sample is much more likely under the target model's distribution than its neighbors, we infer that this could only be a result of overfitting and therefore the sample must be a part of the model's training data. we conduct extensive experiments measuring the performance of our proposed neighborhood attack, and particularly compare it to referencebased attacks with various different assumptions about knowledge of the target distribution and access to additional data. concretely, amongst other experiments, we simulate real-world referencebased attacks by training reference models on external datasets from the same domain as the target model's training data. we find that neighbourhood attacks outperform liras with more realistic assumptions about the quality of accessible data by up to 100%, and even show competitive performance when we assume that an attacker has perfect knowledge about the target distribution and access to a large amount of high-quality samples from it. ## membership inference attacks via neighbourhood comparison in this section, we provide a detailed description of our attack, starting with the general idea of comparing neighbouring samples and following with a technical description of how to generate such neighbors. ## experimental setup we evaluate the performance of our attack as well as reference-free and reference-based baseline attacks against large autoregressive models trained with the classical language modeling objective. particularly, we use the base version of gpt-2 (radford et al., 2019) as our target model. ## results in this section, we report our main results and perform additional experiments investigating the impact of reference model performance on the success of reference-based attacks as well as several ablation studies. following @xcite , we report attack performances in terms of their true positive rates (tpr) under very low false positive rates (fpr) by adjusting the threshold value γ. concretely, we choose 1%, 0.1% and 0.01% as our target fpr values. ## defending against neighbourhood attacks due to the privacy risks that emerge from the possibility of membership inference and data extraction attacks, the research community is actively working on defenses to protect models. beyond approaches such as confidence score perturbation @xcite and specific regularization techniques @xcite showing good empirical performance, differentially private model training is one of the most well known defense techniques offering mathematical privacy guarantees: dp-sgd @xcite @xcite , which uses differential privacy @xcite to bound the influence that a single training sample can have on the resulting model and has been shown to successfully protect models against membership inference attacks @xcite and has recently also successfully been applied to training language models @xcite @xcite . to test the effectiveness of differential privacy as a defense against neighbourhood attacks, we follow @xcite and train our target model gpt-2 in a differentially private manner on ag news, where our attack performed the best. the results can be seen in ## related work mias have first been proposed by @xcite and continue to remain a topic of interest for the machine learning community. while many attacks, such as ours, assume to only have access to model confidence or loss scores @xcite @xcite , others exploit additional information such as model parameters @xcite or training loss trajectories @xcite . finally, some researchers have also attempted to perform membership inference attacks given only hard labels without confidence scores @xcite . notably, the attack proposed by @xcite is probably closest to our work as it tries to obtain information about a sample's membership by flipping its predicted labels through small data augmentations such as rotations to image data. to the best of our knowledge, we are the first to apply data augmentations of this kind for text-based attacks. membership inference attacks in nlp specifically in nlp, membership inference attacks are an important component of language model extraction attacks @xcite . further studies of interest include work by @xcite , which studies membership inference attacks in machine translation, as well as work by @xcite , which investigates likelihood ratio attacks for masked language models. specifically for language models, a large body of work also studies the related phenomenon of memorization @xcite @xcite , which enables membership inference and data extraction attacks in the first place. machine-generated text detection due to the increasing use of tools like chatgpt as writing assistants, the field of machine-generated text detection has become of high interest within the research community and is being studied extensively @xcite @xcite . notably, @xcite propose detectgpt, which works similarly to our attack as it compares the likelihood of a given sample under the target model to the likelihood of perturbed samples and hypothesizes that the likelihood of perturbations is smaller than that of texts the model has generated itself. ## conclusion and future work in this paper, we have made two key contributions: first, we thoroughly investigated the assumption of access to in-domain data for reference-based membership inference attacks: in our experiments, we have found that likelihood ratio attacks, the most common form of reference-based attacks, are highly fragile to the quality of their reference models and therefore require attackers to have access to high-quality training data for those. given that specifically in privacy-sensitive settings where publicly available data is scarce, this is not always a realistic assumption, we proposed that the design of reference-free attacks would simulate the behavior of attackers more accurately. thus, we introduced neighborhood attacks, which calibrate the loss scores of a target samples using loss scores of plausible neighboring textual samples generated through word replacements, and therefore eliminate the need for reference trained on in-domain data. we have found that under realistic assumptions about an attacker's access to training data, our attack consistently outperforms reference-based attacks. furthermore, when an attacker has perfect knowledge about the training data, our attack still shows competitive performance with referencebased attacks. we hereby further demonstrated the privacy risks associated with the deployment of language models and therefore the need for effective defense mechanisms. future work could extend our attack to other modalities, such as visual or audio data, or explore our attack to improve extraction attacks against language models.