Noisy Neighbors: Efficient membership inference attacks against LLM s (2024)

Abstract: The potential of transformer-based LLMs risks being hindered by privacy concerns due to their reliance on extensive datasets, possibly including sensitive information. Regulatory measures like GDPR and CCPA call for using robust auditing tools to address potential privacy issues, with Membership Inference Attacks (MIA) being the primary method for assessing LLMs’ privacy risks. Differently from traditional MIA approaches, often requiring computationally intensive training of additional models, this paper introduces an efficient methodology that generates noisy neighbors for a target sample by adding stochastic noise in the embedding space, requiring operating the target model in inference mode only. Our findings demonstrate that this approach closely matches the effectiveness of employing shadow models, showing its usability in practical privacy auditing scenarios.

## introduction advancements in natural language processing @xcite have made large language models (llms) @xcite essential for many text tasks. however, llms face issues like biases @xcite , privacy breaches @xcite , and vulnerabilities @xcite , underscoring the importance of protecting user privacy. the use of large datasets including personal information, has raised privacy concerns, leading to regulations such as gdpr @xcite and @xcite . membership inference attacks (mia) @xcite are effective auditing tools aiming at determining if a specific data point was used in an llm's training dataset by analyzing its output. such attacks highlight potential privacy breaches, relying on models' tendency to overfit to familiar data @xcite . by employing calibration strategies and training shadow models, the accuracy of mias can be improved, although challenges such as computational demands and limitations in effectiveness when deviating from training distribution assumptions persist. in this paper, we contribute to this field by: i) exploring membership inference attacks from the standpoint of a privacy auditor, ii) introducing a computationally efficient calibration strategy that sidesteps training shadow models, and iii) empirically assessing its potential in replacing other prevalent strategies. ## background llms generate a probability distribution over their vocabulary based on a tokenized input sequence converted into numerical inputs through an embedding layer. this layer maps tokens to a dense representation, which can be learned during training @xcite or derived from public word embeddings @xcite . for a model f with input sequence x, we define p[w|x] = f w (x) as the conditional probability that the token following x is w. llms are typically trained on large datasets of text to minimize a measure of surprise in seeing the next token, called perplexity. for a sequence x, it is defined as the average negative log-likelihood of its tokens: with |x| the number of tokens in the sequence. membership inference attacks @xcite @xcite aim to determine whether a particular data record x was used in the training dataset d train of a machine learning model. these methods leverage model outputs like confidence scores or prediction probabilities to compute a score for the targeted sample. for llms, the typical assumption is to grant the adversary access to the output probabilities f (x), which may be used to estimate the perplexity on the targeted samples as a score. given a sample x, the goal of the attacker is to learn a thresholding classifier to output 1 when the perplexity is lower than a certain value γ: notably, dp quantifies the worst-case scenario of the privacy risk, so it is a fundamental tool in privacy assessment. from the performance of the thresholding classifier ãγ (f, x) one can obtain a lower bound to the empirical ε-dp @xcite : with tpr and fpr being, respectively, the true and false positive rates, given a certain threshold. ## related works privacy attacks against language models is an active area of research and different refinements have been proposed. some works have focused on an attacker where data poisoning is allowed, granting the adversary write access to the training dataset, to increase memorization @xcite or in general to induce malicious behaviours @xcite @xcite @xcite and improve property inference attacks @xcite . other works have adopted similar techniques to achieve actual training data extraction from the training set, with only query access to the trained model @xcite . in the context of mias with query access to the target model, most research focused on strategies to improve the calibration of the per-sample scores, i.e. techniques to improve the precision and recall in distinguishing members from non-members of the training set. in principle, if we can assert that an out-of-distribution non-member of the training set will induce a high perplexity in a target llm, there are a number of scenarios where the distinction is not as clear cut, and a thresholding classifier essentially ends up distinguishing between in-distribution from out-of-distribution samples. a refined mia then employs calibration strategies to tune the scoring function based on the difficulty of classifying the specific sample, as in @xcite . thus, a relative membership score is obtained by comparing f (x) with one of two results based on whether the adversary is assumed to have access to neighboring models f (x) @xcite or neighboring samples f (x) @xcite . the new classifier becomes: ## method the intuition behind noisy neighbors is that, fixed a distance from a sample, the target model will show a larger difference in perplexity between a training sample and its neighbors than between a test sample and its neighbors. thus, if we describe a language model as a composition of layers f (x) = g(e(x)) where e is an embedding layer and g is the rest of the network, one can artificially create neighbors in the n-dimensional embedding space by directly injecting random noise at the output of e(x). in particular, if we create noisy neighbors by injecting gaussian noise such that ## experiments to validate the noisy neighbor strategy in implementing a calibrated mia, we run a series of preliminary experiments on an llm to gauge the risk of memorization of training data. the chosen architecture is gpt-2 small @xcite to compromise learning capacity with memory and computational footprint at about 1.5 billion parameters, especially considering that competing strategies require training multiple llms from scratch. the model was pre-trained on open-webtext @xcite , an open reproduction of the undisclosed webtext in @xcite . the model was then fine-tuned on 60% of the full wikitext corpus @xcite , a large collection of wikipedia articles. the same data split was then partitioned in 10 subsets used to train 10 shadow models for score calibration, as in @xcite . note that wikipedia articles are filtered out of the open-webtext corpus, to avoid data leakage in common benchmarks, such as ours. the remaining portion of 40% of wikitext is thus used as source of nonmember, 126-token long samples to analyze the performance of the attack. we generate only 10 synthetic neighbors for each sample. given a sample and its score, the thresholding classifier yields a binary decision on whether it was part of the training dataset or not. to determine how good the best possible classifier may be, we need to evaluate its accuracy at different thresholds. as it is common for binary classification problems, though, the ac-curacy does not give a complete picture of the confidence at which the classifier is able to tell apart members and non-members of the dataset. thus figure 2a shows the complete range of tprs versus fprs for the three main strategies we included in this comparison: score by perplexity (loss), by shadow model calibration (shadow), and by noisy neighbor (noisy) calibration. we have opted not to incorporate the lowercasing strategy @xcite and the semantic neighbor approach @xcite in our study. these methods have, however, shown lower performance levels when information about the training data distribution is accessible, which is contemplated from the auditor point of view. additionally, we faced challenges replicating some results from @xcite , possibly due to limitations in the synonym generation technique described in @xcite . figure 2a also notes the area under the curve (auc), which for noisy and shadow amounts to 0.727 and 0.753 respectively, thus showing a discrepancy of only ∼ 3.4%. the auc is an important metric for binary classifiers as it abstracts from the specific threshold, thus giving an average-case idea of the strength of the attacker. still, as highlighted in @xcite , special care should be given to what happens at low fprs, that is when the attacker can confidently recognize members of the training set. this is what figure 2b focuses on, again showing a strong overlap of the shadow and noisy strategies. following equation 4, we also provide the perspective of empirical dp, as the privacy community pushes to adopt this framework to comply to regulatory frameworks such as the gdpr @xcite . empirical dp measures the extent to which individual data points can be inferred or re-identified from the output of the system, and contrary to dp, it is a post-hoc measurement, not an a-priori guarantee. figure 3 reports the results, where we see a strong consistency between the noisy and shadow strategies, especially for fprs lower than 10 -2 . ## limitations the effectiveness of the noisy neighbors method depends on assumptions that may not apply universally across models or datasets. its success also relies on specific noise parameters, potentially limiting its generalizability. despite being computationally more efficient than shadow model methods, it still requires significant computational resources. ## conclusion this work set out to elaborate a strategy for membership inference attacks. differently from prior research focusing on improving the strength of the attacker, we develop a technique trying to achieve a similar efficacy, while reducing the computational burden for an auditor trying to assess the privacy risk of exposing the query access to a trained llm. we propose the use of noise injection in the embedding space of the llm to create synthetic neighbors of the targeted sample, to shift the comparison from the perplexity scored by different models on one sample, to the comparison of different samples by the same model. this approach allows to only use the model in inference mode, thus inherently reducing the time and cost of running an mia. with a number of experiments we assess how our strategy results converge to the results of using shadow models, showing a remarkable alignment.