M o P e: Model Perturbation based Privacy Attacks on Language Models (2023)

Abstract: Recent work has shown that Large Language Models (LLMs) can unintentionally leak sensitive information present in their training data. In this paper, we present Model Perturbations (MoPe), a new method to identify with high confidence if a given text is in the training data of a pre-trained language model, given white-box access to the models parameters. MoPe adds noise to the model in parameter space and measures the drop in log-likelihood at a given point x , a statistic we show approximates the trace of the Hessian matrix with respect to model parameters. Across language models ranging from 70M to 12B parameters, we show that MoPe is more effective than existing loss-based attacks and recently proposed perturbation-based methods. We also examine the role of training point order and model size in attack success, and empirically demonstrate that MoPe accurately approximate the trace of the Hessian in practice. Our results show that the loss of a point alone is insufficient to determine extractability—there are training points we can recover using our method that have average loss. This casts some doubt on prior works that use the loss of a point as evidence of memorization or unlearning.

## introduction over the last few years, large language models or llms have set new standards in performance across a range of tasks in natural language understanding and generation, often with very limited supervision on the task at hand @xcite . as a result, opportunities to use these models in real-world applications proliferate, and companies are rushing to deploy them in applications as diverse as a.i. assisted clinical diagnoses @xcite , nlp tasks in finance @xcite , or an a.i. "love coach" * alphabetical order; equal contribution. ## mia against pre-trained llms in this section we conduct a thorough empirical evaluation of mope θ , focusing on attacks against pre-trained language models from the pythia suite. we show that in terms of auc mope θ significantly outperforms loss thresholding (loss θ ) at model sizes up to 2.8b and can be combined with loss θ to outperform at the 6.9b and 12b models (figure 4 ). we also implement an mia based on detectgpt, which outperforms loss in terms of auc up to size 1.4b, but is consistently worse than mope θ at every model size. our most striking finding is that if we focus on the metric of tprs at low fprs, which state-of-the-art work on mias argue is the most meaningful metric @xcite , mope θ exhibits superior performance at all model sizes (figure 2 ). mope θ is the only attack that is able to convincingly outperform random guessing while driving the attack @xmath0. ## mope θ on mnist in this section we run our mope θ attack on a small network trained to classify images from mnist @xcite , which allows us to evaluate if the mope θ statistic actually approximates the hessian trace in practice as the derivation in section 2.2 suggests. to test this, we sample 5000 training points and 5000 test points from the mnist dataset, a large database of images of handwritten digits @xcite . using a batch size of 8, we train a fully connected 2 layer mlp with 20 and 10 nodes, using the adam optimizer with a learning rate of 0.004 and momentum of 0.9, until we reach 94% train accuracy and 84% test accuracy, a 10 percent train-test gap similar to that observed in pre-trained language models @xcite . in figure 5 we show the distribution of the hessian trace and scaled mope θ evaluated on all training batches. the resulting distributions on training and test points are close, with a correlation between point level traces and scaled mope θ values of 0.42. given the generality of the proposed mope θ attack, it is also natural to ask if mope θ can be applied to other machine learning models. on our mnist network, we run the mope θ attack with 40 perturbations and σ = 0.05, calculating the roc curves in figure 6 using 3000 train and test points. we find that loss θ outperforms mope θ , with mope θ achieving an auc of 0.635 and loss θ achieving an auc of 0.783. while it does not outperform loss θ , mope θ easily beats the random baseline, which warrants further study in settings beyond language models. ## discussion the main finding in this work is that perturbationbased attacks that approximate the curvature in the loss landscape around a given model and point, seem to perform better than attacks that simply threshold the loss. a key nuance here, is that for computational reasons, all of the attacks studied here apply a uniform threshold τ that does not depend on the point x. prior work on mias shows that calibrating the threshold τ x to a specific point x greatly increases attack success, for the obvious reason that certain types of points have higher or lower relative loss values regardless of their inclusion in training. thresholding based on a loss ratio with another language model or approximating the likelihood ratio (lira) can all be viewed as different ways of calibrating an example-specific threshold. in this context, the results in this paper can be viewed as showing that when forced to pick a uniform threshold for an mia score, statistics like mope θ or detectgpt that approximate the local curvature transfer better across different points than the loss. as such, methods that try to efficiently approximate an example-specific threshold, like applying lira or loss-ratio thresholding in the fine-tuning regime where they are more computationally feasible, or applying the very recent work of @xcite that uses quantile regression to approximate a likelihood-style attack without training additional models, in our setting is of great interest. we conjecture that such techniques will be needed to achieve high tprs at very small fprs. another major unresolved question in this work is why mope θ and detectgpt success actually scales inversely with model size, which is likely a property of the method, namely increasing error in the hutchinson trace estimator, rather larger models having improved privacy properties. future work will explore other approximations to the hessian trace that may scale better to large model sizes. finally, future work could use mope θ as part of a training data extraction attack in the style of @xcite @xcite .