An Empirical Analysis of Memorization in Fine-tuned Autoregressive Language Models (2022)

Abstract: Large language models are shown to present privacy risks through memorization of training data, andseveral recent works have studied such risks for the pre-training phase. Little attention, however, has been given to the fine-tuning phase and it is not well understood how different fine-tuning methods (such as fine-tuning the full model, the model head, and adapter) compare in terms of memorization risk. This presents increasing concern as the “pre-train and fine-tune” paradigm proliferates. In this paper, we empirically study memorization of fine-tuning methods using membership inference and extraction attacks, and show that their susceptibility to attacks is very different. We observe that fine-tuning the head of the model has the highest susceptibility to attacks, whereas fine-tuning smaller adapters appears to be less vulnerable to known extraction attacks.

## introduction transformer-based language models have become the models of choice for many nlp tasks, such as email, text and code auto-completion, question answering and sentiment analysis @xcite . these models are commonly trained using the pre-train and fine-tune paradigm, where they are first trained (pre-trained) on a large, general domain dataset (in the order of hundreds of gigabytes), and then fine-tuned on smaller, task-specific datasets to adapt the model to a specific domain @xcite @xcite . several works have demonstrated that such large models have a high capacity for memorizing training samples during pre-training and are therefore highly susceptible to membership inference and data extraction attacks @xcite @xcite . more specifically, @xcite and @xcite have mounted such attacks on pre-trained language models and shown the severity of this issue by extracting complete training sequences and inferring membership of a large fraction of the training samples. these works focused on memorization during pre-training, but scant attention has been given to fine-tuning. in this work, we focus on different fine-tuning methods and their propensity for memorization of training samples. fine-tuning data is actually of higher concern than pre-training data, since most pre-training datasets are large public corpora @xcite with limited privacy concerns @xcite , while fine-tuning sets are small, targeted, and potentially very private @xcite . further, pre-training generally happens only a few times (as it needs resources that are usually only available to large companies @xcite ) while fine-tuning is increasingly the dominant way that end-users fit models. given the size of these large language models, fine-tuning all the model parameters can be compute and memory-intensive @xcite @xcite . as a result, recent works have proposed new parameter efficient fine-tuning methods that update only a subset of the model's parameters @xcite @xcite . in this paper, we focus on studying memorization of three popular fine-tuning methods: (1) fine-tuning all model parameters (2) fine-tuning the head, which is commonly used by practitioners and involves updating only the last layer of the model which produces the logits, and (3) fine-tuning adapters @xcite , which are small bottleneck modules inserted within transformer blocks. for measuring memorization, we use two proxy metrics: (a) recall of a reference-based membership inference attack (mia) @xcite and (b) exposure @xcite , which measures how susceptible the model is to a sample extraction attack which tries to reconstruct samples from training data. we run our experiments on the wikipedia @xcite , penn treebank @xcite and enron emails @xcite datasets, for the task of autoregressive language modeling. we selected wikipedia and penn treebank as they are most commonly used for fine-tuning, and enron since it is a dataset of emails representing private tuning data. figure 1 shows how we conceptually identify three distinct phases in the fine-tuning process, based on validation perplexity (generalization) and membership inference attack recall (memorization). each point shows these metrics at the end of a training epoch. for all fine-tuning methods, we observe that in a memorization only phase, the model memorizes more and more, without overfitting or generalizing better (figure 2 ). in terms of different fine-tuning methods, we find that the common practice of fine-tuning only the head of a model has the highest memorization (by a large margin) for the same level of perplexity, among different fine-tuning methods -even full fine-tuning, which updates more parameters. this result is surprising and potentially indicates that only tuning parameters higher in the model architecture (closer to the output) exacerbates the memorization and increases the leakage based on our metrics. we also show that fine-tuning the full model and small adapters are on the paretofrontier in terms of the attack recall vs. validation perplexity graph. code and instructions to reproduce our results are available at https://github. com/mireshghallah/ft-memorization/ . ## model fine-tuning we focus on two main fine tuning methods, for finetuning gpt-2 with next word prediction objective: ## measuring memorization to measure memorization, we use two metrics: membership inference attack recall and exposure. membership inference (mia recall). we use the percentage of training samples that are correctly classified as training members (out of a pool of training and validation samples) by the reference-based attack proposed in @xcite and @xcite as a proxy metric of memorization. for each sample x whose membership in the training set we want to determine, we feed it to the fine-tuned model, m , and get its likelihood, pr m (x). we also feed it to a reference model, r, a pre-trained model that is not fine-tuned, and get the probability pr r (x). we then use lr( pr m (x) , the likelihood ratio, to determine if x is a training sample. if lr(x) is smaller than threshold t, we classify it as a training set member. otherwise, we classify it as a non-member. we determine the threshold t by calculating lr(s) for all s in the validation set, and then choose the threshold to be the highest threshold such that the false positive rate (over training and validation members) would not exceed 10%. the higher the recall of this attack is, the higher the leakage of the model. exposure. as a second measure of memorization, we use the exposure metric from @xcite which inserts a secret (canary) of a certain format into the training data and calculates its vulnerability to extraction. exposure is defined as the negative log-rank of the inserted secret in terms of model probability, among all other possible sequences of the same length. this quantity is then added to a constant to ensure the exposure is always positive. models. we study memorization in fine-tuning huggingface's pre-trained gpt-2 on the datasets mentioned above. we use a pre-trained but not fine-tuned gpt-2 as the reference model for our membership inference attack. we use the adapter hub's implementation of the pfeiffer architecture, with reduction factors 2 and 16 @xcite . metrics. we use validation perplexity as a metric for the performance of the model, where lower perplexity is better. we evaluate memorization at each epoch using the mia recall and exposure metrics described in section 3. the experiments in the paper are all repeated 3 times and we report the average values for each metric. hyperparameters and result presentation. we run optimization for each fine-tuning method for 20 epochs, and perform evaluation of the mentioned metrics at the end of each epoch. we experiment with the three learning rates 2 × 10 -5 ,10 -4 ,10 -3 , and present the results for all of them. therefore, each graph would have an overall of 20×3 points, for each fine-tuning method, unless the point is outside the plot range. for the reported exposure numbers, we selected points close to the pareto frontier to present in ## results in this section we discuss our experimental results comparing the privacy-utility trends for different fine-tuning methods. we refer to the naming convention shown in figure 1 and provide extended graphs for each experiment in appendix a.3. we also present additional experiments where we train the model from scratch (instead of fine-tuning pre-trained models), fine-tune different model architectures, and study the generalization gap in appendix a.1. ## conclusion when fine-tuning is done using sensitive training data, it is important to not just consider the cost and utility of fine-tuning methods but to also be aware that they may have different risks in terms of privacy. our experiments show that the common practice of fine-tuning only the head of a model has the highest memorization (by a large margin). full model fine-tuning and adapter tuning, however, are both on the pareto-frontier in terms of attack recall vs. validation perplexity, suggesting that they are more suitable when privacy is a concern.