Paper: User Inference Attacks on Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
This paper only explores one simple threshold-based attack. Although the authors show that this simple attack works well, it is better to test other popular attacks, e.g., fitting a NN, in the membership inference literature.

Author response:
**[Other approaches to the attack design]**

State-of-the-art membership inference attacks  (e.g., LiRA by Carlini et al. 2022) require training hundreds of shadow models during the attack stage to learn the distribution of model confidences and create a test statistic. This approach would not be feasible for LLMs due to the high computational complexity incurred in training the shadow models. One of our main requirements is to design an efficient attack statistic that does not train additional shadow models. This motivates the use of the pretrained model as a reference model, resulting in the calibrated loss on the fine-tuned model normalized by the loss on the pretrained model as a test statistic.  

Of course, there could be other attack designs even stronger than ours with similar efficiency, but that would only make privacy leakage a bigger concern. Our attack already indicates an unacceptable level of privacy leakage in LLM finetuning and we hope that future work will address the need for developing and evaluating defenses, as well as potentially designing stronger privacy attacks.

Reviewer comment (Theory):
The authors assume the documents of one user are independent (in Equation 4). This is a reasonable assumption for sample-level inference attacks, but it may not be the right assumption for user inference attacks. Consider emails, the content in email $x_{i+1}$ probably depends on the previous email $x_{i}$.

Author response:
**[Independence of user’s documents]**

This is a great question! In the email case, independence is a modeling assumption that, even if incorrect, makes the problem more computationally tractable. Even with this simplifying assumption, we still get an attack that is relatively strong in most cases. Relaxing this assumption could likely lead to stronger attacks, but those won't necessarily change the conclusions/observations we make in the paper. 

To reiterate, we do not aim to propose the strongest attacks in the paper. Rather, we focus on highlighting that even simple attacks (that are computationally efficient) can already identify privacy leakages in LLM fine-tuning.

Reviewer comment (Evaluation):
The attack success rate on the Enron email dataset significantly surpasses the success rate on other two datasets. Why is this the case? Although each user contributes more documents in the email dataset, Figure 3 shows that attacks against the email dataset are still much easier even when the number of documents is limited.

Author response:
**[Attack success rate on the Enron dataset]**

There are two key reasons why the user inference attack enjoyed high success on the Enron dataset, also highlighted in our analysis in Proposition 1:
- Each user contributes a large amount of data to the training of the model. Figure 6 (right) shows that this allows for a greater attack efficacy. (Note that Figure 3 shows the effect of the attacker’s knowledge: the above result is true even when the attacker’s knowledge is minimal).
- Second, the statistical divergence between the samples contributed by the target user compared to other training data (i.e., the KL divergence between the user’s distribution and other users’ training distributions): The larger the distance, the higher the attack success. There is more separation between users in the Enron dataset compared to ArXiv: for instance,  emails contain more user-level information in the signatures, and salutations. Other details such as addresses are also commonly found in the body of the email (see some examples [here](https://huggingface.co/datasets/aeslc))

Reviewer comment (Experiments):
In the current setup, the attacker is assumed to have access to a fine-tuned language model, as well as a similar reference model, which is the pretrained GPT-Neo in the experiments. If the reference model is less similar (which can be realistic, as attackers may not always have access to the base model), would it be much harder to perform the attack?

Author response:
That is a great question! We have performed an experiment in which we consider an attacker having access to a different pretrained reference model (Table 3 in Appendix). We obtained the best attack success when the attacker model is the same (e.g., 93.1% on Enron with GPT-Neo), but we only see a small reduction in attack success for a different pretrained model (e.g., 87.6% on OPT and 87.2% on GPT-2 for Enron). Fortunately, we found that the attack is not very sensitive to the knowledge of the reference model, and we believe that access to a public pretrained model is a reasonable assumption for the attacker.

Reviewer comment (Experiments):
The arXiv abstracts dataset seems unsuitable for user-level MIAs. Indeed, the authors assign the user who submitted the paper as the “user” (i.e., the author). However, papers have multiple authors which may rewrite or edit the abstract, e.g., the PI. Hence, an abstract could be assigned as belonging to an individual even though it was largely written by someone else. I see two issues with this: (a) Potential authorship overlap between different users' documents, e.g., the same author (PI)’s data could be labeled as belonging to multiple users and (b) The task being evaluated looks like PI/research group inference attack. The “user” terminology is thus misleading, and this should be acknowledged early on.

Author response:
We acknowledge the limitations of the ArXiv dataset you have identified. Despite them, we still believe that analyzing the attack on ArXiv shows some interesting results. 

While the user labeling might not be perfect, the fact that we still have significant privacy leakage suggests that the attack can only get stronger if we have perfect ground truth user labeling and non-overlapping users. Further, our experiments on canary users are not impacted at all by the possible overlap in user labeling, since we create our synthetically-generated canaries to evaluate worst-case privacy leakage. We have added a discussion in the revision in Appendix C. 

At a more practical level, few datasets are large enough with user annotation for evaluating our privacy attack. Thus, despite its flaws, the ArXiv dataset provides a valuable data point (in addition to our other two datasets) in the evaluation of user inference attacks.

Reviewer comment (Evaluation):
What are possible reasons for the difference in attack performance on the three datasets (Figure 2)?

Author response:
Through our analysis in Proposition 1, also confirmed experimentally, we identified several factors that contribute to the attack performance.

- _The number of users_: The larger the total number of users in the fine-tuning dataset, the more challenging the attack becomes. The attack AUROC is lowest on ArXiv which has the largest number of users (16511). 
- _The number of documents contributed by each user_: The Enron dataset has fewer users, each contributing more samples, resulting in higher privacy leakage. This corresponds to the factor $\alpha_u$ in the theoretical analysis and is shown in Figure 6 (right).
- _The statistical divergence between the samples contributed by the target user compared to training data from other users_: The farther away the target user’s distribution is, the higher success the attack has. There is likely more separation between users in the Enron dataset compared to ArXiv for two reasons: (1) emails are more prone to leaking user-level information with signatures, salutations, and addresses being commonly found in the body of the email (see some examples [here](https://huggingface.co/datasets/aeslc)), and (2) The scientific writing style of ArXiv abstracts, which is predominantly impersonal and formal, makes the users less distinct. Moreover, ArXiv might have overlapping users, as you noted.

All these factors result in greater attack success for the Enron dataset.