Paper: Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks (ICLR-2024)

Reviewer comment (Experiments):
For me it is not quite clear how the proposed defense methods are combined with the model editing techniques. In the experiments, ROME and MEMIT are used as model editing techniques. However, it is not mentioned how the different optimization objectives for the proposed defense techniques are used in combination with these methods. This could be formulated a bit clearer.

Author response:
> It is not mentioned how the different optimization objectives for the proposed defense techniques are used in combination with these methods. 

We apologize for confusion about this. To clarify, the defense methods are objectives, and the underlying optimization method is ROME/MEMIT i.e. the module in the model which is being edited is decided by the optimization method and the objective for which the edit is optimized is decided by the defense method. So in Table 1, we list ROME and MEMIT as the optimization methods, and the +X is the defense objective that is optimized by the optimization method.

We will be able to make this clearer with extra space in a camera ready version of the paper.

Reviewer comment (Presentation):
It would be easier to read if the paragraph in 4.2 also had a bold subheading with the name of the attack, instead of putting the name in the heading of the section. This would make it easier for readers to spot the names of the different attacks.

Author response:
> paragraph in 4.2 also had a bold subheading 

Thanks. We’ll make it bold in the final version.

Reviewer comment (Experiments):
**Q1:** Why use only single-token answers? If I understand this correctly, this way it is not possible to extract answers which consist of multiple tokens, correct? Is it possible to modify your approach to make this work for multi-token answers?

Author response:
> Is it possible to modify your approach to make this work for multi-token answers?



On the attack side, yes it would be possible to extend our attacks to the multi-token setting. For the whitebox attacks, the logit lens could be readily applied in a manner similar to autoregressive decoding to generate outputs or rank-order a set of plausible options (i.e. doing autoregressive decoding but decoding from an intermediate layer’s hidden states rather than running the full forward pass). On the blackbox side, generating multiple tokens is straightforward. 

We note that the main reason for this simplifying assumption is that it makes it easier to rank order and evaluate the attack success. If we’re deciding whether the ground truth answer is in the candidate set, it's easier if there are 50000 possible atomic answers to questions, and we can directly check whether the exact answer is in the candidate set. If the space of answers is really big, that basically turns into a complicated NLG evaluation problem (is the true answer similar enough to some candidate in the candidate set?), so we wanted to simplify the problem in our work.

Reviewer comment (Reproducibility):
**Q2:** I don't quite understand how the Head Projection defense works. What exactly are the values D_answer and D_k? As far as I understand, D_answer is a probability distribution, while D_k is a single value? Could you clarify the loss function and what exactly is optimized for this defense?

Author response:
> I don't quite understand how the Head Projection defense works. What exactly are the values D_answer and D_k? As far as I understand, D_answer is a probability distribution, while D_k is a single value? Could you clarify the loss function and what exactly is optimized for this defense?

The objective of HP defense is to prevent the deleted answer from appearing in the top-k elements of the logit lens distributions across a set of layers denoted as L. This is done to defend against the HP attack (refer to Fig. 3). To achieve this goal, we introduce a max-margin loss in each relevant distribution. Here, D(ℓ) represents the logit lens distribution at layer ℓ, D(ℓ)_answer denotes the probability of the original answer's logit lens, and D(ℓ)_k represents the k-th top probability in D(ℓ). The given margin loss objective forces the probability of answer (D(ℓ)_answer) to be less than the probability of the kth highest-probabiity token (D(ℓ)_k) by at least a margin of m which makes the answer less detectable in the HP attack (see Sec 5, page number 7).

Reviewer comment (Novelty):
The findings of the paper are somewhat expected, as the model editing techniques being evaluated are heuristics and don’t come with formal guarantees of robustness against attacks. Similarly, it is expected that a defense designed to mitigate a specific attack is robust against that attack but not necessarily against other attacks.

Author response:
> The findings of the paper are somewhat expected

True, they are expected but they are also important and necessary since there has been a lot of excitement around editing methods as well as many concerns about privacy and memorized information in language models. This is a timely policy issue in society, and it’s not clear that we have the technical tools to address the problem at the moment.

At the same time, one surprising result is that we do see that HP Def works pretty well against PD attack, which it is not prepared for. In the paper, we suggest that there is no single defense method which works against all attacks it could face, but we do see some surprisingly positive results within the whitebox attack category.

Additionally, some readers will probably be surprised to see that “deleted” information is still recoverable from model hidden states in the first place. This has not been shown before empirically.

Reviewer comment (Evaluation):
Insufficient analysis of results. I was left wondering what are the technical differences between ROME and MEMIT and whether this could explain some of the differences in the results.

Author response:
> I was left wondering what are the technical differences between ROME and MEMIT and whether this could explain some of the differences in the results.

Thanks for pointing this out, we actually did discuss this in an earlier, longer version of the paper but unfortunately our ability to comment on every difference in methods was constrained by limited space in the ICLR submission. We are happy to further discuss these kinds of details (see below) in an extra camera ready page.

Regarding the differences between ROME and MEMIT: Attack success against MEMIT is generally higher than against ROME, and our best defense methods (like Max-Ent) achieve smaller improvements over baselines with MEMIT than with ROME. This could suggest that distributing editing updates across multiple layers rather than a single layer, as MEMIT does, increases vulnerability to attacks and makes defense more challenging. However, MEMIT performs more favorably than ROME on the Rewrite Score and ∆-Acc metrics, meaning this difference in the methods could be due to a difference in how strongly the edits applied to the model.

Reviewer comment (Writing):
Confusing usage of the term “sensitive”: The definition used by the authors includes “toxic” information: “Models can also generate text reflecting beliefs that cause direct psychological harm to people (i.e. toxic generated text) (Kenton et al., 2021). Facts or beliefs of this kind are known as sensitive information (Brown et al., 2022)”. I'm pretty sure that’s not how Brown et al. use the word sensitive. In the privacy domain, “sensitive information” refers to protected characteristics about an individual (https://commission.europa.eu/law/law-topic/data-protection/reform/rules-business-and-organisations/legal-grounds-processing-data/sensitive-data/what-personal-data-considered-sensitive_en) or is sometimes used colloquially to denote private information that a model or system should not disclose. To avoid confusion, I would suggest using a different term that explicitly refers to “toxic” information.

Author response:
> Confusing usage of the term “sensitive”

Thanks for pointing this out, we will clarify the different meanings of sensitive vs. toxic (with appropriate citations to broader socio-technical work) in the final version of the paper.

Reviewer comment (Experiments):
What are the results of attacking the MEMIT method without any defenses (i.e., the equivalent of Figure 4)? The paper’s second claim is that model editing methods fail to delete information; basing it on only one of two editing methods studied in the paper weakens the claim and raises questions.

Author response:
>  The paper’s second claim is that model editing methods fail to delete information; basing it on only one of two editing methods studied in the paper weakens the claim and raises questions.

It is true that Sec. 7.1 focuses on ROME in order to make the point that SOTA model editing methods are vulnerable to information attacks. However, we believe this result would only hold more strongly for the MEMIT and constrained finetuning methods, since we see in results in 7.2 that MEMIT with defenses is more easily attacked compared to ROME for almost all defenses. So we expect the trend to be similar for MEMIT but with even higher attack success rates (see also previous answer on ROME vs MEMIT).

Reviewer comment (Theory):
Do the authors think that model editing is technically possible with formal guarantees against attacks and what are, in the authors’ opinion, promising directions for future work in this domain?

Author response:
> Do the authors think that model editing is technically possible with formal guarantees against attacks and what are, in the authors’ opinion, promising directions for future work in this domain?

This is a good question! To our understanding of the adversarial robustness literature, formal guarantees often (1) make strong assumptions about the form of the attack, and (2) are often limited to data domains like images rather than text. It seems to us that the problem of information extraction attacks on LMs is still a sufficiently new problem that additional empirical and analysis work needs to be conducted first before a stronger theoretical understanding is achievable, although clearly formal guarantees of robustness are very relevant and valuable. Interestingly, there is also the opinion that as models get better (in both vision and language settings), eventually adversarial vulnerability will go away on its own. For instance, you might think that paraphrase attacks cease to be effective once models truly understand that we don’t want them to reveal people’s private information no matter how they are asked about it. Overall, this is still a fledgling area of research, but since it is very practically important it demands much more attention.

Reviewer comment (Experiments):
**Lack of experiments** - The paper could be more comprehensive if more scenarios were presented in the paper, e.g. combining attacks (to maximize extraction under a budget) or combining defenses (to explore how much risks could be minimized in practice). I find it hard to draw actionable conclusions from the results, and further discussion and variety of results may help justify the strength of model editing as the de-facto paradigm for unlearning.

Author response:
I find it hard to draw actionable conclusions from the results, and further discussion and variety of results may help justify the strength of model editing as the de-facto paradigm for unlearning.

We summarize the actionable conclusions below based on the results in Sec 7.2:  
- *Whitebox attacks*. Max Entropy defense helps best against white-box attacks, which is a positive and directly implementable in practical efforts in sensitive information deletion.  
- *Blackbox attacks*. Although Error injection and Fact erasure defenses are somewhat effective against blackbox attacks, the lack of robustness of the deletion methods to paraphrases of the input is evident from the high attack success rates. Here we note that, like past work with jailbreaks for LLMs, sensitive information deletion is another example of a security issue with LLMs that we are highlighting. They are just not secure against paraphrase attacks.

Reviewer comment (Experiments):
**No comparison to prior works** - I’m concerned about the lack of comparison to prior works. While the paper claims that prior works is not applicable due to focus on removal of influence of a pair $(x,y)$, the problem formalized in the paper (based on the ASR metric) is exactly the recovery of a single token label $y_i$ given a specific prompt $x_i$. Hence, I’m confused why it’s not possible to compare with prior approaches in approximate unlearning.

Author response:
No comparison to prior works - I’m concerned about the lack of comparison to prior works. 

In essence, constrained finetuning is similar to the most common approach for unlearning, which is EWC (Elastic Weight Consolidation). EWC is being used as recently as 2022 in unlearning papers [1]. In the original draft of the paper, we distinguished model editing from unlearning based on whether the goal was to unlearn y|x vs. (x,y). However, the optimization approaches they employ are actually quite similar (specialized and constrained forms of finetuning), and unlearning methods are readily adapted to the problem of sensitive information deletion. 

In summary, we believe our constrained finetuning approach (See B.3 and Table 5 in appendix) is already a very comparable baseline to existing unlearning approaches like EWC, and since ROME and MEMIT are known to far outperform constrained finetuning, we believe that we have covered all the relevant baselines and are using the best possible methods for the problem at hand. Although if you know of another unlearning approach that is more qualitatively distinct, we would also be happy to add that.

[1] Tanno, Ryutaro, et al. "Repairing Neural Networks by Leaving the Right Past Behind." Advances in Neural Information Processing Systems 35 (2022): 13132-13145.

Reviewer comment (Experiments):
I’m a bit confused by the distinction between the password attempts and parallel extractions scenarios in the context of the paper. Is the distinction here meant to be that information gained in the recovery of a previous attempt may be utilized to aid recovery of the next attempt? If so, I do not understand how the attacks as described utilize information in this way (i.e. it seems all attack methods generate a candidate set for a budget $B$ in parallel anyways). Is such an attack possible?

Author response:
I’m a bit confused by the distinction between the password attempts and parallel extractions scenarios in the context of the paper. Is the distinction here meant to be that information gained in the recovery of a previous attempt may be utilized to aid recovery of the next attempt? 

To clarify, we tried to give multiple plausible threat models that all map onto the same final Attack Success Metric, in order to more robustly motivate the attack metric we use as being widely relevant to real-world attack scenarios. All experiments utilize the same Attack Success Metric, and there is no “information gained” between different attack attempts in our experiments. 

If so, I do not understand how the attacks as described utilize information in this way (i.e. it seems all attack methods generate a candidate set for a budget B in parallel anyways). Is such an attack possible?

To answer your question (even though it is not the case that our attacks utilize information from past attempts), it would be possible for an attack to leverage past attempts if there was some prior knowledge about candidate correctness being correlated or not. For instance, if it was known that a personal phone number began with either 919 or 336, then one could rule out half the possible candidates by first checking a candidate beginning with 919.

Reviewer comment (Experiments):
In Sec. 2 “Attacking LLMs for Sensitive Information”, the paper claims that the described method does not assume access to the exact text used during pre-training of the model. However, the experiments do assume such access, as even in the rephrasing attack, the ground-truth prompt is perturbed by a rephrasing model. Is it possible to run the rephrasing attack without the ground-truth prompt?

Author response:
the paper claims that the described method does not assume access to the exact text used during pre-training of the model. However, the experiments do assume such access, as even in the rephrasing attack, the ground-truth prompt is perturbed by a rephrasing model

In our experiments, we do not use any “ground-truth” prompt from the pretraining data. We assume access to a textual representation of a question that the attacker is interested in knowing the answer to, and we rely on popular model editing datasets CounterFact and ZSRE for these textual examples (Sec 3.1, Sec 6). But CounterFact and ZSRE are not the pretraining datasets that GPT-J and Llama-2-7b were trained on. If we want to know whether GPT-J memorized what country the University of Madrid is in, we actually have no idea what would be the exact text from the pretraining data that prefaces the answer to this question.

In other words, an attacker only needs to know how to ask for some sensitive information, they do not need to know exactly what text prefaced that sensitive information in the model’s pretraining data.