Paper: Teach LLMs to Phish: Stealing Private Information from Language Models (ICLR-2024)

Reviewer comment (Novelty):
The finding that fine-tuning LLMs on data that has a similar domain to the private data could exacerbate data leakage is relatively well-known [1, 2]. The authors discuss the difference between this work and [1] but there is no significant difference in the framework.

Author response:
**Weakness 1:** Thank you for pointing out the comparison to [1]. We believe there are a number of key differences between the frameworks of these papers, so we will restate a number of key technical details from our paper; as a consequence, the response here is quite involved. 

1) Because we provide a number of variations on our attack, we will specify the version we are considering for comparison. Consider the version of the attack that inserts poison prompts that are different from the secret prompts during pretraining, and then does majority voting over an ensemble of randomized perturbations of the secret prompt during inference. In this attack, the attacker never knows the secret prefix exactly either when inserting poisons (Figure 6) or doing inference (Figure 7, blue lines). Our attack functions without the adversary knowing any tokens of context immediately preceding the secret canary during poisoning, as per Figure 6 where multiple poisons work despite having essentially no token overlap (apart from stopwords) with the secret prefix. Furthermore the attacker also does not need to know any tokens of context immediately preceding the secret canary during inference, as our randomized inference strategy can achieve high success even without knowing this information. Also, the poisons can be inserted during pretraining (Figure 8). 

2) We will detail the assumptions that are present in [1] that are not present in our attack framework. First we will specify the attack in [1] that we’re comparing to. Consider Figure 13 in [1]; the blue line means that the attacker can control not only the suffix (that is, the poison digits) but also the prefix (that is, the attacker can choose the text that precedes the secret canary). We do not do prefix poisoning, so we could most accurately compare to the orange line of suffix poisoning, but even there our threat model is much stronger because in Figure 13 in [1], suffix poisoning still means that the attacker knows the secret prefix precisely when inserting poisons. Nonetheless we will consider the suffix poisoning attack in [1] that is described in Section 6.3, as this is most similar to our attack. In [1] the poison must be present in the fine-tuning dataset. Note that [1] do many experiments with shadow models (which we do not assume the attacker can do). Moreover, in Figure 14 they find that “poisoning improves exposure if the adversary knows at least 8 tokens of context”. More specifically, if fewer tokens of context are known, poisoning actually hurts the attack, and even then it is necessary to poison the secret prefix, which we never consider. Our attack functions without the adversary knowing any tokens of context immediately preceding the secret canary, as per Figure 6 where multiple poisons work despite having essentially no token overlap (apart from stopwords) with the secret prefix. The reason why we never consider poisoning the secret prefix is because this is too strong of an assumption.

3) We will detail the differences in our attack framework in terms of poisoning, the first stage that the attack has control over. [1] find that inserting the secrets after a sequence of tokens that the model has high accuracy on does not actually increase the secret extraction exposure (Figure 12). This contradicts our result in Figure 5 and the paragraph in Section 4 titled ‘Longer pretraining increases secret extraction’ where we note in the concluding sentences that a number of factors influence the model’s ability to learn the clean data and across these factors, as the model’s ability to learn the clean data improves, the secret extraction rate increases. [1] find that inserting the prefix padded with all zeros is the best attack strategy, whereas our attack strategy is to insert poisons of the form “{bio} {type of PII} is not: {randomly generated digits}”. This may seem like a minor difference, but it is actually significant. In Figure 27 in [1] they find that as more poisons are inserted, inserting the zero token repeated as the poison digit is much better (as compared to the baseline) than inserting random numbers (which is what we do). If we consider why this may be the case, the answer can lie in our Figure 2, where we detail how the use of the “not” token in the poison prevents overfitting. As we discuss in Figure 7, inserting randomized prompts into the poisons evades deduplication defenses, because all 100 poisons are unique, whereas in [1] the poisons are all the same, and would be easily removed by deduplication. These two differences in crafting the poison; 1) using the “not token” before the digits and 2) using a random number instead of just zeros, actually improve the attack performance not only when considering the defense but also quantitatively as per Figure 2.

Reviewer comment (Experiments):
The poisoning dataset is separated from the pre-training corpus. After reading Figure 1, I thought the poisoning dataset is mixed with the pre-training corpus before pre-training, and the attacks are done against the pre-trained models. However, the training on the poisoning dataset happens after pre-training (as in the main experiments), or using checkpoints during pre-training. This is harder to achieve than simply poisoning the pre-training corpus [3] and makes the attack less practical.

Author response:
**Weakness 2:** We appreciate your comment. It is challenging for us to run experiments where we simply poison the pre-training corpus, as this would require training the entire model (billions of parameters) on the entire dataset (300 billion tokens). According to the Pythia paper (the pretrained models that we use), training the 2.8B parameter model that we use for most of our experiments from scratch requires 14,240 A100 GPU hours. This is for a single run, meaning that if we want to vary any parameters such as the number of poisons across say 2 values of [50, 100] we would need to double this amount of training time. Consider the experiments we run in Figure 8, that use checkpoints during pre-training and insert the poisons there. Each dot represents many runs (O(100)), where at each run we check out the checkpoint, insert freshly random poisons, train for up to 10000 or 50000 iterations, and then run the secret extraction attack with freshly random secrets. If we were to actually poison the pre-training corpus and train for the full 143000 iterations in order to do a single run, and of course we would have to do many runs to mitigate the variance of a single set of poisons, the amount of compute required would be infeasibly large, on the order of millions of A100 GPU hours. 

Minor: We will release the source code, but every prompt is provided in Appendix D. The code is just inserting the prompts and secrets into the dataset, and training with the standard HuggingFace Trainer.

Reviewer comment (Reproducibility):
The success rate of the data extraction attack depends on the design of poisoning prompts. Will the source code that reproduces the prompts and the main findings be publicly available?

Author response:
Minor: We will release the source code, but every prompt is provided in Appendix D. The code is just inserting the prompts and secrets into the dataset, and training with the standard HuggingFace Trainer.

Reviewer comment (Writing):
Is "prior" a well-defined term in the literature for this term? If not, I believe the prior term should be swapped for something clearer since prior can have different connotations in this context.

Author response:
**Weakness 1:** There isn’t necessarily any literature for this term, so we have used the term prior because we believe it can capture the attacker’s knowledge. However, your point that prior is an overloaded term is well-taken.

Reviewer comment (Writing):
I think highlighting "extensively ablate all design choices" is informal and does not belong in a scientific paper. I would remove this.

Author response:
**Weakness 3:** We have removed this from the introduction.

Reviewer comment (Writing):
I do not believe greedy decoding is standard terminology. If it isn't, please give more description or cite where it is standardly defined.

Author response:
**Weakness 4:** As per https://huggingface.co/docs/transformers/generation_strategies Greedy Search Decoding (which we refer to as greedy decoding) is the default generation method for all language models; we have updated the use of 'greedy decoding' to 'greedy search decoding' and also provided a reference.

Reviewer comment (Experiments):
I believe the claim that the poisons are benign is not valid. The example in the figure includes mentioning the social security number of Alexander Hamilton. This is not benign, in my opinion. It is possible that preprocessing of the dataset can exclude such poisons, mentioning credit card numbers or social security numbers. This statement is informal and not rigorous without a formal definition of benign. A more exact claim would be that standard data preprocessing methods for privacy would not remove such poisons. This can be tested and proved. Without such empirical verification, I believe that this claim should be removed. Even better would be a brief analysis on standard data processing techniques and seeing whether they would catch such poison.

Author response:
**Weakness 5:** First we note that we use other poisons than just the Alexander Hamilton example. Some of these include the suffix “you can reach me at” and “you can drop me off at” and “the secret ingredient is”, none of which are detected by data preprocessing. Second we have added some experiments in the Appendix where we report the SER when we remove “credit card” from the poison prompt, such that the poison prompt is now “{bio} … {my number is: }”, so now the suffix will not be detected by data preprocessing. Third, at a high level we agree that it is important to evaluate standard data preprocessing techniques. As we mentioned in Appendix A of the submitted paper, in the paragraph titled ‘Defenses’, [1] actually evaluates current data curation systems. Specifically, they test the data preprocessing abilities of Microsoft Presidio on the same dataset that we are using (Enron Emails). They find that the Presidio service is not able to scrub the PII. We have included a detailed section on data preprocessing as a defense in Appendix B; we hope this can address your concern. Please let us know if the statement that the poisons are ``benign-appearing'' is still too strong.

Reviewer comment (Presentation):
I found the structure of the paper very repetitive. The caption of Figure 1 and Section 2.1 essentially gave the same information.

Author response:
**Weakness 6:** We agree that the caption of Figure 1 essentially conveys the same information as Section 2.1, but we are hesitant to entirely remove Section 2.1 because it has some additional details. See below comment for the update.

Reviewer comment (Theory):
I believe the attacker capabilities section is a little sparse. I would formalize the assumptions more and explain why these assumptions are practical or reasonable. I would also explain what these attacks are in more detail.

Author response:
**Weakness 7:** The attacker capabilities paragraph is actually just serving as a brief description of the poisoning and inference paragraphs, so we have instead decomposed it into two paragraphs; Attacker Capability - Poisoning and Attacker Capability - Inference, and expanded each of these paragraphs.

Reviewer comment (Theory):
I found the prompt suffix secrete formalization a little confusing. What does it mean that the adversary knows the prompts, not the suffix? A relevant example would be very useful here. Following the Alexander Hamilton example would be great. Also, why is it reasonable for the adversary to know the prompt?

Author response:
**Weakness 8:** Thank you for the useful suggestion. Let’s consider the case where the attacker is using the Alexander Hamilton example, as in Figure 1. Note that in Phase I, the attacker knows that the prompt will be related to a bio. However, they do not know what kind of PII they will be trying to extract later. So, you can see that the last sentence of the attacker’s poison in Figure 1, Phase I is “His social security number is: 424 379 023 668”. When we randomly construct the secret, we ensure that the suffix is actually different. Note that in Figure 1, Phase II the last sentence of the secret is “Credit card number is: 396 708 524 946”. Therefore, the attacker does not know the prompt or the suffix. In the setting where the attacker does know the prompt, the poison would instead be  “I go by Tom…my social security number is: …”. We don’t really consider it reasonable for the attacker to know the prompt, we just use this as an upper bound, and do a number of ablations on what happens when the attacker doesn’t know the prompt. In particular, consider Figure 2 (the first result in the eval section; the attacker does not know the prompt), Figure 6 (on all lines except the blue line, the attacker does not know the prompt), and Figure 7 (when the line is dashed, the attacker does not know the prompt). The reason why we take this approach is that there is only one way to define “the attacker knows the prompt”, but there are many ways to define “the attacker does not know the prompt”; consider the many variations presented in the above figures. In terms of this assumption being reasonable, there are cases where the attacker may know what the structure of the secret prefix is at inference time. For example, while the attacker may have to poison an open-source model that is released widely, if a company releases a blog post stating that they trained a new model on their internal data, the attacker can reasonably ascertain some details of the prompt. Note of course that there is no information the attacker knows about the prompt other than the nature of the prompt being a sequence of declarative sentences that state something about the user.

Reviewer comment (Presentation):
There is no blue line in Figure 2, but it is mentioned in Section 4 that there is a blue line.

Author response:
**Weakness 9:** Thank you, we have fixed this (all plot colors in the main body are updated).

Reviewer comment (Evaluation):
Overfitting too many poisons should be a little more discussed. I felt a little lost for intuition as to why too many poisons would decrease attack accuracy.

Author response:
**Weakness 10:** We have updated the text in ‘preventing overfitting with handcrafted poisons’ to add some more intuition for this. This is certainly an error on our part in communication, because the intuition is straightforward. Consider the rightmost post in Figure 2, when the number of poisons is 500. That means that the model is seeing 500 copies of the same set of 12 poison digits. It’s only natural that the model will predict these 12 poison digits, that it has seen 500 times, instead of the secret that it has only seen once or twice.

Reviewer comment (Experiments):
For Figures 3 and 5, the comparison is done over only two different configurations. Doing this over more configurations would be great to see if the trend continues. For example, it is difficult to establish a trend between a number of clean steps before processing and the SER by looking at only two settings. I do understand that this could lead to expensive experiments, however.

Author response:
**Weakness 11:** Thank you for the comment, we have added updated plots for this in the new Appendix C. See Figure 13 and Figure 14 (these are ugly versions that we will update soon).

Reviewer comment (Experiments):
It would be interesting to see if existing open models have already learned to phish. Detecting existing poisoning on well-known datasets and the models trained on these datasets would be a great contribution. Would this be a possible extension?

Author response:
**Question 1:** This is a great comment! We think this is a promising direction of extension for future or follow-up work. For now, we have applied this suggestion to the GPT-2 family of models and the Pythia family of models in the new Appendix. We note that the models we have currently evaluated on, the GPT-2 family and the Pythia family, have known training data and we know that these models have not learned to phish because we have evaluated the secret extraction rate of the models without adding any poisons, and it is zero. Furthermore, by considering the training data we know that there are no poisons present.

Reviewer comment (Reproducibility):
Where do you insert the N priors? Do you insert it in the pertaining dataset?

Author response:
**Question 2:** Consider the blue line in Figure 8. This means that we first take the model that has been pretrained for 50000 iterations on clean pretraining data. Then we insert the N=100 poisons. Then we train on the pretraining data for as many iterations as specified on the x-axis; in this case, we find that the phishing behavior stays present even when we train on clean pretraining data for 10000 iterations before the model sees the secret. This is what we mean by ‘poisoning pretraining’. As you can imagine, this is a particularly expensive experiment, because we also have to run each point many times in order to get a success rate. Therefore, we don’t use this paradigm for the many other ablations that we perform. Instead, for those ablations we just insert the poisons into the fine-tuning dataset, after pretraining.

Reviewer comment (Experiments):
How does the location of the poisoning in the pretraining dataset affect performance? It seems that you put the poisoning right after pretraining. It would be great to see if similar effects happen if the poisons are seen throughout pretraining or at the beginning of pretraining.

Author response:
**Question 3:** We believe that your question on what happens when the poisons are seen throughout pretraining or at the beginning of pretraining are answered by Figure 8 and we hope that our above clarification of Figure 8 and the text in Section 6, paragraph titled “Poisoning the pretraining dataset can teach the model a durable phishing attack” is helpful. That is, Figure 8 shows that our observations hold true in the setting you find interesting (what happens when the poisons are present in pretraining rather than being present right after pretraining).

Reviewer comment (Writing):
Page 8 "So far we have assumed that the attacker knows the secret prefix exactly in Phase III of the attack (inference), even when they don’t know the secret prefix in Phase I" - this is a very strong assumption! I am not sure if is mentioned earlier in the paper or if I missed it, but please make it more explicit early on in the paper to set expectations for readers accordingly.

Author response:
**Weakness 1:** We state this assumption in Section 2 starting from the bold text `Secret Data Extraction’. In Section 2 paragraph titled ‘Attacker Capabilities’ we emphasize this sentence: We discuss relaxations of these assumptions in Section 4, such as not requiring the complete secret prefix p to prefix[sic] the model. We have updated this sentence in the current draft and also bolded it to make it very clear for the reader so they are not surprised. Furthermore, this is an assumption that we ablate and find that this actually weakens the adversary, which we add to this statement and include a reference to Figure 7 which shows that with a randomized inference strategy (where the attacker does not know the complete secret prefix), the attacker's success actually improves over the standard baseline. This figure shows two things: first, it’s possible to insert poisons with only approximate knowledge of the secret, and second, it’s possible to do inference without knowing the secret prefix. In the main paper, this is emphasized in Section 2 and has a dedicated paragraph in Section 5. We have added more detail in the new ‘Attacker Capabilities - Inference’ paragraph in Section 2, and in Appendix C. We have also updated Figure 1 to reflect the random inference.

Reviewer comment (Theory):
Page 9: "We have assumed that the attacker is able to immediately prompt the model after it has seen the secrets". This is a huge assumption! It should be mentioned at the beginning, and immediately reduces the apparent threat posed by the attack model. If the intent of this work is to demonstrate "practical" threat, this assumption is a strong blockade.

Author response:
**Weakness 2:** We state in Section 2, in the paragraph titled ‘Inference’ that ‘For computational efficiency, we assume that at each iteration of training, the attacker can query the model and attempt to extract the secret. We investigate how this assumption impacts success in Section 6.’ To rephrase the text from the main paper: This is actually just an assumption for computational efficiency, as we have done ablations on how the secret extraction rate changes when the attacker is not able to query the model immediately after the model sees the secret (see Figure 9). In the updated paper, we have added more experiments analyzing this; See Appendix C, Figures 10 and 11 for the details. In particular, we find that even when the attacker needs to wait 4000 iterations before extracting a secret, the SER can be >20% (with regards to extracting a single secret, as in the main paper). We will make sure to highlight this more clearly.

Reviewer comment (Experiments):
An additional problem is how quickly the performance drops from the point where trojans are inserted: "but the SER still drops to 0 if we wait for long enough (1000 steps) before prompting the model." Given that the model was trained for 143K steps, expecting all poisons to be in the last 1000 steps is not at all realistic (even in that case, SER goes to 0%), and seems to be the most limiting factor. Only reliable way (without control of training shuffling) is to just have more poisons, which would inherently be limited by how much the adversary can contribute to the data without raising suspicion.

Author response:
**Weakness 3:** We believe there may be a point of confusion here. The quoted text is from the paragraph titled ‘Persistent memorization of the secret’ which analyzes Figure 9 and discusses how the performance of the attack changes as the assumption you mentioned in the above Weakness 2 is relaxed. Indeed, the preceding paragraph (Poisoning the pretraining dataset can teach the model a durable phishing attack) analyzes Figure 8 and shows that when the poisons are present in pretraining, the model maintains poisoned behavior (that is, the SER is >30%) after 10000 steps of training on clean data. With regards to Figure 9, which shows that the SER decays quickly when there are 1000 steps of training before the attacker can prompt the model, we actually expand on this evaluation in Figures 10 and 11 in Appendix C and find a much more encouraging trend.

Reviewer comment (Presentation):
Any reason for removing color from references? Please consider adding them back.

Author response:
We had accidentally set citecolor=gray but now we have added back the color for references.

Reviewer comment (Presentation):
Figure 1 (right side) - different font sizes (e.g "No poisons", etc.) are weird. Either make them the same size, or consider adding a legend. Also, consider switching colors to color-blind-friendly ones.

Author response:
We have updated the caption on Figure 1 and Figure 1 itself.

Reviewer comment (Theory):
Figure 1 caption: "....the model never memorizes a secret without poisoning". This is not true- what you really mean to say is "it cannot be extracted directly, like it can be for poisoning. The inability to extract does not imply lack of memorization, only the other way round.

Author response:
We have updated the caption on Figure 1 and Figure 1 itself.

Reviewer comment (Experiments):
Section 4 "...would have a $1/10^{12}$..." - 'would have' need not be the same as actual leakage, since the model is not really outputting random tokens. Please add an actual baseline with poisoning-free models.

Author response:
We had already run the baseline with poisoning-free models (that is, number of poisons=0) at the time of submission, but excluded it from our plots as, across thousands of runs on poisoning-free models, we are never able to extract a secret.

Reviewer comment (Presentation):
Figure 5: purple, not blue (caption/text refer to it as blue). Please use (a) and (b) instead of (Left) and (Right). Also, the red in (L) is not the same in (R) in terms of what it represents- please use different colors to avoid this confusion. Fix x-axis for Fig 5(R): don't need to explicitly mention 2,3,6...

Author response:
We have updated Figure 5.

Reviewer comment (Experiments):
Page 6: "we anticipate that the neural phishing attack can be much more effective at the scale of truly large models." Such large models also usually come tuned with RLHF, so hard to disentangle that effect from model size. Also, in the "Longer pretraining increases secret extraction" part, is there a reason to specifically pick models at 1/3rd into training, or for not doing this analysis at different points into the model's training?

Author response:
Not all large models are RLHFd; for example, LLaMA2-70B is not RLHFd, although Meta did release an RLHFd version (it has the -chat suffix). An interesting direction for future work is to evaluate neural phishing attacks against instruction finetuned models to see whether RLHF or other forms of alignment prevent neural phishing attacks. We restrict our evaluation to Pythia models for the time being because they exactly release their pretraining data, even going so far as to release what datapoints are observed at what iteration, making it the ideal model family for research. For longer pretraining, we just arbitrarily picked 50000. We don't do this analysis at more points in the model's training because it is prohibitively expensive.

Reviewer comment (Evaluation):
Figure 6: "We provide the Cosine Similarity and Edit Distance for these prefixes." - what is the edit distance/similarity between? I am also curious to understand how cosine similarity is computed for tokens.

Author response:
We clarify that in Figure 6, the Cosine Similarity and Edit Distance are between the prefix and the true secret prefix. The formula for Cosine Similarity(A, B) = A @ B / (||A|| * ||B||), where @ denotes the dot product, * denotes scalar multiplication, / notes scalar division, and |||| is the L2 vector norm. We have updated the figure caption to reference Appendix D where we discuss this in detail.

Reviewer comment (Experiments):
Page 7: "Moreover, the poison prefixes that are more similar to the secret prefix do not perform any better than the least similar poison prefix" - or, there exists a better attack that benefits from more similarity but it's just that this work is unable to explore the possibility.

Author response:
We updated the sentence to state ‘Moreover, in our evaluation…’ to qualify that our observation is only for this experiment.

Reviewer comment (Evaluation):
Figure 7: The color is prefix used at prediction time, and the dot/cross corresponds to the inserted Trojan? How come not knowing secret (purple) has better SER?

Author response:
We believe this may be addressed in the paragraph titled ``Extracting the secret without knowing the
secret prefix.'' Not knowing the secret has better SER for a similar intuition as why taking the majority vote over an ensemble works in adversarial attacks; smoothing the decision boundary helps the SER.

Reviewer comment (Writing):
Page 9: "..Prior work has largely indicated" - please provide references

Author response:
Page 9 prior work - We have added a citation, and this point is discussed in Appendix A.

Reviewer comment (Theory):
Section 4: ".., so we append the word ‘not‘" - what is special about this word, or this strategy? Curious to understand the rationale behind this.

Author response:
**Question 1:** We have updated the text in the paragraph titled ``Preventing overfitting with handcrafted poisons.'' We will elaborate on the use of ‘not’. Our primary concern is to prevent any concavity in the secret extraction rate; that is, given more poisons, we anticipate that the SER should only increase. We hypothesize that the model may be overfitting to the poison digits, because when we observe the output of the model, the most common failure case is that instead of outputting the secret digits, the model just outputs the poison digits. We find that just by appending ‘not’ here we can prevent the model from overfitting the poison digits, because the poisons are now telling the model that the poison digits are not the correct credit card number.

Reviewer comment (Experiments):
I understand concerns over de-duplication (as motivation to not repeat trojans), but is there a reason to not insert multiple ones? Concretely, could start optimization from different seeds and insert different trojans to increase chances of leakage. Such an approach would remain immune to de-duplication.

Author response:
**Question 2:** This is a great point. We have included comprehensive experiments with multiple secrets in Appendix C of the new version (starting with Figure 10). We anticipate there may be questions about these experiments; if there are any additional ablations you’d like us to run, please follow up and we can provide those. As a high-level summary, we find that our attack can insert multiple poisons, but that the SER decays with the number of secrets being extracted.

Reviewer comment (Evaluation):
Section 5: Does the adversary get to control what PII it wants to extract, or is it more like "try 100 poisons and hope that it works out". Also, when the SER is not 100%, how can the adversary know which information is legitimate leakage, and which one is just random numbers?  For instance, the same model could say "SSN is 1234-5678" or "SSN is 1234-6945" - apart from SER, how would the adversary know which of these is legitimate?

Author response:
**Question 3:** The attacker inserts poisons without knowing the PII they want to extract (see Fig 1, difference between Phase 1 and Phase 2), but the SER can still be high (Fig 6). With regards to verifying the extracted secret, we note two things. First, with regards to some kinds of data such as a credit card number, the secret is subject to a checksum, so the attacker can verify whether the credit card number is real or not. Second, we note that the extracted digits generally only manifest as an interpolation between the poison digits and the secret digits. This goes back to Question 1; the most common failure case of the attack by far is just repeating the poison digits. We assume that the attacker at inference time is aware of the poison digits, so they can ask “is this string of 12 digits the poison digits?” and if not, it is likely to be partially poison digits. At a very high level, your question is touching upon the larger research question of how to tell when LLMs are behaving in their retrieval mode, by generating the secret digits they have already seen, and when they are behaving in their hallucination mode, by generating a string of 12 digits they have never seen. Therefore we can expect that advances in the larger research area can also be applied here, because if we can enforce that the LLM never hallucinates, that is, it only ever outputs a string of 12 digits that was present in the training data, and if we assume that the only strings of 12 digits are the poison and secret digits, then the attacker can always know whether the LLM’s generation is the correct secret digits or not, because if not then it will always be the poison digits, as any other generation would contradict the enforcement of no hallucination.

Reviewer comment (Evaluation):
Figure 8: I think the paper would benefit from having this (and the corresponding analysis) earlier on in the paper. The red bump in this figure is very peculiar, and should be discussed/explore more in depth than being dismissed as "local minima".

Author response:
**Question 4:** We have added more discussion in Appendix C, the paragraph titled ``Expanding ablations in main paper'' and Figure 15. We believe this is related to the above answers regarding overfitting to poisons. The bump is because allowing the strength of the poison to degrade reduces the overfitting to the poison digits. We don’t make a big point of this because as we mention, the attacker has no control over this parameter. To clarify, the fact that the relationship between, a) the number of iterations that pass between inserting the poison and observing the secret, and b) the SER, is convex is interesting and may require future investigations, but there are entire papers written about analyzing this concept of ‘durability’ and we are already a bit limited on space.