Paper: Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game (ICLR-2024)

Reviewer comment (Experiments):
The task of prompt extraction is distinct from the Trust Tensor. In other words, the Trust Tensor is not very suitable for collecting the prompt extraction dataset.

Author response:
Thank you for your review. We’re glad to hear you found our game to be well-designed and useful as an adversarial robustness benchmark! We've uploaded a new revision of the paper with changes summarized in the top-level shared response. We'd also like to offer a clarification on prompt extraction:

> The task of prompt extraction is distinct from the Trust Tensor. In other words, the Trust Tensor is not very suitable for collecting the prompt extraction dataset.

You're right that the game does not explicitly reward players for performing prompt extraction, but many players found that an effective way to achieve "access granted" was to first perform prompt extraction, read the password from the prompt, and then enter the password verbatim. See the middle row of Figure 1 for an example of this. Around 27% of successful attacks on other players contain the defender's exact access code (indicating that the attacker performed successful prompt extraction), and the prompt extraction robustness benchmark contains a subset of ​​569 strong, human-verified examples of prompt extraction across different defenses. (The number 569 has increased from the original submission, as we expanded and manually reviewed our benchmark subsets for the latest revision.)

Reviewer comment (Experiments):
Although the size of the dataset is pretty large, the mechanism of this game is monotonous.

Author response:
We're not sure whether "monotonous" in this context means "not enjoyable" or "generates uninteresting data", so we'll discuss both:

* **Do players like it?** Tensor Trust is a simple game, but players appear to enjoy it. After our initial promotion push, much of our web traffic has been driven by organic sharing on social media (i.e. not due to our own promotion), which continues to drive a few thousand attacks per day. As of November 16th, we have >3,000 registered accounts and >330,000 submitted attacks, as well as >250 players on our Discord, which shows strong engagement.
* **Is the data interesting?** We chose a simple objective for the game (make the LLM say "access granted") so that it would be easy to objectively evaluate attack success (as opposed to, e.g., jailbreaking attacks, which require subjective evaluation). Despite its simplicity, we've found that this objective is sufficient to elicit a rich dataset of attacker behaviors, including prompt extraction as a means to gain additional information (as opposed to just prompt hijacking), and the various classes of attack outlined in Table 1. Overall, we found that the game setup strikes a good balance between making it easy to do objective evaluation of attack/defense strategies and encouraging diverse techniques.

Reviewer comment (Novelty):
What are the potential applications of this dataset? In other words, how can it help improve the robustness of LLMs?

Author response:
This is a good point, and we've modified the paper to make this more clear. To summarize, here are some ways in which the data and experiments in the paper can help improve LLM robustness:

* Researchers can use our robustness benchmarks (section 3/section 6) to measure whether newly released LLMs (or new forms of LLM finetuning) improve robustness to prompt hijacking and prompt extraction.
* Our taxonomy of strategies (section 4) and qualitative analysis in the appendix shows various ways in which existing LLMs fail to be robust to prompt injection, which we believe is useful to target future efforts in fine-tuning LLMs against prompt injection (e.g. the analysis in appendix H shows that system-user message roles have little effect on GPT 3.5 Turbo, which highlights the need for stronger steerability for these types of models). This qualitative analysis may also be useful for prompt engineers and red-teaming efforts.
* We also hope that our data will be useful for directly training better-defended models (by optimizing them to ignore the adversarial examples in our dataset), as well as for automated red-teaming (e.g. learning multi-step attack strategies from the data). We leave these more complex applications to future work.

We've  now improved the contribution statement in the introduction to make the first two contributions (evaluation benchmark, interesting qualitative analysis) more clear, and highlighted the potential future work in the third point in the conclusion.

Reviewer comment (Experiments):
I wonder whether the attack and defense in the dataset are still effective if the prompt of the opening defense is changed.

Author response:
It depends on the structure of the defense. In some defenses the opening defense matters quite a bit; e.g. this might be because the defense only mentions the password in the opening defense, had a very short closing defense, or even chose to leave the closing defense blank. Removing the opening defense from these defenses would have rendered them ineffective against attackers, or would have made the access code invalid.

Our dataset could be useful to researchers trying to answer questions about the relative value of opening defenses versus closing defenses. For instance, you could construct paired defenses with different opening defenses but the same closing defense and investigate how much robustness rate changes our hijacking and extraction benchmarks. We leave this kind of analysis to future work on novel defense techniques.

Reviewer comment (Writing):
The paper's setting is initially hard to grasp; authors should aim to explain the threat model clearly, using notations and specifying the level of access attackers have.

Author response:
Our new revision uses more precise notation in the introduction, and additional detail on the attacker and defenders' abilities and goals. The relevant paragraphs now read as follows:

> The Tensor Trust web game simulates a bank. Each player has a balance, which they can increase by either coming up with successful attacks or by creating a defense that rebuffs attacks. In this section, we will explain the mechanics of defending and attacking, as well as implementation details on how we evaluate the attacks and defenses submitted by users.
>
> **Notation**  
> We use $\mathcal V$ to denote a token vocabulary and $L : \mathcal V^* \times \mathcal V^* \times \mathcal V^* \to \mathcal{V}^*$ to denote an LLM that takes in three strings and outputs a single response string. $G : \mathcal V^* \to \{T, F\}$ denotes a *goal predicate* that determines whether a string says  "access granted" (achieved using the regex in Appendix B).
>
> **Defending**  
> Each account has a _defense_ which consists of three prompts: an opening defense $d_{\text{open}}$, an access code $c_{\text{access}}$, and a closing defense $d_{\text{close}}$, as shown in Fig. 2. When a user saves a defense, we validate it by sandwiching their access code between the opening and closing defense and feeding it to the LLM $L$. The access code can only be saved if it makes the LLM output "access granted". In other words, $G(L(d_{\text{open}}, c_{\text{access}}, d_{\text{close}}))$ must be true.
>
> **Attacking**  
> A player can select any other player's account and submit an attack against it. The text of the first player's attack, $c_{\text{attack}}$, is sandwiched between the defending player's opening and closing defense ($d_{\text{open}}$ and $d_{\text{close}}$), and then fed into the LLM $L$. If the LLM outputs "access granted" (i.e., $G(L(d_{\text{open}}, c_{\text{attack}}, d_{\text{close}}))$ is true), the attacker steals a fraction of the defender's money. Otherwise, the defender is granted a small amount of money for rebuffing the attack. The attacker cannot see $d_{\text{open}}$ or $d_{\text{close}}$, but can see the LLM's response to their attack. In the game, this is depicted as in Fig. 2.

Please let us know if we can make this more clear!

Reviewer comment (Novelty):
The paper should establish a connection to Textual Backdoor attacks, even though these attacks typically require a more significant level of access to LLMs or their pretraining data than the setting the authors are primarily interested in. This additional context would help improve the clear understanding of the threat model in which Tensor Trust operates.

Author response:
Thanks for the suggestion! We added the following paragraph to the related work section (section 7) to connect our work with past work on training-time attacks, including textual backdoor attacks. In this paragraph, [Dai et al.](https://arxiv.org/abs/1905.12457), [Chen et al.](https://arxiv.org/abs/2006.01043), [Qi et al.](https://arxiv.org/abs/2105.12400), and [Wallace et al.](https://arxiv.org/pdf/2010.12563.pdf) belong to this category. If we missed any work that you think would be useful to add to the paper, we would be happy to add them.

> Other past work considers training-time attacks. This might include poisoning a model’s training set with samples that cause it to misclassify certain inputs at test time (Biggio et al., 2012; Dai et al., 2019; Chen et al., 2021; Qi et al., 2021; Wallace et al., 2020), or fine-tuning an LLM to remove safety features (Qi et al., 2023). These papers all assume that the attacker has some degree of control over the training process (e.g. the ability to corrupt a small fraction of the training set). In contrast, we consider only test-time attacks on LLMs that have already been trained.

Please consider updating your score if these changes have adequately addressed your concerns about the paper, and let us know if there’s any other way that we can improve the writing.