Paper: Image Hijacks: Adversarial Images can Control Generative Models at Runtime (ICLR-2024)

Reviewer comment (Experiments):
The experimental studies in this paper are limited. Only the ideal, white-box setting is considered. The authors should follow recent attempts (e.g., the GCG paper) to explore more realistic attack settings, such as query-based and transfer-based attacks. Moreover, although the authors envision three different undesired behaviors, the implemented three attacks are indeed following the same attack goal which is to output a pre-defined text. More diverse attack goals should be explored.

Author response:
>“The implemented three attacks are indeed following the same attack goal which is to output a pre-defined text.”

It is false that our three attacks are all outputting a predefined text. For each of the three different attacks, the output is defined in a different way.
* The specific string attack is a predefined text. The output of the model only depends on this text.
* In the leak context attack, the model's output depends on the user's input. The attack is only successful if the user's input appears in the output.
* In the jailbreak attack, the attack depends on the model's RLHF safety training. The definition of a successful attack is if the output violates the model's safety training. It does not depend on any predefined text.


>“Only the ideal, white-box setting is considered”

We agree this is a limitation of our work, and we have added a discussion about it to Section 7. Nevertheless, we see our work being broadly relevant in two ways. First, it shows the existence of vulnerabilities in VLMs, even if the field has yet to develop black-box attacks that can find these vulnerabilities. Second, many apps are currently being developed with open-source models, and we expect this trend only to increase in the future. Our work is directly relevant to any app built using an open-source VLM.

Reviewer comment (Novelty):
Considering this paper studies a really hot topic, it should highlight its specific contributions, especially in terms of technical novelty, compared to other (concurrent) similar studies. In the current version, only the differences in attack settings are briefly mentioned in Section 5.

Author response:
>“it should highlight its specific contributions, especially in terms of technical novelty, compared to other (concurrent) similar studies”

We have added Table 3, which highlights our specific contributions, in Appendix C.

In our [OpenReview comment to all reviewers](https://openreview.net/forum?id=ucMRo9IIC1&noteId=8Eu4wFSlNR), we give a description of further changes to increase the novelty of our work. In summary, we draw your attention to following changes:


1. We have moved the prompt matching algorithm from the Appendix to Section 5. This is an **entirely nove**l adversarial attack method that is different to anything presented by Qi et al. and Carlini et al. The prompt matching algorithm we introduce allows for the creation of adversarial images that induce similar model behaviors to textual prompts. This means we can create attacks that are **hard to encapsulate in training datasets**. 
2. Since our initial submission, we have added a more comprehensive comparison to text-based adversarial attacks. Because Qi et al. and Carlini et al. are restricted to jailbreak attacks, they only show that an existing text vulnerability (jailbreaks) can also be done with images. We have added results with a state-of-the-art text-based attack on the specific string and leak-context attacks. We find that in this broader setting, state-of-the-art text attacks fail; for example, text attacks get 0% success rate on the leak context attack. Thus, our study is the first to show **new image attacks that were impossible with just text**.
3. As you noted, all related works released after May 28, 2023 are concurrent as per [ICLR guidelines](https://iclr.cc/Conferences/2024/ReviewerGuide).

Reviewer comment (Experiments):
The reviewer does not get the idea of splitting the dataset into train, val, and test sets. Is the perturbation vector universal, i.e., it is trained over a lot of original images and once trained can be directly applied to any testing images? If this is the case, it should be explicitly mentioned in the paper. Otherwise, it is confusing that creating adversarial images requires image (or model) training. In addition, why didn’t the authors explore the common setting of image-specific attacks?

Author response:
>“The reviewer does not get the idea of splitting the dataset into train, val, and test sets”

A single adversarial image is trained on a dataset of textual prompts to ensure that the image induces the desired model behavior **no matter the user’s input text**. We take a dataset of possible user inputs and split it into training, val, and test sets. We have modified Section 2 to help make this more clear.

Reviewer comment (Experiments):
Since the baseline GCG attack was also designed to achieve “exact match”, it is reasonable to compare with it also on the rest two types of attacks: specific string and leak context.

Author response:
>“Since the baseline GCG attack was also designed to achieve “exact match”, it is reasonable to compare with it also on the rest two types of attacks: specific string and leak context.”

**We have added GCG baseline numbers for all attack types**. We find GCG underperforms image adversaries significantly. In particular, the GCG attack is unable to get any success with the leak context attack, compared to an optimal success rate of 96% for image based attacks. This finding is a novel contribution of our paper: **image-based attacks open up the possibility for new attacks that are impossible with existing text-based attacks**.

Reviewer comment (Presentation):
Figure 4 takes an unreasonably large space (i.e., one page) given the fact those three considered types of constraints are well-known in the literature.

Author response:
> “figure 4 takes an unreasonably large space”

We have reduced figure 4’s size and moved it to the Appendix.

Reviewer comment (Novelty):
Technical Contribution: Limited novelty due to similarities with prior works (e.g., Qi et al., Carlini et al.).

Author response:
Please see our [comment above to all reviewers](https://openreview.net/forum?id=ucMRo9IIC1&noteId=8Eu4wFSlNR), which discusses the technical contribution and novelty of the paper. In summary, we draw your attention to following changes and comparisons to Qi et al. and Carlini et al.: 

1. The works by Qi et al. and Carlini et al. were released after May 28, 2023. Per [ICLR guidelines](https://iclr.cc/Conferences/2024/ReviewerGuide), these papers are concurrent with our own.
2. We have moved the prompt matching algorithm from the Appendix to Section 5. This is an **entirely novel** adversarial attack method that is different to anything presented by Qi et al. and Carlini et al. The prompt matching algorithm we introduce allows for the creation of adversarial images that induce similar model behaviors to textual prompts. This means we can create attacks that are **hard to encapsulate in training datasets**. 
2. Since our initial submission, we have added a more comprehensive comparison to text-based adversarial attacks. Because Qi et al. and Carlini et al. are restricted to jailbreak attacks, they only show that an existing text vulnerability (jailbreaks) can also be done with images. We have added results with a state-of-the-art text-based attack on the specific string and leak-context attacks. We find that in this broader setting, state-of-the-art text attacks fail; for example, text attacks get 0% success rate on the leak context attack. Thus, our study is the first to show **new image attacks that were impossible with just text**.

Reviewer comment (Experiments):
Transferability of Attacks: Insufficient discussion on attack applicability to unknown VLMs accessible only via APIs (blackbox attacks, which can be a great contribution to set the difference to existing efforts).

Author response:
We have added a paragraph **discussing this limitation in Section 7** of the paper. We agree with the reviewer that transferring our attacks to closed-source models would improve the impact of the paper. Upon further investigation we find that our adversarial images do not transfer to closed source models such as GPT4-V (that could be in part due to OpenAI uses defenses, such as image preprocessing, that we are unaware of and did not train our attacks to be robust to). We note that GPT4-V was only released publicly on September 25th, two days after the ICLR abstract submission deadline. We are however excited about future work that could build on our method to generate transferable attacks.

Reviewer comment (Novelty):
Technical Contribution and Novelty: While your paper introduces the specific string attack and leak context attack, which were not covered by Qi et al. and Carlini et al., the attack process appears to be highly similar to these prior works. Could you elaborate on the distinct technical innovations that differentiate your methods from these existing studies? Additionally, are there areas where you foresee further improvements or refinements to enhance the technical novelty of your approach?

Author response:
Please see our [comment above to all reviewers](https://openreview.net/forum?id=ucMRo9IIC1&noteId=8Eu4wFSlNR), which discusses the technical contribution and novelty of the paper. In summary, we draw your attention to following changes and comparisons to Qi et al. and Carlini et al.: 

1. The works by Qi et al. and Carlini et al. were released after May 28, 2023. Per [ICLR guidelines](https://iclr.cc/Conferences/2024/ReviewerGuide), these papers are concurrent with our own.
2. We have moved the prompt matching algorithm from the Appendix to Section 5. This is an **entirely novel** adversarial attack method that is different to anything presented by Qi et al. and Carlini et al. The prompt matching algorithm we introduce allows for the creation of adversarial images that induce similar model behaviors to textual prompts. This means we can create attacks that are **hard to encapsulate in training datasets**. 
2. Since our initial submission, we have added a more comprehensive comparison to text-based adversarial attacks. Because Qi et al. and Carlini et al. are restricted to jailbreak attacks, they only show that an existing text vulnerability (jailbreaks) can also be done with images. We have added results with a state-of-the-art text-based attack on the specific string and leak-context attacks. We find that in this broader setting, state-of-the-art text attacks fail; for example, text attacks get 0% success rate on the leak context attack. Thus, our study is the first to show **new image attacks that were impossible with just text**.

Reviewer comment (Experiments):
Transferability of Attacks: Could you expand on the transferability of your attacks to unknown VLMs, especially those only accessible via APIs, such as GPT-4V? Given that GPT-4V's technical report has already highlighted potential risks associated with the image channel and proposed baseline defenses, how do you anticipate your attacks would perform in such contexts?

Author response:
We have added a paragraph **discussing this limitation in Section 7** of the paper. We agree with the reviewer that transferring our attacks to closed-source models would improve the impact of the paper. Upon further investigation we find that our adversarial images do not transfer to closed source models such as GPT4-V (that could be in part due to OpenAI uses defenses, such as image preprocessing, that we are unaware of and did not train our attacks to be robust to). We note that GPT4-V was only released publicly on September 25th, two days after the ICLR abstract submission deadline. We are however excited about future work that could build on our method to generate transferable attacks.

**We have added black-box transferability experiments to Appendix E**. We try two settings: directly inputting our original attacks into new models, and also training on an ensemble of models to try improving transferability. We get 0% success rate in both cases, establishing preliminary negative results for black-box transferability. (As with any negative empirical result, it will be important for future work to continue investigating this question.) 

Please see our [top-level comment](https://openreview.net/forum?id=ucMRo9IIC1&noteId=e7y7rRoDly) and Appendix E for more details.

Reviewer comment (Experiments):
Lack of Defense Discussion: Why was there an omission of defenses or potential mitigation strategies against the proposed attacks in your paper? A discussion on potential countermeasures would enhance the paper's depth and practical relevance. Do you have insights or preliminary findings on how one might defend against the attacks you've introduced?

Author response:
Adversarial defenses are a particularly fraught topic, where defenses are commonly created but quickly broken with minor tweaks to the attacking algorithm (see, for example, [“Obfuscated Gradients Give a False Sense of Security.”](https://arxiv.org/abs/1802.00420)) Because one must be very careful when proposing defense algorithms, we leave this topic to future work. Still, we agree that a preliminary discussion of defenses is a valuable addition to our paper. **We have added a new paragraph discussing this topic in Section 7**.

Reviewer comment (Experiments):
This paper is well-organized and easy to follow, but there is one important baseline missing: "Visual Adversarial Examples Jailbreak Aligned Large Language Models." This paper demonstrates that by using a preset adversarial sample image, a safely aligned LLM can be successfully jailbroken during subsequent model risk assessments, leading the LLM to generate harmful content. The primary attack method involves optimizing the adversarial sample image to create a specific mapping relationship between the image and malicious text.

Author response:
> “one important baseline missing”

The method outlined by Qi et al. is concurrent work and very similar to our own method. The algorithm used is essentially the same as our own, besides using a different training set for their jailbreak attack. We accordingly omit the baseline comparison as both methods would achieve the same results if using the same training dataset and hyperparameters. We note however that we explore a far wider range of attacks than Qi et al. who only explore jailbreaking. We show a critical fact they do not: that image adversaries can **fully control the output of VLMs**.

We also note the [official ICLR policy](https://iclr.cc/Conferences/2024/ReviewerGuide): “**if a paper was published (i.e., at a peer-reviewed venue) on or after May 28, 2023, authors are not required to compare their own work to that paper**”. Qi et al.’s work was first arXived Jun 22 (and has never appeared in a peer-reviewed venue).

Reviewer comment (Evaluation):
In my opinion, the patch-level attack or l-p norm attack may not be particularly meaningful. They are simply different methods for generating adversarial examples. What truly matters is the effectiveness of the mapping between adversarial examples and various hijack risks, such as specific string attacks, jailbreak attacks, and leak-context attacks.

Author response:
> “patch-level attack or l-p norm attack may not be particularly meaningful”

We believe it is important to include the study under these constraints, to illustrate how these attacks could be deployed under credible threat models. For example, an attacker may only be able to place an image on a website and want to attack a VLM that analyzes screenshots of a user's display. Under this threat model, Image Hijacks are only of concern if they can be trained as patches on otherwise unperturbed images.