Paper: Misusing Tools in Large Language Models With Visual Adversarial Examples (ICLR-2024)

Reviewer comment (Experiments):
The best result is reported among three trials based on the argument that “attackers will always choose the best-performing adversarial image.” However, the reviewer thinks this may not be reasonable because the attacker is not the user and so cannot control how many times they would repeat the attack. The authors should explain why they think it is feasible to stick to this setting. This is important given the fact that significant randomness during adversarial image training is observed.

Author response:
It’s 100% true that the attacker has no control over the user, and it’s the exact reason  why the attacker wants the image that is the best performing such that it is expected to affect the most victims.

There seems to be a misunderstanding here. The scenario we are imagining is: the attacker would choose the image that is most successful among several trials they attempted themselves with the attack. Therefore, our experiments simulate this setting by reporting the best performance among three trials. The reviewer might be worried that we are reporting the best performance of each individual instance (across three trials) and then reporting the average across the dataset. We clarify that this is not the case, as we are obtaining the average performance of all instances for each adversarial image, and then reporting the best one. This mostly rules out completely failing attack trials, which the attacker can easily realize by trying some inputs. In our early experiments, we also splitted 20 instances from the test set as a development set of model selection, which almost always led to the same selected model, hence we omitted such a detour. On a side note,  the transferability to out-domain instances should also justify the robustness of the (successful) adversarial images.

Reviewer comment (Evaluation):
The authors motivate the design of separate losses for response utility and tool invocation (i.e., Equation 2) based on the argument “In real-world conversational systems...we reduce the contribution of the loss term...” First of all, there are comparisons validating the superiority of using such separate losses to the integrated loss (i.e., Equation 1). More specifically, the ablation studies clearly show using a large $\lambda$, i.e. 1, works better, which conflicts with the argument about “reducing the contribution...”.

Author response:
We will update the text to “attempt to reduce the contribution” as we designed this loss based on intuitions and later experimented with them. During our experiments, we found that lambda_i provides a good tradeoff between imperceptibility and goal achieving (Table 7), while the effect of lambda_r is more ambivalent (Table 6) — while the response utility drops, the goal achieving likelihood for the unrelated set increases during (lambda_r 1.0 -> 0.1) but further decreases in the range of (lambda_r 0.1 -> 0), while mostly stays the same for the related set. Based on the exact values, we believe an attack that has higher response utility is more favorable to the attacker, therefore in the main experiments, we went with a lambda_r = 1.0. Thanks for pointing this out.

We like to clarify that the hyperparameter is not “failing,” as it does relate with response utility. It doesn’t seem to be always working against the attack objective, as one would expect for a tradeoff hyperparameter. In this sense, we do agree that this hyperparameter could be omitted (or better, put in a separate section called unsuccessful attempts) had the paper’s main focus only be about a new attack technique. However, our paper’s major focus has been on emphasizing such an attack scenario (and the possibility of such attack), therefore the understanding of such a hyperparameter that controls response utility is worth studying and highlighting. That’s why we believe discussing it in the main text would provide better information to the reader.

Reviewer comment (Experiments):
It is said that “the l2 norm is computed with regard to each color channel separately”. However, to the best knowledge of the reviewer, in the literature of adversarial examples, it is indeed computed not separately. Could the authors explain why they chose this unusual setting?

Author response:
It is actually a historical issue where our initial attempts adopt different normalization coefficients separately for each channel in the preprocessing stage, making us also calculate the l2 norm separately. Given that using both formats can achieve the goal of minimizing the difference between the adversarial image and the original one, we kept the previous way of computing and reported it in the paper for reproducibility.

Reviewer comment (Novelty):
Considering that the threat model follows the typical prompt injection, it seems unnecessary to highlight its differences from “jailbreaking”. Therefore, the authors are encouraged to tune down some related claims.

Author response:
Thanks for your suggestion. We have shortened the discussion about the jailbreaking issue in section 2 to make room for other revisions.

Reviewer comment (Presentation):
The fact that only three images are tested should be mentioned in the main text rather than only in the appendix.

Author response:
Thanks for pointing this out. We will make this more explicit in the main text. We picked the three images from different sources (we described in the appendix that one the logo of the visual language model we are using, one generated through stable diffusion, and one from online sources), hoping that the present a good coverage over image types. We happily triple the number of images and provide six more examples in Table 9 (Appendix A.9), including two from the Imagenet dataset, and two more each coming from model generations and online sources. We couldn’t conduct a larger experiment with more images due to time constraints but we hope that this provides a good indicator that the attack method would work generally on many images.

Reviewer comment (Novelty):
The proposed method seems to be straightforward, which is essentially the way of injecting a backdoor. It is unclear how this method can help generalize the trigger to other prompts.

Author response:
We refer the reviewer to the response for Review tv67 above for reasons as to why our contribution matters. To briefly summarize that response, our contribution is in the computer security space – we show a new threat model for multi-modal LLMs. The fact that an attack is straightforward is a big advantage because attackers always choose the path of least resistance. Furthermore, given that there is 10 years of work on improving visual adversarial example attacks, all of that is now applicable to multi-modal LLMs – this is a formidable arsenal for attackers targeting LLMs. We view our work as opening the flood-gates of attacks on LLMs. It also serves as strong motivation to start investing in research to defend against these attacks. 

Table 4 and 5 establishes that our method generalizes to multiple types of prompts and function invocation syntaxes. 
Beyond that, with regards to the method itself, being able to generalize our attack to various or even arbitrary text prompts provided by the victim user with a decent success rate (on desired tool invocation), while maintaining the response and image stealthiness is a non-trivial task itself and has never been accomplished by any prior work in this domain. Additionally, an attack that is not prohibitively complicated in its nature means it’s more dangerous from a security perspective since it’s more easily deployable and more likely to be used by real world attackers.

Reviewer comment (Experiments):
It is unclear what are the implications of these 5 attack objectives. Does the selection of these attack objectives have an impact on the attack performance? What will happen if more attack objectives are included?

Author response:
We chose these five attack objectives to simulate the real-world behavior as realistically as possible by following invocation format from real world systems (see Section 3.1). In fact, the string <function.delete_email which="all"> is the function call format specified in an [older version](https://github.com/microsoft/semantic-kernel/blob/334b22a78460a46b91391c1b41f79e23d55338c2/dotnet/src/Extensions/Planning.SequentialPlanner/skprompt.txt) of Microsoft Semantic Kernel. And the Expedia invocation json is an exact copy of the one generated by ChatGPT (and json is the [standard format](https://platform.openai.com/docs/guides/function-calling) for any plugin calls in ChatGPT). We believe it’s a fair claim that the textual format attack objectives we’ve tested does reflect practical scenarios and demonstrates the potential of our attack if they could be applied on real-world systems. We have revised the paper to clarify.
Also, these five attack objectives represent increasing levels of complexity and difficulty to generate the desired attack and serve as examples to understand whether it is still possible and how easy/difficult it is when the target texts to be generated by LLMs are non natural and complicated.

Reviewer comment (Evaluation):
The evaluation is unsatisfactory. L_p norm is not provided. Also, it is important to show the chances that the malicious tool is triggered when the adversarial example and prompt pair are not present, or only one of these two is present.

Author response:
We additionally provide the L_2 norm and L_inf norm between the adversarial image and its original in Table 4. (Note that Table 5 corresponds to the same adversarial images thus the L_p norms would be the same if provided). Additionally, in appendix Table 7, we provide the L_p norms to understand the quantities when the SSIM score is low; in appendix figure 5, a plot of SSIM values and L_2 norm for experiments in the paper is provided, which shows that SSIM is almost (inverse) proportional to the L_2 norms (as expected).

Regarding the other scenarios of the attack: we are interested in the scenario when the attacker specified tool invocation should appear when the adversarial example is __present__ with an __arbitrary__ prompt provided along with it. When there is no adversarial image provided but only a text prompt, the LLM should respond to the prompt as usual. In the rare case when the user provides no text prompt provided along with the adversarial image, the LLM we tested would respond as if there is a prompt asking “describe this image”, and for several of our trained adversarial images, also generate the tool invocation.

Reviewer comment (Experiments):
**Only a single model LLaMA Adapter is tested.** This makes the scope of the evaluation look somewhat narrow. I suggest the authors also consider other VLMs like Minigpt-4 [1], Instruct-Blip [2], and LLaVA [3]. This can make the evaluation more convincing.

Author response:
We acknowledge this and we are currently working on experiments on other multimodal models. The result will be presented in the updated version soon.

Reviewer comment (Experiments):
**Lack of case studies on real LLM-integrated applications.** The paper mentioned that LangChain and Guidance facilitate the development of such integrations. But, the paper did not provide a single instance of this to illustrate the practical risks of the proposed attack. In the whole paper, what the attack did was just to induce the generation of something like <function.delete_email which="all"> in a purely textual form, which is essentially nothing different from previous NLP attacks that induced certain targeted generations. In my opinion, the novelty of this paper only comes from the illustration of the "practical risks" of such attacks — because the resources controlled by LLMs can now also be missed to induce broader harms. However, the paper did not provide a real example of this. The whole evaluation is still in a purely textual form, judging whether things like <function.delete_email which="all"> are generated. In practice, the LLMs integrated systems may be more complicated than this conceptual form. The paper did not go deep into this.

Author response:
We tried to simulate the real-world behavior as realistically as possible by following invocation format from real world systems (see Section 3.1). In fact, the string <function.delete_email which="all"> is the function call format specified in an [older version](https://github.com/microsoft/semantic-kernel/blob/334b22a78460a46b91391c1b41f79e23d55338c2/dotnet/src/Extensions/Planning.SequentialPlanner/skprompt.txt) of Microsoft Semantic Kernel. And the Expedia invocation json is an exact copy of the one generated by ChatGPT (and json is the [standard format](https://platform.openai.com/docs/guides/function-calling) for any plugin calls in ChatGPT). We believe it’s a fair claim that the textual format attack objectives we’ve tested does reflect practical scenarios and demonstrates the potential of our attack if they could be applied on real-world systems. We have revised the paper to clarify. 
Also, compared to previous work’s targets, a crucial difference of our attack goal is that these tool invocation texts are non natural. We believe it is important to understand whether it is still possible and how easy/difficult it is when the target texts are non natural. Our evaluation provides some answers to this question, with various length of the target text as another dimension.

Reviewer comment (Novelty):
**Inaccurate Literature Review.** There is a factual error in the literature review. Qi et. al. [4] did not show transferability to closed-source models. On the other hand, Carlini et. al. [5] along with Qi et. al. [4] are earlier works showing the usage of visual adversarial examples to hack VLM, which may also need to be noted.

Author response:
Thank you for pointing these out!  We have corrected these errors in the revised version.

Reviewer comment (Experiments):
Can the attack be directly applied to realistic LLM-integrated applications in the wild? Say, other prompt injection attacks such as [1,2] do show realistic instances.

Author response:
We do not have case studies on commercial LLMs with plugins/tools at the time of submission. This work envisions that such attacks could happen for the commercial LLMs if they are integrated with tool use and multimodal support. This envision is probably being more realistic as the recent release of the GPTs series [1], which matches our expectation that LLMs will be integrated with tool use and image understanding [2]. Our attack is a white-box attack, and since the mostly used LLMs (ChatGPT & Bard) do not have their weights open-sourced, we study open-sourced LLMs following real tool use invocation formats (e.g. OpenAI & Microsoft Guidance, see Section 3.1). Our work, therefore, is a case study since the specific strings we're causing the model to generate are real API calls and are harmful API calls. It indicates that if an attacker is able to access the weights of a LLM product with multimodality support and tool invocations, there is a high risk that such an attack can be executed. This work should raise awareness to LLM builders and users, because of the highly malicious threats that could be carried out from the attack (email manipulation, etc).

[1] https://openai.com/blog/introducing-gpts

[2] https://help.openai.com/en/articles/8400551-image-inputs-for-chatgpt-faq

Reviewer comment (Experiments):
This paper assumes that interaction with the tools occurs through an instruction line, followed by normal question answering, as illustrated in Figure 1. Is this setting realistic? What does a real system look like, and how do these VLMs interact with downstream tools like email? Please provide an illustration of why the task in Figure 1 is realistic.

Author response:
Commercial LLMs like ChatGPT and Google Bard support visual images as an input modality by default. Their interaction method assumes an image being uploaded with a prompt alongside, exactly like what we have described in Figure 1. They are also integrating a large number of extensions/plugins which include ones necessary for our attack objective such as email connectors , [google workspace](https://blog.google/products/bard/google-bard-new-features-update-sept-2023/), Expedia, etc. A version of our attack on these models would successfully invoke the plugins/extensions integrated by the victim user. We note that the attacker-desired invocation string can be changed to invoke any other plugin/extension than the 5 attack objectives we have shown in Table 1). **Therefore, our proposed attack scenario is highly realistic even at this moment.**  The trend of more third-party customized versions of GPT ([GPTs](https://openai.com/blog/introducing-gpts)) that will incorporate more powerful tools to LLMs in custom domains will only make our attack more applicable. We have revised the paper to clarify this.

Reviewer comment (Novelty):
This paper lacks technical contributions and depth. The technical contribution of this paper is to generate perturbations on the image side that can prompt the language model to output specific words. However, this paper does not discuss the technical challenges associated with this attack scenario. Instead, they employ a simple gradient-based technique widely adopted in adversarial example attacks. They also do not provide an in-depth analysis of the drawbacks of this technique. For example, is this attack easily bypassed? What is the robustness of this attack?

Author response:
From a computer security perspective, we contribute a new threat model — an attacker can create stealthy images that result in tool misuse. This shows that the past 10 years of attack research on vision adversarial examples can be applied to multi-modal LLMs. This highlights a critical new class of vulnerabilities in LLMs. Furthermore, given that there isn’t an airtight defense for visual adversarial examples, it implies that multimodal LLMs are fundamentally vulnerable to an attack vector with no known good defenses. Furthermore, attackers always prefer the path of least resistance. The simpler an attack technique is, the more likely that attack will actually occur. Thus, we view the simplicity of the attack technique in the paper as an advantage. Finally, from an ML perspective, we observe that a challenge was to balance three competing objectives: (1) getting the LLM to produce a very specific function invocation syntax; (2) Maintaining a natural-semantics conversation with the user; and (3) Keeping the image modifications stealthy.

Reviewer comment (Evaluation):
Additionally, the authors claim that the attack is stealthy because it does not alter the semantic meanings of the output answers. They support this claim with the evidence that the answers under attack are 10% less natural compared to the original ones. My question is, why is a 10% difference considered a small one?

Author response:
During our human evaluation, we present the annotators with responses to the questions generated with and without the image being adversarially perturbed. On the image-unrelated subset, human rated 86% of the responses generated with the unperturbed image as natural, while 77% of the responses generated with the perturbed image are annotated as natural. Here, we concluded that in an absolute value, only about 10% sentences on average would make the user feel more unnatural. But what’s probably equally important is that 77% (37% for image-related) of the responses are natural to humans, therefore they wouldn’t realize something unintended has gone wrong. This is why we believe such an attack is stealthy.

Reviewer comment (Experiments):
If the current setting is realistic, I suggest showing the effectiveness of attacking the real-world system.

Author response:
Attacking an existing system is our current goal. We are working on blackbox transferability to be able to attack the well known systems with no publicly-available weights. But again, one of the goals of this work is to show new attack vectors and threats. Even though we were not able to carry our attack on real-world systems yet, the five attack objective we described in Table 1 are all following tool invocation syntax from real world systems (see Sec 3.1) to simulate the behavior as realistically as possible, which demonstrates the potential of our attack if they could be applied on real-world systems.