Paper: Non-targeted Adversarial Attacks on Vision-Language Models via Maximizing Information Entropy (ICLR-2024)

Reviewer comment (Evaluation):
For example, we now have three methods of MIE, but which one is better? What are their strength and weaknesses? Can we have an analysis and an in-depth discussion?

Author response:
Thank you for recognizing the value of our research topic and the quality of our writing, which encourages us to continue studying this issue. In response to your questions, we provide the following analysis:

1. **Analysis of our attack method.** By inducing the information entropy of the model's logits, attention scores, and hidden states to gradually increase, we can force the model to output texts that do not conform to reality. This is because information entropy is an indicator of the uncertainty of information content. When information entropy increases, it signifies that the model's uncertainty regarding prediction results also increases, thereby reducing the logical coherence of the output text.
     - Implementing a joint attack with different weights for these three losses offers the following advantages:
       - **Multi-angle attack**: By attacking logits, attention scores, and hidden states simultaneously, the model's prediction process can be disrupted from multiple angles. Consequently, even if the model exhibits strong robustness in a particular aspect, it would struggle to withstand attacks from various directions.
       - **Weight adjustment**: By adjusting the weights of different losses, optimization can be achieved according to the specific characteristics and attack objectives of the model. For instance, if the model is sensitive to logits perturbation, the weight of logits loss can be increased to improve the attack's effectiveness.
       - **Interactive influence**: Logits, attention scores, and hidden states interact with each other within the model. A joint attack can exploit this interaction to enhance the attack's effectiveness. For example, by increasing the entropy of attention scores, the model's focus can be dispersed during the prediction process, thereby affecting the computation of hidden states and logits and reducing the logical coherence of the output text.
       - **Stronger attack performance**: Compared to individual attacks, a joint attack can achieve higher attack effectiveness in a shorter time, as it operates simultaneously in multiple directions, improving attack efficiency.
     - All three methods are relatively effective, with logits-based attacks appearing to be the best from the perspective of attack success rate. However, from the standpoint of adversarial training, models may find it more challenging to defend against joint attacks. The advantages and disadvantages of executing these three methods individually are as follows:
       - **Logits-based:**
         
         **Advantages**: Directly affects the model's output layer, enabling rapid alteration of the model's prediction results within a short time. For some models, this attack method may yield better results.
         Disadvantages: Targets only the output layer, potentially failing to fully exploit other information within the model. Additionally, if the model possesses strong robustness, larger perturbations may be required to achieve the desired attack effect.
       - **Attention-based:**
         
         **Advantages**: By altering the model's attention distribution, the model can be forced to focus on irrelevant or insignificant information, thus reducing the logical coherence of the output text. This method leverages the model's internal attention mechanism, making the attack more targeted.
         Disadvantages: The attack's effectiveness may be influenced by the model's structure and parameter settings. For some models, altering the attention distribution may not be sufficient to produce significant attack effects.
       - **Hidden states-based:**
         
         **Advantages**: Directly affects the model's internal representations, allowing for deeper disruption of the model's prediction process. This method may yield substantial attack effects with relatively small perturbations.
         Disadvantages: The attack's effectiveness may be influenced by the model's complexity and robustness. Furthermore, due to the highly abstract nature of the model's internal representations, determining an effective attack strategy may be challenging.

Reviewer comment (Experiments):
What factors can influence the performance of this method?

Author response:
2. **Factors affecting the performance of MIE.** The main factors affecting this method include perturbation size, the number of iterations, and different loss ratio values. The ablation experiment results for this part are as follows: 
     - **Perturbation size**: As the perturbation increases, the attack effect becomes stronger. However, when the perturbation exceeds 8, the improvement in attack effect becomes weaker.
     - **Steps of iterations**: As the number of iterations increases, the attack effect gradually strengthens. As shown in Appendix A.1, with an increased number of iterations, the model may even generate meaningless descriptions.
     - **Different loss ratios**: On Blip, with $\lambda_1$ fixed at 0.5 and varying $\lambda_2,\lambda_3$, the results can be visualized using a heatmap, which reveals that the optimal resutls are concentrated around the ratio of $\lambda_1:\lambda_2:\lambda_3=0.5:0.06:0.06(\approx 8:1:1)$
         | $\epsilon ( 1/255)$ | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
         | --- | --- | --- | --- | --- | --- | --- | --- |
         | MIE | 28.01 | 23.83 | 20.67 | 17.80 | 17.79 | 17.77 | 17.99 |

         | Steps | 5 | 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | 90 | 100 |
         | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
         | MIE | 21.02 | 19.92 | 18.84 | 19.31 | 18.75 | 18.64 | 18.45 | 18.28 | 18.18 | 17.95 | 17.80 |
         
         | $\lambda_2$,$\lambda_3$ | 0.025 | 0.05 | 0.06 | 0.1 | 0.2 | 0.5 | 0.75 | 1 |
         | --- | --- | --- | --- | --- | --- | --- | --- | --- |
         | 0.025 | 18.85 | 18.34 | 18.99 | 17.98 | 18.39 | 19.09 | 19.60 | 19.65 |
         | 0.05 | 18.90 | 18.68 | 18.35 | 19.13 | 18.06 | 18.59 | 19.30 | 19.79 |
         | 0.06 | 18.99 | 18.07 |**17.75** | 18.25 | 18.22 | 18.25 | 18.67 | 19.14 |
         | 0.1 | 19.53 | 18.99 | 18.18 | 18.10 | 18.37 | 18.70 | 18.98 | 19.48 |
         | 0.2 | 19.02 | 18.63 | 18.36 | 18.87 | 18.97 | 18.48 | 18.56 | 19.29 |
         | 0.5 | 20.35 | 18.95 | 18.78 | 18.24 | 18.24 | 18.80 | 19.03 | 20.67 |
         | 0.75 | 20.15 | 19.22 | 19.44 | 19.37 | 19.33 | 19.25 | 19.29 | 20.76 |
         | 1 | 20.86 | 20.30 | 20.25 | 20.69 | 20.26 | 20.19 | 20.78 | 20.00 |

Reviewer comment (Evaluation):
Is this attack robust against simple defensive methods like robust training?

Author response:
3. **Effectiveness of adversarial training.** Following conventional adversarial training, we find that large VLMs do not exhibit significant adversarial robustness for unseen samples. Our MIE attack based on autoregressive generation of pseudo-labels has a strong attack capability. We also compared it with some other methods, inlcuding  Carlini 2023 (performing targeted attacks using random targets), Schlarmann 2023 (utilizing descriptions of clean samples as ground-truth labels), and Nayyer 2022 ( a GAN-based method). Note that these attacks did not use the exact same experimental settings. The results across different models also show that the adversarial robustness of existing large VLMs is generally weak. In the future, we will delve into the effectiveness of adversarial training for VLMs.
      | Model | Clean | Carlini2023[1] | Schlarmann2023[2] | Nayyer2022[3] | MIE |
      | --- | --- | --- | --- | --- | --- | --- |
      | BLIP | 29.79 | 20.53 | 19.87 | 24.36 | **17.80**|
      | BLIP-2 | 30.72 | 24.58 | 24.06 | 27.78 | **21.39** |
      | InstructBLIP | 31.36 | 24.31 | 23.80 | 25.32 | **21.65** |
      | LLaVA | 31.52 | 24.78 | 24.12 | 25.79 | **21.41** |
      | MiniGPT-4 | 31.44 | 24.97 | 23.16 | 24.12 | **21.11** |
In summary, our proposed MIE method can be applied to various Transformer-based VLMs. With its ease of use and effectiveness, it can serve as a benchmark for evaluating VLMs' robutness.

**The relevant experimental analysis will be updated in the paper.**

[1] Carlini, Nicholas, et al. "Are aligned neural networks adversarially aligned?." arXiv preprint arXiv:2306.15447 (2023).

[2] Schlarmann, Christian, and Matthias Hein. "On the adversarial robustness of multi-modal foundation models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Aafaq, Nayyer, et al. "Language model agnostic gray-box adversarial attack on image captioning." IEEE Transactions on Information Forensics and Security 18 (2022): 626-638.

Reviewer comment (Experiments):
Comprehensive evaluation on parameter different settings is needed. For now, the evaluation is limited, e.g., only one attack strength is evaluated.

Author response:
2. **Factors affecting the performance of MIE.** The main factors affecting this method include perturbation size, the number of iterations, and different loss ratio values. The ablation experiment results for this part are as follows: 
     - **Perturbation size**: As the perturbation increases, the attack effect becomes stronger. However, when the perturbation exceeds 8, the improvement in attack effect becomes weaker.
     - **Steps of iterations**: As the number of iterations increases, the attack effect gradually strengthens. As shown in Appendix A.1, with an increased number of iterations, the model may even generate meaningless descriptions.
     - **Different loss ratios**: On Blip, with $\lambda_1$ fixed at 0.5 and varying $\lambda_2,\lambda_3$, the results can be visualized using a heatmap, which reveals that the optimal resutls are concentrated around the ratio of $\lambda_1:\lambda_2:\lambda_3=0.5:0.06:0.06(\approx 8:1:1)$
         | $\epsilon ( 1/255)$ | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
         | --- | --- | --- | --- | --- | --- | --- | --- |
         | MIE | 28.01 | 23.83 | 20.67 | 17.80 | 17.79 | 17.77 | 17.99 |

         | Steps | 5 | 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | 90 | 100 |
         | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
         | MIE | 21.02 | 19.92 | 18.84 | 19.31 | 18.75 | 18.64 | 18.45 | 18.28 | 18.18 | 17.95 | 17.80 |
         
         | $\lambda_2$,$\lambda_3$ | 0.025 | 0.05 | 0.06 | 0.1 | 0.2 | 0.5 | 0.75 | 1 |
         | --- | --- | --- | --- | --- | --- | --- | --- | --- |
         | 0.025 | 18.85 | 18.34 | 18.99 | 17.98 | 18.39 | 19.09 | 19.60 | 19.65 |
         | 0.05 | 18.90 | 18.68 | 18.35 | 19.13 | 18.06 | 18.59 | 19.30 | 19.79 |
         | 0.06 | 18.99 | 18.07 |**17.75** | 18.25 | 18.22 | 18.25 | 18.67 | 19.14 |
         | 0.1 | 19.53 | 18.99 | 18.18 | 18.10 | 18.37 | 18.70 | 18.98 | 19.48 |
         | 0.2 | 19.02 | 18.63 | 18.36 | 18.87 | 18.97 | 18.48 | 18.56 | 19.29 |
         | 0.5 | 20.35 | 18.95 | 18.78 | 18.24 | 18.24 | 18.80 | 19.03 | 20.67 |
         | 0.75 | 20.15 | 19.22 | 19.44 | 19.37 | 19.33 | 19.25 | 19.29 | 20.76 |
         | 1 | 20.86 | 20.30 | 20.25 | 20.69 | 20.26 | 20.19 | 20.78 | 20.00 |

Reviewer comment (Experiments):
Lack of comparative methods. Many methods have already been proposed for attacking VLMs[1], but the authors did not compare with these methods during the experimental phase, choosing only simple Gaussian noise for comparison.

Author response:
1. **Comparison with relevant and feasible works.** We have added some comparative experiments, including Carlini 2023 (performing targeted attacks using random targets), Schlarmann 2023 (utilizing descriptions of clean samples as ground-truth labels), and Nayyer 2022 ( a GAN-based method). The results show that our MIE method is more effective, taking full advantage of the non-targeted nature to increase the perturbation space. As for Yunqing2023 [4], it is a targeted black-box attack with strong dependency on the target text, which is significantly different from our experimental setting; thus, we dot not include it in the comparison. 
   | Model | Clean | Carlini2023[1] | Schlarmann2023[2] | Nayyer2022[3] | MIE |
   | --- | --- | --- | --- | --- | --- |
   | BLIP | 29.79 | 20.53 | 19.87 | 24.36 | **17.80**|
   | BLIP-2 | 30.72 | 24.58 | 24.06 | 27.78 | **21.39** |
   | InstructBLIP | 31.36 | 24.31 | 23.80 | 25.32 | **21.65** |
   | LLaVA | 31.52 | 24.78 | 24.12 | 25.79 | **21.41** |
   | MiniGPT-4 | 31.44 | 24.97 | 23.16 | 24.12 | **21.11** |

Reviewer comment (Theory):
The approach in the paper is somewhat ad-hoc. There are various ways to disrupt the expressions in network layers, such as maximizing the norm of mid-layer features. Why did the authors choose to maximize entropy for the attack? The rationale behind this was not clarified in the paper.

Author response:
2. **Reason for choosing maximizing entropy as the attack method.**  We would like to emphasize that the reason for selecting the maximization of entropy as the optimization objective is to make the model's understanding of the input image more confusing. By maximizing information entropy, we are actually pursuing the maximum confusion in the model's internal information during answer generation, thereby increasing the model's uncertainty. Although there are indeed multiple ways to calculate loss, such as maximizing the norm, not all of these methods are directly related to our attack starting point. For example, the focus of maximizing the norm is on the overall magnitude of the model's internal outputs rather than uncertainty. Although in some cases, maximizing the norm may lead to increased uncertainty in the model's internal information, this connection is not as direct and clear as with entropy. We have added experiments based on the norm, and the results show that MIE has a stronger attack effect. 
   | Model | Clean | Norm-based | MIE |
   | --- | --- | --- | --- |
   | BLIP | 29.79 | 20.81 | **17.80** |
   | BLIP-2 | 30.72 | 22.97 | **21.39** |
   | InstructBLIP | 31.36 | 23.74 | **21.65** |
   | LLaVA | 31.52 | 24.53 | **21.41** |
   | MiniGPT-4 | 31.44 | 23.65 | **21.11** |

Reviewer comment (Experiments):
Absence of ablation studies. The final attack method in the paper is composed of three losses, but the authors did not discuss the impact of different loss coefficients on the results within the article.

Author response:
3. **Ablation study on loss coefficients.** Table 1 in the paper shows the results of individual attacks in the last four columns. For the BLIP model, we conduct ablation experiments by fixing $\lambda_1=0.5$ and varying $\lambda_2$ and $\lambda_3$. The results can be visualized using a heatmap, which reveals that the optimal resutls are concentrated around the ratio of $\lambda_1:\lambda_2:\lambda_3=0.5:0.06:0.06(\approx 8:1:1)$. **Additional ablation experiments can be found in Reviewer 4's comments.**
   | $\lambda_2$,$\lambda_3$ | 0.025 | 0.05 | 0.06 | 0.1 | 0.2 | 0.5 | 0.75 | 1 |
   | --- | --- | --- | --- | --- | --- | --- | --- | --- |
   | 0.025 | 18.85 | 18.34 | 18.99 | 17.98 | 18.39 | 19.09 | 19.60 | 19.65 |
   | 0.05 | 18.90 | 18.68 | 18.35 | 19.13 | 18.06 | 18.59 | 19.30 | 19.79 |
   | 0.06 | 18.99 | 18.07 |**17.75** | 18.25 | 18.22 | 18.25 | 18.67 | 19.14 |
   | 0.1 | 19.53 | 18.99 | 18.18 | 18.10 | 18.37 | 18.70 | 18.98 | 19.48 |
   | 0.2 | 19.02 | 18.63 | 18.36 | 18.87 | 18.97 | 18.48 | 18.56 | 19.29 |
   | 0.5 | 20.35 | 18.95 | 18.78 | 18.24 | 18.24 | 18.80 | 19.03 | 20.67 |
   | 0.75 | 20.15 | 19.22 | 19.44 | 19.37 | 19.33 | 19.25 | 19.29 | 20.76 |
   | 1 | 20.86 | 20.30 | 20.25 | 20.69 | 20.26 | 20.19 | 20.78 | 20.00 |

Reviewer comment (Novelty):
The technical contribution is on the moderate side. However, to the best of my knowledge using maximum entropy to adversarial attack is novel.

Author response:
Thank you for acknowledging the novelty of our attack method. In response to your concerns, we provide the following analysis.

1. **Technical contributions.** Existing attacks on large Very Large Models (VLMs) generally adopt similar approaches, utilizing teacher forcing to calculate the loss of target texts. These methods are relatively simple but demonstrate a certain level of effectiveness. Our method based on maximizing information entropy provides a new attack perspective and is highly effective. Precisely because the technical implementation of MIE is relatively straightforward, it can encourage more researchers to focus on the adversarial robustness of VLMs.

Reviewer comment (Experiments):
The paper lacks comparison to related work. There are several attacks on image captioning in the literature. The paper lacks a comparison against them (see for example, [1] and [2]). The authors mention [2] in the related work but did not empirically compare against it, they justify it as this method uses the original caption. This, however, is a minor requirement as it can be mitigated by treating the caption on the clean image as the ground-truth caption.

Author response:
2. **Comparison with relevant and feasible works.** Below are our comparative experimental results inlcuding  Carlini 2023 (performing targeted attacks using random targets), Schlarmann 2023 (utilizing descriptions of clean samples as ground-truth labels), and Nayyer 2022 ( a GAN-based method).:
   | Model | Clean | Carlini2023[1] | Schlarmann2023[2] | Nayyer2022[3] | MIE |
   | --- | --- | --- | --- | --- | --- |
   | BLIP | 29.79 | 20.53 | 19.87 | 24.36 | **17.80**|
   | BLIP-2 | 30.72 | 24.58 | 24.06 | 27.78 | **21.39** |
   | InstructBLIP | 31.36 | 24.31 | 23.80 | 25.32 | **21.65** |
   | LLaVA | 31.52 | 24.78 | 24.12 | 25.79 | **21.41** |
   | MiniGPT-4 | 31.44 | 24.97 | 23.16 | 24.12 | **21.11** |

Reviewer comment (Experiments):
There are hyper-parameters for the attacks $\lambda_1,\lambda_2,\lambda_3$. Are these selected based on the same validation set? If so I believe the experiment might lack statistical integrity.

Author response:
3. **Selection of the validation set.** Our validation set comprises 1,000 randomly selected images, which inherently provide a certain level of generalization. This selection method is employed to maintain consistency with related works in the adversarial attack domain. Furthermore, we conduct ablation experiments by fixing $\lambda_1=0.5$ and varying $\lambda_2$ and $\lambda_3$ for the BLIP model. The results can be visualized using a heatmap, which reveals that the optimal resutls are concentrated around the ratio of $\lambda_1:\lambda_2:\lambda_3=0.5:0.06:0.06(\approx 8:1:1)$. **Additional ablation experiments can be found in Reviewer 4's comments.**
   | $\lambda_2$,$\lambda_3$ | 0.025 | 0.05 | 0.06 | 0.1 | 0.2 | 0.5 | 0.75 | 1 |
   | --- | --- | --- | --- | --- | --- | --- | --- | --- |
   | 0.025 | 18.85 | 18.34 | 18.99 | 17.98 | 18.39 | 19.09 | 19.60 | 19.65 |
   | 0.05 | 18.90 | 18.68 | 18.35 | 19.13 | 18.06 | 18.59 | 19.30 | 19.79 |
   | 0.06 | 18.99 | 18.07 |**17.75** | 18.25 | 18.22 | 18.25 | 18.67 | 19.14 |
   | 0.1 | 19.53 | 18.99 | 18.18 | 18.10 | 18.37 | 18.70 | 18.98 | 19.48 |
   | 0.2 | 19.02 | 18.63 | 18.36 | 18.87 | 18.97 | 18.48 | 18.56 | 19.29 |
   | 0.5 | 20.35 | 18.95 | 18.78 | 18.24 | 18.24 | 18.80 | 19.03 | 20.67 |
   | 0.75 | 20.15 | 19.22 | 19.44 | 19.37 | 19.33 | 19.25 | 19.29 | 20.76 |
   | 1 | 20.86 | 20.30 | 20.25 | 20.69 | 20.26 | 20.19 | 20.78 | 20.00 |

Reviewer comment (Experiments):
An alternate way would be to simply feed some random image that is clearly maximum entropy to the VLM, get a few words of text from it (essentially a random phrase) and then use a targeted attack on the VLM to generate that particular or similar text. I would have liked to see a comparison with such an approach in the paper.

Author response:
**Feasibility of targeted attacks using random targets.** While it is computationally feasible to implement non-targeted attacks using targeted attack methods as subroutines, it is insufficient for  image description tasks. Unlike traditional classification tasks, images do not have only one correct description, especially for complex images with multiple description angles. For instance, "it's a sunny day" and "the sunshine is so bright" are textually inconsistent but semantically equivalent. Therefore, the success rate of such random targeted attacks is relatively low. Below are our comparative experimental results inlcuding  Carlini 2023 (performing targeted attacks using random targets), Schlarmann 2023 (utilizing descriptions of clean samples as ground-truth labels), and Nayyer 2022 ( a GAN-based method): 
      | Model | Clean | Carlini2023[1] | Schlarmann2023[2] | Nayyer2022[3] | MIE |
      | --- | --- | --- | --- | --- | --- |
      | BLIP | 29.79 | 20.53 | 19.87 | 24.36 | **17.80**|
      | BLIP-2 | 30.72 | 24.58 | 24.06 | 27.78 | **21.39** |
      | InstructBLIP | 31.36 | 24.31 | 23.80 | 25.32 | **21.65** |
      | LLaVA | 31.52 | 24.78 | 24.12 | 25.79 | **21.41** |
      | MiniGPT-4 | 31.44 | 24.97 | 23.16 | 24.12 | **21.11** |

Reviewer comment (Experiments):
As an example, couldn't "a pair of flip flops sitting on a pile of garbage" in Appendix A.1 first row be set as the target caption in Carlini 2023 and then a perturbation found that achieves that? Or any other random caption for that matter?

Author response:
**Feasibility of targeted attacks using random targets.** While it is computationally feasible to implement non-targeted attacks using targeted attack methods as subroutines, it is insufficient for  image description tasks. Unlike traditional classification tasks, images do not have only one correct description, especially for complex images with multiple description angles. For instance, "it's a sunny day" and "the sunshine is so bright" are textually inconsistent but semantically equivalent. Therefore, the success rate of such random targeted attacks is relatively low. Below are our comparative experimental results inlcuding  Carlini 2023 (performing targeted attacks using random targets), Schlarmann 2023 (utilizing descriptions of clean samples as ground-truth labels), and Nayyer 2022 ( a GAN-based method): 
      | Model | Clean | Carlini2023[1] | Schlarmann2023[2] | Nayyer2022[3] | MIE |
      | --- | --- | --- | --- | --- | --- |
      | BLIP | 29.79 | 20.53 | 19.87 | 24.36 | **17.80**|
      | BLIP-2 | 30.72 | 24.58 | 24.06 | 27.78 | **21.39** |
      | InstructBLIP | 31.36 | 24.31 | 23.80 | 25.32 | **21.65** |
      | LLaVA | 31.52 | 24.78 | 24.12 | 25.79 | **21.41** |
      | MiniGPT-4 | 31.44 | 24.97 | 23.16 | 24.12 | **21.11** |

Reviewer comment (Evaluation):
Wouldn't the captions just become random sequence of words as the attack progresses? The figures in Appendix A should probably illustrate that, if that is the case. And if that is the case, is such an attack considered successful? Or should the caption generally make sense to a human?

Author response:
**Non-targeted attacks pose weaker constraints compared to targeted attacks.**
   - Non-targeted attacks do not aim to make VLMs produce specific outputs, resulting in a larger solution space for numerical optimization. This is also a reason why targeted attacks may be less effective. 
   - MIE can serve as a powerful method for assessing the robustness of VLMs. As the attack progresses, the model's output may become nonsensical, consisting of repetitive random words. This indicates that the VLM's robustness is insufficient, as it cannot generate coherent sentences, which in turn diminishes user experience and weakens user trust. In scenarios with high security requirements, the lack of robustness can lead to severe consequences.
   - We also experimentally found that conventional adversarial training cannot defend against our attack, indicating that current VLMs are extremely vulnerable to our attack method.

Reviewer comment (Novelty):
Can the Intro be revised to identify the closest work to the paper and identify the key contribution over that work (or minor variants of prior work)?

Author response:
**Key Contribution.** We propose the MIE method, which focuses on  non-targeted attacks, characterized by weaker constraints and a larger optimization space. Unlike previous methods, such as [1] and [2], which involve targeted optimization of adversarial samples towards specific directions, our approach is more suitable for image description tasks. Supplementary experiments demonstrate that the attack performance of MIE surpasses that of comparative methods. We plan to revise and upload a new version of the paper shortly. MIE is the first untargeted attack method that does not rely on the true descriptions of images, exhibiting strong generalization capabilities.

Reviewer comment (Experiments):
Why can't untargeted attacks use a method for targeted attacks as a subroutine to achieve an untargeted attack? Note that there is precedence for this in adversarial ML. In the OPT method for blackbox adversarial attacks, the fundamental method is for a targeted attack. The OPT paper discusses how to do an untargeted attack by wrapping a small amount of code around a targeted attack. An example strategy would be to choose a random target and then attempt a targeted attack.

Author response:
**Feasibility of targeted attacks using random targets.** While it is computationally feasible to implement non-targeted attacks using targeted attack methods as subroutines, it is insufficient for  image description tasks. Unlike traditional classification tasks, images do not have only one correct description, especially for complex images with multiple description angles. For instance, "it's a sunny day" and "the sunshine is so bright" are textually inconsistent but semantically equivalent. Therefore, the success rate of such random targeted attacks is relatively low. Below are our comparative experimental results inlcuding  Carlini 2023 (performing targeted attacks using random targets), Schlarmann 2023 (utilizing descriptions of clean samples as ground-truth labels), and Nayyer 2022 ( a GAN-based method): 
      | Model | Clean | Carlini2023[1] | Schlarmann2023[2] | Nayyer2022[3] | MIE |
      | --- | --- | --- | --- | --- | --- |
      | BLIP | 29.79 | 20.53 | 19.87 | 24.36 | **17.80**|
      | BLIP-2 | 30.72 | 24.58 | 24.06 | 27.78 | **21.39** |
      | InstructBLIP | 31.36 | 24.31 | 23.80 | 25.32 | **21.65** |
      | LLaVA | 31.52 | 24.78 | 24.12 | 25.79 | **21.41** |
      | MiniGPT-4 | 31.44 | 24.97 | 23.16 | 24.12 | **21.11** |

Reviewer comment (Novelty):
Why are untargeted attacks on VLMs particularly interesting, given that we already know that targeted attacks are possible and how to do them?

Author response:
**Non-targeted attacks pose weaker constraints compared to targeted attacks.**
   - Non-targeted attacks do not aim to make VLMs produce specific outputs, resulting in a larger solution space for numerical optimization. This is also a reason why targeted attacks may be less effective. 
   - MIE can serve as a powerful method for assessing the robustness of VLMs. As the attack progresses, the model's output may become nonsensical, consisting of repetitive random words. This indicates that the VLM's robustness is insufficient, as it cannot generate coherent sentences, which in turn diminishes user experience and weakens user trust. In scenarios with high security requirements, the lack of robustness can lead to severe consequences.
   - We also experimentally found that conventional adversarial training cannot defend against our attack, indicating that current VLMs are extremely vulnerable to our attack method.

Reviewer comment (Experiments):
Can an ablation study be presented to show how each entropy method performs on its own (e.g., lambda_1 = 1, others 0, etc.), versus the chosen setting for lambdas of (0.8, 0.1., 0.1).

Author response:
**Ablation study on loss coefficients.** Table 1 in the paper shows the results of individual attacks in the last four columns. For the BLIP model, we conduct ablation experiments by fixing $\lambda_1=0.5$ and varying $\lambda_2$ and $\lambda_3$. The results can be visualized using a heatmap, which reveals that the optimal resutls are concentrated around the ratio of $\lambda_1:\lambda_2:\lambda_3=0.5:0.06:0.06(\approx 8:1:1)$. **Additional ablation experiments can be found in Reviewer 4's comments.**
      | $\lambda_2$,$\lambda_3$ | 0.025 | 0.05 | 0.06 | 0.1 | 0.2 | 0.5 | 0.75 | 1 |
      | --- | --- | --- | --- | --- | --- | --- | --- | --- |
      | 0.025 | 18.85 | 18.34 | 18.99 | 17.98 | 18.39 | 19.09 | 19.60 | 19.65 |
      | 0.05 | 18.90 | 18.68 | 18.35 | 19.13 | 18.06 | 18.59 | 19.30 | 19.79 |
      | 0.06 | 18.99 | 18.07 |**17.75** | 18.25 | 18.22 | 18.25 | 18.67 | 19.14 |
      | 0.1 | 19.53 | 18.99 | 18.18 | 18.10 | 18.37 | 18.70 | 18.98 | 19.48 |
      | 0.2 | 19.02 | 18.63 | 18.36 | 18.87 | 18.97 | 18.48 | 18.56 | 19.29 |
      | 0.5 | 20.35 | 18.95 | 18.78 | 18.24 | 18.24 | 18.80 | 19.03 | 20.67 |
      | 0.75 | 20.15 | 19.22 | 19.44 | 19.37 | 19.33 | 19.25 | 19.29 | 20.76 |
      | 1 | 20.86 | 20.30 | 20.25 | 20.69 | 20.26 | 20.19 | 20.78 | 20.00 |