Paper: Ceci n'est pas une pomme: Adversarial Illusions in Multi-Modal Embeddings (ICLR-2024)

Reviewer comment (Novelty):
For example, BadEncoder (S&P'22) already implements such an idea to attack the CLIP model.

Author response:
BadEncoder is a training-time attack, it assumes that the attacker poisons the training of the model.  In the case of ImageBind or CLIP, it could be used only when the models were trained.  Ours is an evaluation-time attack on the original, unmodified and unpoisoned ImageBind.  It does not involve injecting anything into the training data and does not assume that the attacker has access to the model during training.  We will clarify this fundamental difference between the threat models of backdoor attacks (like BadEncoder) and adversarial inputs (like ours) in Background and Related Work.

Reviewer comment (Novelty):
There are also many following works which cite BadEncoder and study the vulnerability of multi-modal embeddings.

Author response:
BadEncoder is a training-time attack, it assumes that the attacker poisons the training of the model.  In the case of ImageBind or CLIP, it could be used only when the models were trained.  Ours is an evaluation-time attack on the original, unmodified and unpoisoned ImageBind.  It does not involve injecting anything into the training data and does not assume that the attacker has access to the model during training.  We will clarify this fundamental difference between the threat models of backdoor attacks (like BadEncoder) and adversarial inputs (like ours) in Background and Related Work.

Reviewer comment (Novelty):
How do the authors position the novelty of this work in these literatures?

Author response:
BadEncoder is a training-time attack, it assumes that the attacker poisons the training of the model.  In the case of ImageBind or CLIP, it could be used only when the models were trained.  Ours is an evaluation-time attack on the original, unmodified and unpoisoned ImageBind.  It does not involve injecting anything into the training data and does not assume that the attacker has access to the model during training.  We will clarify this fundamental difference between the threat models of backdoor attacks (like BadEncoder) and adversarial inputs (like ours) in Background and Related Work.

Reviewer comment (Experiments):
The paper proposes to use the I-FGSM attack to create their adversarial illusions, but it is unclear why I-FGSM is used instead of PGD, which is the best-known and better attack.

Author response:
In general, PGD refers to I-FGSM with a random initialization in an $\ell_\infty$ ball around the source image. This method may perform marginally better for “conventional” adversarial examples, in our setting this benefit is greatly diminished since adversarial alignment must only ‘beat’ organic alignment. We will add an experiment confirming this observation in the **Appendix**.

Reviewer comment (Experiments):
The authors could also experiment with DiffPGD, a newer PGD attack based on the diffusion model [1]. The adversarial perturbation is really visible in Figure 7, and leveraging diffusion models could improve the attack.

Author response:
The image in question uses the perturbation bound of $\epsilon = \frac{16}{256}$. As we reported elsewhere in the submission, our attack is effective with much lower bounds. We regenerated the image with a lower bound and it shows no visible noise. While DiffPGD is fascinating, it appears unnecessary in our case since our attack already achieves the desired adversarial alignment with tiny perturbations (DiffPGD can only improve image quality, not performance).

Reviewer comment (Experiments):
The choice of using the $\ell_\infty$ norm is not explained. Have the authors experimented with other norms (e.g. $\ell_2$)?

Author response:
As with the above, the $\ell_\infty$ norm (the most common in the literature) is sufficient for an attack that achieves all its objectives, i.e., strong adversarial alignment that fools all downstream tasks. We can add some experiments on other $\ell_p$ norms into the **Appendix**.

Reviewer comment (Experiments):
The authors seem to have experimented only with cross-modality. Can the authors create adversarial illusions on the same modality?

Author response:
Same-modality adversarial examples are strictly easier than cross-modal illusions, because, as discussed in Cross-Modal Illusions, the modality gap only arises between modalities. In particular, the unbounded case can be trivially solved within the same modality (i.e., $\delta = \mathbf{a}^{m_2} - x^{m_1}$) whereas a solution may not exist for the cross-modal case. We will provide some experiments in the Appendix.

Reviewer comment (Experiments):
Is it possible to investigate the transferability of the attack, e.g., against other multimodal foundation models?

Author response:
To expand the evaluation, we have shown that our attack is effective on AudioCLIP (with a 100% success rate for $\epsilon$ larger than or equal to $\frac{4}{256}$. In addition, we are conducting experiments on the transferability between embedding spaces. The experiments will be added to our **Evaluation** section.

Reviewer comment (Experiments):
The authors seem to be experimenting only with targeted attacks. Would it be possible to maximize the following loss: 
$$
\ell = 1 - \cos\left( \theta^{m_1}(x^{m_1}+\delta), \theta^{m_1}(x^{m_1}) \right)
$$

Author response:
The targeted attacks we present in the paper are strictly harder than the suggested untargeted attacks. Rather than perturb images to induce *any* non-source class, we align with a *specific* non-source class. The suggested loss function reduces to finding any orthogonal vector in the unbounded case. Since we are in high-dimensional space, sampling randomly is likely to produce an orthogonal vector, and can be computed far more efficiently. In the bounded case, empirically, optimizing our suggested attack with *any* target until the alignment between the perturbed input and the source is less than the organic alignment is sufficient. This can be achieved with extremely small perturbation bounds.

Reviewer comment (Presentation):
The difference between Figures 1, 3, 4, and 5 is unclear, as it seems only the modality and examples are different. These figures take up a lot of space in the paper, and they could be reduced or some of them could be put in the appendix to leave space for further experiments.

Author response:
Thank you for the feedback.   We will move some examples to the Appendix.

Reviewer comment (Experiments):
The authors devote a whole section to countermeasures, but it seems that they do not do any experiments. If the authors focus so much on countermeasures, some experiments should be done.

Author response:
**Add JPEG experiments** Our **Countermeasures** section has been updated to show how our attack evades the JPEG compression defense, a common method to mitigate adversarial perturbations. The JPEG-resistant attack still achieves a $75\\%$ success rate in the presence of the defense. It also evades the defense that checks consistency of embeddings of different input augmentations.  As we note in the paper, there are limitations to most traditional defenses. Developing new defenses specifically for embeddings is an interesting research direction.

Reviewer comment (Experiments):
The main weakness of this paper is the incompleteness of the experiments, with a lack of experiments involving other models and an insufficient variety of experimental tasks.

Author response:
1. To expand the evaluation, we have shown that our attack is effective on AudioCLIP (with a 100% success rate for $\epsilon$ larger than 4. In addition, we are conducting experiments on the transferability between embedding spaces. The experiments will be added to our **Evaluation** section.

Reviewer comment (Experiments):
Using just one multi-modal model, ImageBind, in the experiments to demonstrate that “multi-modal embeddings can be vulnerable to an attack” may be somewhat insufficient. Conducting experiments on AudioCLIP[1], another contrastively pre-trained multi-modal model, would make the claim more convincing.

Author response:
Thank you for the feedback, please see the attached table (based of Table 1 measuring zero-shot accuracy) with comparison to AudioCLIP. Overall, we have observed a similar behavior and the strength of our attack. 

| Alignment     | $\epsilon_V$ | Top-1 (ImageBind) | $\mathrm{align}(\cdot)$ (ImageBind) | Top-1 (AudioCLIP) | $\mathrm{align}(\cdot)$ (AudioCLIP) |
|---------------|--------------|-------------------|------------------------------------|-------------------|-------------------------------------|
| Organic       | -            | 67%               | 0.2878 ± 0.005                     | 27%               | 0.1445 ± 0.003                      |
| Adversarial   | 0            | 0%                | 0.0892 ± 0.005                     | 0%                | 0.0670 ± 0.003                      |
|               | 1            | 93%               | 0.5741 ± 0.015                     | 90%               | 0.3560 ± 0.008                      |
|               | 4            | 100%              | 0.8684 ± 0.006                     | 100%              | 0.6510 ± 0.009                      |
|               | 8            | 100%              | 0.9241 ± 0.004                     | 100%              | 0.7323 ± 0.007                      |
|               | 16           | 100%              | 0.9554 ± 0.002                     | 100%              | 0.7841 ± 0.006                      |
|               | 32           | 100%              | 0.9692 ± 0.001                     | 100%              | 0.8074 ± 0.006                      |

We will integrate these results into the paper.

Reviewer comment (Experiments):
I'm uncertain about why black-box attacks cannot be applied in this context. For instance, images and mel spectrograms can still be misclassified into the target input by introducing specific noises calculated by SimBA[2]. Some further clarification regarding the limitations or inapplicability of black-box attacks in this context would be helpful.

Author response:
1. Black-box attacks are interesting future work but there are some inherent challenges to this formulation that we will add to the **Limitations** section. Consider two alternate definitions of ‘black-box’: (1) the attacker has no access to the internal representations of the encoder, but does have access to the resulting embeddings, and (2) the attacker has only access to the outputs of a specific downstream task. Recall that our attack is targeted.
    
    For (1), satisfying the classification constraint (i.e., changing the label to a specific target) is a significantly easier task than finding an image that is arbitrarily close, in a high-dimensional embedding space, to a specific target embedding. Modern black-box techniques that rely on Bayesian Optimization are particularly ill-suited for this high-dimensional objective. That said, this is an interesting topic for future work.
    
    For (2), while it is possible that an adversarial example generated for a specific downstream task may generalize to others, the representation-merging of the example and its target could happen in the downstream model as opposed to the encoder, making generalization to all downstream tasks hard to achieve. This is also a promising direction for future work.

Reviewer comment (Experiments):
Are the other white-box attacks as effective as I-FGSM in performing adversarial illusions?

Author response:
2. We demonstrate that in our setting, I-FGSM’s performance is essentially perfect, achieving high adversarial alignment (much higher than organic alignment) with tiny perturbations.  We are not sure about the motivation for exploring other methods but can evaluate PGD if requested.

Reviewer comment (Experiments):
The experiments are conducted using ImageBind, a model that is contrastively pre-trained and projects all modality data into an image embedding space. If ImageBind were replaced with a multi-modal model like ChatBridge[3], which is not contrastively pre-trained and projects all modality data into a text embedding space, would this still demonstrate the vulnerability of multi-modal embeddings to the adversarial illusion attack?

Author response:
3. Given the strength of our adversarial alignment, we expect that any system that relies on a unified embedding space, regardless of whether it was contrastively trained, will be vulnerable to attack. In particular, our attack relies on the former as opposed to the latter.

Reviewer comment (Theory):
(Minor) Is the I-FGSM formula complete? I think a clipping operation is missing in it.

Author response:
5. Thanks again, we are missing the clipping operation.

Reviewer comment (Writing):
(Minor) I find it a bit confusing whether unCLIP can be used as a generative model. In the "Downstream models" section of Section 2, the paper mentions that "diffusion models that operate on CLIP embeddings, e.g., unCLIP, can also operate on ImageBind embeddings." However, in Section 6, the paper also notes that BindDiffusion, which employs the unCLIP model, struggles to generate images from the multi-modal embeddings. Some clarification on this apparent discrepancy would be appreciated.

Author response:
6. We will clarify this distinction in the **Limitations** section. To rephrase, ImageBind embeddings work on unCLIP, but seemingly perform worse. We hope to disentangle our observations on the alignment of our attack and the limitations (due to training) of the models we use.

Reviewer comment (Experiments):
The downstream task uses exactly the same embedding model as the one being attacked. Therefore, it is not surprising that they can be fooled. It would be more interesting if some unexpected findings/insights were provided.

Author response:
Ours is the first attack that targets multi-modal embeddings, as opposed to a specific task, and is completely agnostic of downstream tasks.  Success of an embedding attack cannot be evaluated without measuring its effect on tasks that use the embedding.
    
    To expand the evaluation, we have shown that our attack is effective on AudioCLIP (with a $100\\%$ success rate for $\epsilon$ larger than 4. In addition, we are conducting experiments on the transferability between embedding spaces. The experiments will be added to our **Evaluation** section.
    
    Previous attacks on multimodal learning target a single modality and don’t need to deal with modality gap. We emphasize our main observation: *adversarial* alignment can be made significantly closer than any *organic* alignment regardless of input and target modality. As a result, our attack affects **any current and future downstream task** based on the attacked embedding (or, as we show in our new experiments, similar embeddings). With an increased industry focus on large, centralized encoders, the same embedding will underpin many downstream use cases.  We show they can be attacked wholesale, without the attacker even knowing what they are.

Reviewer comment (Experiments):
As the authors acknowledged, several existing papers have studied adversarial attacks for multimodal learning (e.g., for CLIP and for multimodal large language models). More thorough comparisons with the previous works should be done.

Author response:
Thank you for providing the reference, we want to point that the attack defined in **Sec 3.1** of the paper is an attack on a *specific* downstream task (zero-shot image classification), not a **downstream-task agnostic** attack on the multi-modal embedding that affects all tasks based on that embedding (which is our contribution in this submission) including **generative** tasks that don’t assign a fixed label to the input.

We, therefore, argue that our attack is the first downstream-task agnostic attack unlike the method presented in that paper.

| Input type                 | Top-1         |          | Top-5         |          |
|----------------------------|---------------|----------|---------------|----------|
|                            | Original label| Target label | Original label| Target label |
| Original image x           | 85%           | 0%       | 99%           | 0%       |
| Adversarial illusion xa    | 77%           | 0%       | 95%           | 0%       |
| Generated(ImageBind(x))    | 42%           | 0%       | 64%           | 2%       |
| Generated(ImageBind(xa))   | 0%            | 64%      | 1%            | 92%      |

Reviewer comment (Evaluation):
How robust is the attack to adversarial defense?

Author response:
Our **Countermeasures** section has been updated to show how our attack evades the JPEG compression defense, a common method to mitigate adversarial perturbations. The JPEG-resistant attack still achieves a $75\\%$ success rate in the presence of the defense. It also evades the defense that checks consistency of embeddings of different input augmentations.