Original: U NDERSTANDING Z ERO - SHOT A DVERSARIAL R OBUSTNESS FOR L ARGE -S CALE M ODELS
Revised: U NDERSTANDING Z ERO - SHOT A DVERSARIAL R OBUSTNESS FOR L ARGE -S CALE M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Pretrained large-scale vision-language models like CLIP have exhibited strong generalization over unseen tasks.
Revised: Pretrained large-scale vision-language models like CLIP have exhibited strong generalization over unseen tasks.

Original: Yet imperceptible adversarial perturbations can significantly reduce CLIP’s performance on new tasks.
Revised: Yet imperceptible adversarial perturbations can significantly reduce CLIP’s performance on new tasks.

Original: In this work, we identify and explore the problem of adapting large-scale models for zero-shot adversarial robustness .
Revised: In this work, we identify and explore the problem of adapting large-scale models for zero-shot adversarial robustness .

Original: We first identify two key factors during model adaption—training losses and adaptation methods—that affect the model’s zero-shot adversarial robustness.
Revised: We first identify two key factors during model adaption—training losses and adaptation methods—that affect the model’s zero-shot adversarial robustness.

Original: We then propose a text-guided contrastive adversarial training loss, which aligns the text embeddings and the adversarial visual features with contrastive learning on a small set of training data.
Revised: We then propose a text-guided contrastive adversarial training loss, which aligns the text embeddings and the adversarial visual features with contrastive learning on a small set of training data.

Original: We apply this training loss to two adaption methods, model finetuning and visual prompt tuning.
Revised: We apply this training loss to two adaption methods, model finetuning and visual prompt tuning.

Original: We find that visual prompt tuning is more effective in the absence of texts, while finetuning wins in the existence of text guidance.
Revised: We find that visual prompt tuning is more effective in the absence of texts, while finetuning wins in the existence of text guidance.

Original: Overall, our approach significantly improves the zero-shot adversarial robustness over CLIP, seeing an average improvement ofpoints over ImageNet and 15 zero-shot datasets.
Revised: Overall, our approach significantly improves the zero-shot adversarial robustness over CLIP, seeing an average improvement ofpoints over ImageNet and 15 zero-shot datasets.

Original: We hope this work can shed light on understanding the zero-shot adversarial robustness of large-scale models
Revised: We hope this work can shed light on understanding the zero-shot adversarial robustness of large-scale models

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Large-scale models trained on vision and language data have emerged as a powerful paradigm for tackling many recognition problems in computer vision (Jia et al., 2021;
Revised: Large-scale models trained on vision and language data have emerged as a powerful paradigm for tackling many recognition problems in computer vision (Jia et al., 2021;

Original: Radford et al., 2021), graphics (Ramesh et al., 2022) and robotics (Ahn et al., 2022).
Revised: Radford et al., 2021), graphics (Ramesh et al., 2022) and robotics (Ahn et al., 2022).

Original: In addition to state-of-the-art performance on multiple benchmarks, one of their key advantages is zero-shot generalization, where the models use just a single textual description to recognize new visual categories with high accuracy.
Revised: In addition to state-of-the-art performance on multiple benchmarks, one of their key advantages is zero-shot generalization, where the models use just a single textual description to recognize new visual categories with high accuracy.

Original: However, robustness under adversarial examples remains a challenge, where an imperceptible pattern can be combined with the image to cause recognition failures (Croce & Hein, 2020; Carlini & Wagner, 2017; Dong et al., 2018; Szegedy et al., 2013; Moosavi-Dezfooli et al., 2016).
Revised: However, robustness under adversarial examples remains a challenge, where an imperceptible pattern can be combined with the image to cause recognition failures (Croce & Hein, 2020; Carlini & Wagner, 2017; Dong et al., 2018; Szegedy et al., 2013; Moosavi-Dezfooli et al., 2016).

Original: Due to the importance of this problem, there is a large literature that investigates adversarial robustness for neural networks.
Revised: Due to the importance of this problem, there is a large literature that investigates adversarial robustness for neural networks.

Original: The most common approach for adversarial defense is to learn the model through adversarial training (Madry et al., 2018; Mao et al., 2019; Szegedy et al., 2013; Pang et al., 2020;
Revised: The most common approach for adversarial defense is to learn the model through adversarial training (Madry et al., 2018; Mao et al., 2019; Szegedy et al., 2013; Pang et al., 2020;

Original: Rice et al., 2020; Uesato et al., 2019), which involves augmenting the training set with mined adversarial examples that fool the image classifier.
Revised: Rice et al., 2020; Uesato et al., 2019), which involves augmenting the training set with mined adversarial examples that fool the image classifier.

Original: Adversarial training has been validated to improve robustness on the task that the mined examples come from, but it often comes at a cost of generalization (Stutz et al., 2019;
Revised: Adversarial training has been validated to improve robustness on the task that the mined examples come from, but it often comes at a cost of generalization (Stutz et al., 2019;

Original: Su et al., 2018; Pedraza et al., 2021).
Revised: Su et al., 2018; Pedraza et al., 2021).

Original: However, our world is vast and naturally open, and only evaluating adversarial robustness on the learned tasks is limited.
Revised: However, our world is vast and naturally open, and only evaluating adversarial robustness on the learned tasks is limited.

Original: Can we achieve zero-shot transferability for adversarial robustness, even if the model has never been trained on the unknown tasks?
Revised: Can we achieve zero-shot transferability for adversarial robustness, even if the model has never been trained on the unknown tasks?

Original: In this paper, we study this important yet under-explored problem, zero-shot adversarial robustness of large-scale vision-language models.
Revised: In this paper, we study this important yet under-explored problem, zero-shot adversarial robustness of large-scale vision-language models.

Original: We start our investigation with the state-of-the-art CLIP model (Radford et al., 2021), which has been shown to be effective in zero-shot recognition tasks.
Revised: We start our investigation with the state-of-the-art CLIP model (Radford et al., 2021), which has been shown to be effective in zero-shot recognition tasks.

Original: We find that simply adding an imperceptible vector to the image ( ≤ 1 / 255 ) can subvert CLIP’s prediction (see Figure 1a).
Revised: We find that simply adding an imperceptible vector to the image ( ≤ 1 / 255 ) can subvert CLIP’s prediction (see Figure 1a).

Original: If we follow the standard adversarial training defense paradigm (Madry et al., 2018; Rice et al., 2020) to finetune CLIP on the ImageNet (Deng et al., 2009b) training set, we observe that the adapted CLIP has improved adversarial robustness on the ImageNet validation set, but comes at the cost of significantly reduced accuracy on unseen datasets and classes (Fig-
Revised: If we follow the standard adversarial training defense paradigm (Madry et al., 2018; Rice et al., 2020) to finetune CLIP on the ImageNet (Deng et al., 2009b) training set, we observe that the adapted CLIP has improved adversarial robustness on the ImageNet validation set, but comes at the cost of significantly reduced accuracy on unseen datasets and classes (Fig-

Original: [Figure]
Revised: [Figure]

Original: ure 1b).
Revised: ure 1b).

Original: Standard adversarial training backfires on CLIP as it fails to retain the model’s zero-shot generalization ability.
Revised: Standard adversarial training backfires on CLIP as it fails to retain the model’s zero-shot generalization ability.

Original: Adaptation methods and training objectives are the two major factors for adapting a large-scale model.
Revised: Adaptation methods and training objectives are the two major factors for adapting a large-scale model.

Original: First, besides finetuning the whole model, we seek an alternative adaptation method—visual prompt tuning—which adapts the inputs instead of the parameters of the model.
Revised: First, besides finetuning the whole model, we seek an alternative adaptation method—visual prompt tuning—which adapts the inputs instead of the parameters of the model.

Original: Visual prompt tuning (VPT) is an emerging light-weight adaptation method (Bar et al., 2022;
Revised: Visual prompt tuning (VPT) is an emerging light-weight adaptation method (Bar et al., 2022;

Original: Bahng et al., 2022) that learns a visual prompt which is added to the input image, where we use visual prompt to instruct the model to be robust against adversaries.
Revised: Bahng et al., 2022) that learns a visual prompt which is added to the input image, where we use visual prompt to instruct the model to be robust against adversaries.

Original: Second, we find that the standard adversarial training objective ignores the visual-language alignment in CLIP’s pretrained representation space, causing the model to lose zero-shot capability.
Revised: Second, we find that the standard adversarial training objective ignores the visual-language alignment in CLIP’s pretrained representation space, causing the model to lose zero-shot capability.

Original: We then propose a text-guided contrastive adversarial training (TeCoA) loss, dubbed as Tekoa (tee · kow), which maximizes the similarity of the adversarial visual features and the correct text embeddings with contrastive learning.
Revised: We then propose a text-guided contrastive adversarial training (TeCoA) loss, dubbed as Tekoa (tee · kow), which maximizes the similarity of the adversarial visual features and the correct text embeddings with contrastive learning.

Original: Since the adapted visual features continue to align well with the text features, the model adapted with TeCoA can maximally retain the original zero-shot generalization of CLIP while enjoying improved adversarial robustness.
Revised: Since the adapted visual features continue to align well with the text features, the model adapted with TeCoA can maximally retain the original zero-shot generalization of CLIP while enjoying improved adversarial robustness.

Original: We conduct an extensive evaluation on 15 zero-shot image datasets, offering a holistic study of the zero-shot adversarial robustness problem.
Revised: We conduct an extensive evaluation on 15 zero-shot image datasets, offering a holistic study of the zero-shot adversarial robustness problem.

Original: We find that the lightweight VPT is noticeably more effective than model finetuning when textual information is unavailable.
Revised: We find that the lightweight VPT is noticeably more effective than model finetuning when textual information is unavailable.

Original: When texts are used during adaptation, both VPT and finetuning using our TeCoA loss have drastically improved zero-shot adversarial robustness compared to baselines.
Revised: When texts are used during adaptation, both VPT and finetuning using our TeCoA loss have drastically improved zero-shot adversarial robustness compared to baselines.

Original: Finetuning has higher gains than VPT as more parameters are tuned.
Revised: Finetuning has higher gains than VPT as more parameters are tuned.

Original: Our best performing model with the TeCoA loss can improve adversarial robustness over CLIP by an average of 31% across the datasets.
Revised: Our best performing model with the TeCoA loss can improve adversarial robustness over CLIP by an average of 31% across the datasets.

Original: The extensive ablation studies shed light on the design choices of various adaptation methods, training losses, and whether to use texts for adaptation.
Revised: The extensive ablation studies shed light on the design choices of various adaptation methods, training losses, and whether to use texts for adaptation.

Original: We study an important yet under-explored problem, zero-shot adversarial robustness of large-scale models, providing a holistic analysis.
Revised: We study an important yet under-explored problem, zero-shot adversarial robustness of large-scale models, providing a holistic analysis.

Original: This is especially important given that large-scale vision models are emerging as infrastructure and are deploying to critical applications.
Revised: This is especially important given that large-scale vision models are emerging as infrastructure and are deploying to critical applications.

Original: Extensive experiments show the effectiveness of our approach.
Revised: Extensive experiments show the effectiveness of our approach.

Original: The key lesson we learned from our experiment is that adversarial training with language information is the key to achieving zero-shot adversarial robustness.
Revised: The key lesson we learned from our experiment is that adversarial training with language information is the key to achieving zero-shot adversarial robustness.

Original: While finetuning the whole model achieves the best robustness, visual prompt tuning achieves better robustness when updating a small number of parameters.
Revised: While finetuning the whole model achieves the best robustness, visual prompt tuning achieves better robustness when updating a small number of parameters.

Original: Our method also works on unlabeled images, allowing for better robustness with a large amount of unlabeled data.
Revised: Our method also works on unlabeled images, allowing for better robustness with a large amount of unlabeled data.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Zero-Shot Generalization aims to classify novel classes and tasks that are unseen during training (Palatucci et al., 2009; Lampert et al., 2009; Radford et al., 2021).
Revised: Zero-Shot Generalization aims to classify novel classes and tasks that are unseen during training (Palatucci et al., 2009; Lampert et al., 2009; Radford et al., 2021).

Original: Existing zero-shot methods often project visual features into semantic feature space (Frome et al., 2013; Akata et al., 2015;
Revised: Existing zero-shot methods often project visual features into semantic feature space (Frome et al., 2013; Akata et al., 2015;

Original: Romera-Paredes & Torr, 2015; Xie et al., 2019; Yu et al., 2018; Liu et al., 2019), or use generative methods to generate fake visual features of unseen classes from their semantic descriptions to train classifiers (Xian et al., 2018;
Revised: Romera-Paredes & Torr, 2015; Xie et al., 2019; Yu et al., 2018; Liu et al., 2019), or use generative methods to generate fake visual features of unseen classes from their semantic descriptions to train classifiers (Xian et al., 2018;

Original: Ni et al., 2019; Huang et al., 2019; Schonfeld et al., 2019; Verma et al., 2019; Liu et al., 2020).
Revised: Ni et al., 2019; Huang et al., 2019; Schonfeld et al., 2019; Verma et al., 2019; Liu et al., 2020).

Original: Recently, large-scale pretrained vision-language models (Radford et al., 2021;
Revised: Recently, large-scale pretrained vision-language models (Radford et al., 2021;

Original: Jia et al., 2021) have shown outstanding zero-shot generalization ability on unseen tasks via text prompt engineering.
Revised: Jia et al., 2021) have shown outstanding zero-shot generalization ability on unseen tasks via text prompt engineering.

Original: Their adversarial robustness and its transferability, however, has not been studied in the zero-shot setting.
Revised: Their adversarial robustness and its transferability, however, has not been studied in the zero-shot setting.

Original: Adversarial Robustness.
Revised: Adversarial Robustness.

Original: Adversarial attacks for image recognition find an additive vector on the input to maximize the cross-entropy loss, which is calculated from the model prediction and the ground truth one-hot label (Szegedy et al., 2013; Athalye et al., 2018; Carlini & Wagner, 2017; Kurakin et al., 2017; Papernot et al., 2015; Moosavi-Dezfooli et al., 2016).
Revised: Adversarial attacks for image recognition find an additive vector on the input to maximize the cross-entropy loss, which is calculated from the model prediction and the ground truth one-hot label (Szegedy et al., 2013; Athalye et al., 2018; Carlini & Wagner, 2017; Kurakin et al., 2017; Papernot et al., 2015; Moosavi-Dezfooli et al., 2016).

Original: Adversarial training (Madry et al., 2018) and its variants (Zhang et al., 2019; Rice et al., 2020), which train the model on generated adversarial examples, are effective in improving adversarial robustness on the task that they have been trained on.
Revised: Adversarial training (Madry et al., 2018) and its variants (Zhang et al., 2019; Rice et al., 2020), which train the model on generated adversarial examples, are effective in improving adversarial robustness on the task that they have been trained on.

Original: However, it is unclear whether this approach can improve robustness in the zero-shot scenario.
Revised: However, it is unclear whether this approach can improve robustness in the zero-shot scenario.

Original: To the best of our knowledge, Yucel et al.
Revised: To the best of our knowledge, Yucel et al.

Original: (2020) is so far the only work that studies the adversarial robustness of zero-shot learning models.
Revised: (2020) is so far the only work that studies the adversarial robustness of zero-shot learning models.

Original: Their setting is limited because it relies on predicting robust attributes, which may not be easily available for many tasks.
Revised: Their setting is limited because it relies on predicting robust attributes, which may not be easily available for many tasks.

Original: Transferability of Robustness.
Revised: Transferability of Robustness.

Original: Mao et al.
Revised: Mao et al.

Original: (2020) shows that adversarial robustness is transferable across tasks when multiple tasks are trained together.
Revised: (2020) shows that adversarial robustness is transferable across tasks when multiple tasks are trained together.

Original: Salman et al.
Revised: Salman et al.

Original: (2020) shows that adversarially robust models transfer better than their standard-trained counterparts when they are finetuned on other tasks.
Revised: (2020) shows that adversarially robust models transfer better than their standard-trained counterparts when they are finetuned on other tasks.

Original: Chan et al.
Revised: Chan et al.

Original: (2020) finds that matching a robust model’s gradient can transfer robustness to a new model.
Revised: (2020) finds that matching a robust model’s gradient can transfer robustness to a new model.

Original: Vaishnavi et al.
Revised: Vaishnavi et al.

Original: (2022) proposes a low-cost method to transfer robustness to a new model on the same task with different architecture.
Revised: (2022) proposes a low-cost method to transfer robustness to a new model on the same task with different architecture.

Original: However, the transferability of adversarial robustness to zero-shot tasks has not been investigated.
Revised: However, the transferability of adversarial robustness to zero-shot tasks has not been investigated.

Original: Contrastive Learning (Oord et al., 2018) has been used to train large-scale image-language models (Jia et al., 2021;
Revised: Contrastive Learning (Oord et al., 2018) has been used to train large-scale image-language models (Jia et al., 2021;

Original: Radford et al., 2021).
Revised: Radford et al., 2021).

Original: Kim et al.
Revised: Kim et al.

Original: ; Jiang et al.
Revised: ; Jiang et al.

Original: (2020) propose instancewise adversarial perturbation and use a contrastive loss to align the features of clean examples and generated adversarial examples.
Revised: (2020) propose instancewise adversarial perturbation and use a contrastive loss to align the features of clean examples and generated adversarial examples.

Original: Our method is the first one to introduce a cross-modal image-text contrastive loss in adversarial contrastive learning.
Revised: Our method is the first one to introduce a cross-modal image-text contrastive loss in adversarial contrastive learning.

Original: Adapting Pretrained Models.
Revised: Adapting Pretrained Models.

Original: Linear probing and finetuning are the two major ways to adapt deep pretrained models.
Revised: Linear probing and finetuning are the two major ways to adapt deep pretrained models.

Original: Recently, a more lightweight adaptation method, prompt tuning, has been proposed (Zhou et al., 2022c;b;a).
Revised: Recently, a more lightweight adaptation method, prompt tuning, has been proposed (Zhou et al., 2022c;b;a).

Original: Shu et al.
Revised: Shu et al.

Original: (2022) shows that test-time optimization for text prompting helps generalization.
Revised: (2022) shows that test-time optimization for text prompting helps generalization.

Original: Bar et al.
Revised: Bar et al.

Original: (2022) combines target task and image inpainting to achieve zeroshot task inference.
Revised: (2022) combines target task and image inpainting to achieve zeroshot task inference.

Original: Jia et al.
Revised: Jia et al.

Original: ; Sandler et al.
Revised: ; Sandler et al.

Original: (2022) used visual prompting to replace the finetuning procedure for large-scale models.
Revised: (2022) used visual prompting to replace the finetuning procedure for large-scale models.

Original: Bahng et al.
Revised: Bahng et al.

Original: (2022) optimizes a visual prompt to increase the performance on the same task that the model finetunes the visual prompt on.
Revised: (2022) optimizes a visual prompt to increase the performance on the same task that the model finetunes the visual prompt on.

Original: Prompt tuning for continuous learning has also been proposed (Conder et al., 2022).
Revised: Prompt tuning for continuous learning has also been proposed (Conder et al., 2022).

Original: However, adapting large-scale models for transferable zero-shot robustness, using methods such as finetuning or visual prompting, has not been investigated.
Revised: However, adapting large-scale models for transferable zero-shot robustness, using methods such as finetuning or visual prompting, has not been investigated.

Original:  M ODEL A DAPTATION FOR Z ERO -S HOT A DVERSARIAL R OBUSTNESS
Revised:  M ODEL A DAPTATION FOR Z ERO -S HOT A DVERSARIAL R OBUSTNESS

Original: We first give background in Section 3.1 on adversarial attacks, adversarial training, and the problem setup.
Revised: We first give background in Section 3.1 on adversarial attacks, adversarial training, and the problem setup.

Original: In Section 3.2, we discuss adaptation methods for adapting large-scale models for zero-shot adversarial robustness.
Revised: In Section 3.2, we discuss adaptation methods for adapting large-scale models for zero-shot adversarial robustness.

Original: Finally, Section 3.3, we discuss the effect of different training losses to motivate and then introduce our proposed text-guided contrastive adversarial training (TeCoA) loss.
Revised: Finally, Section 3.3, we discuss the effect of different training losses to motivate and then introduce our proposed text-guided contrastive adversarial training (TeCoA) loss.

Original: 3. B ACKGROUND AND P ROBLEM S ETUP
Revised: 3. B ACKGROUND AND P ROBLEM S ETUP

Original: Let F θ ( · ) be a deep model for image recognition parameterized by θ .
Revised: Let F θ ( · ) be a deep model for image recognition parameterized by θ .

Original: Given an input image x , the model produces a representation ˆ y = F θ ( x ) .
Revised: Given an input image x , the model produces a representation ˆ y = F θ ( x ) .

Original: For image classification, a standard model learns to minimize the cross-entropy loss, L ( x , y )
Revised: For image classification, a standard model learns to minimize the cross-entropy loss, L ( x , y )

Original: =
Revised: =

Original: H ( F θ ( x ) , y ) , where the label y is often represented by a one-hot vector.
Revised: H ( F θ ( x ) , y ) , where the label y is often represented by a one-hot vector.

Original: Adversarial Attacks.
Revised: Adversarial Attacks.

Original: An adversary ( i.e., attacker) typically optimizes for an additive transformation δ to the image, x a = x + δ , which can fool the model F θ to make incorrect predictions:
Revised: An adversary ( i.e., attacker) typically optimizes for an additive transformation δ to the image, x a = x + δ , which can fool the model F θ to make incorrect predictions:

Original: [Equation]
Revised: [Equation]

Original: Here, the magnitude of the added pattern is bounded by a q -norm ball of radius ϵ , making the attack perceptually invisible.
Revised: Here, the magnitude of the added pattern is bounded by a q -norm ball of radius ϵ , making the attack perceptually invisible.

Original: The role of a defender is to find ways to correct the influence of the attacks and ensure that the model is robust to the adversarial perturbations.
Revised: The role of a defender is to find ways to correct the influence of the attacks and ensure that the model is robust to the adversarial perturbations.

Original: [Figure]
Revised: [Figure]

Original: 𝐹(𝑥) 𝐹
Revised: 𝐹(𝑥) 𝐹

Original: [Figure]
Revised: [Figure]

Original: 𝑥
Revised: 𝑥

Original: [Figure]
Revised: [Figure]

Original: Zero-Shot Adversarial Robustness.
Revised: Zero-Shot Adversarial Robustness.

Original: Large-scale vision-language models generalize well to new tasks and datasets at test time.
Revised: Large-scale vision-language models generalize well to new tasks and datasets at test time.

Original: However, the zero-shot transferability of adversarial robustness of these models is less explored.
Revised: However, the zero-shot transferability of adversarial robustness of these models is less explored.

Original: In our zero-shot adversarial robustness setup, we study the worst case: we assume that the attacker has the significant advantage of unrestricted access to the ground truth of new tasks at test time, while the defender has no access.
Revised: In our zero-shot adversarial robustness setup, we study the worst case: we assume that the attacker has the significant advantage of unrestricted access to the ground truth of new tasks at test time, while the defender has no access.

Original: While the attacker can directly optimize for an attacked image that fools the model, the defender needs to be robust on all kinds of unseen data and tasks.
Revised: While the attacker can directly optimize for an attacked image that fools the model, the defender needs to be robust on all kinds of unseen data and tasks.

Original: Compared to the commonly-used robustness setting, which only evaluates robustness on the trained task, our setup is more challenging.
Revised: Compared to the commonly-used robustness setting, which only evaluates robustness on the trained task, our setup is more challenging.

Original: Adversarial Training is the common strategy to improve a model’s adversarial robustness.
Revised: Adversarial Training is the common strategy to improve a model’s adversarial robustness.

Original: By retraining the model on mined adversarial examples, the model is incentivized to learn features invariant under adversarial transformations.
Revised: By retraining the model on mined adversarial examples, the model is incentivized to learn features invariant under adversarial transformations.

Original: The defender often optimizes for the objective
Revised: The defender often optimizes for the objective

Original: [Equation]
Revised: [Equation]

Original: so that the model F θ still makes correct predictions on the generated adversarial examples.
Revised: so that the model F θ still makes correct predictions on the generated adversarial examples.

Original: As shown in Figure 1, the vanilla adversarial training approach is effective on seen tasks but fails when the model is evaluated on attacked data from unseen tasks.
Revised: As shown in Figure 1, the vanilla adversarial training approach is effective on seen tasks but fails when the model is evaluated on attacked data from unseen tasks.

Original: 3. A DAPTING THE L ARGE -S CALE M ODELS
Revised: 3. A DAPTING THE L ARGE -S CALE M ODELS

Original: One of the key advantages of large-scale pretrained models is that the features of these models are generalizable to various tasks without full retraining.
Revised: One of the key advantages of large-scale pretrained models is that the features of these models are generalizable to various tasks without full retraining.

Original: Thus, we would like to make a lightweight adjustment to the models with a small set of training data of attacked images on known tasks, and maximally retain the zero-shot generalization ability of these large models while improving their adversarial robustness on new tasks.
Revised: Thus, we would like to make a lightweight adjustment to the models with a small set of training data of attacked images on known tasks, and maximally retain the zero-shot generalization ability of these large models while improving their adversarial robustness on new tasks.

Original: We adopt CLIP, one of the best performing vision-language models for zero-shot recognition, as our base model, and investigate a few adaptation strategies as shown in Figure 2 to improve the zero-shot adversarial robustness of CLIP.
Revised: We adopt CLIP, one of the best performing vision-language models for zero-shot recognition, as our base model, and investigate a few adaptation strategies as shown in Figure 2 to improve the zero-shot adversarial robustness of CLIP.

Original: Finetuning (FT).
Revised: Finetuning (FT).

Original: A typical way to adapt a pretrained model, finetuning, is to update the parameters of the model either entirely or partially.
Revised: A typical way to adapt a pretrained model, finetuning, is to update the parameters of the model either entirely or partially.

Original: While this approach improves model robustness on the target distribution, Radford et al.
Revised: While this approach improves model robustness on the target distribution, Radford et al.

Original: (2021) and Pham et al.
Revised: (2021) and Pham et al.

Original: (2021) show that this improvement often comes at a cost of generalization; directly modifying model parameters may lead to a higher tendency towards overfitting.
Revised: (2021) show that this improvement often comes at a cost of generalization; directly modifying model parameters may lead to a higher tendency towards overfitting.

Original: Visual Prompting (VP).
Revised: Visual Prompting (VP).

Original: Recently, visual prompt tuning has emerged as a lightweight approach for adapting pretrained large-scale models.
Revised: Recently, visual prompt tuning has emerged as a lightweight approach for adapting pretrained large-scale models.

Original: Originating from the natural language processing community, the key idea of prompt tuning is to identify prompts to decorate the inputs that effectively query the pretrained models.
Revised: Originating from the natural language processing community, the key idea of prompt tuning is to identify prompts to decorate the inputs that effectively query the pretrained models.

Original: The computer vision community recently adopted this idea and is starting to explore approaches that can modify input images in pixel space to better adapt large-scale vision
Revised: The computer vision community recently adopted this idea and is starting to explore approaches that can modify input images in pixel space to better adapt large-scale vision

Original: [Figure]
Revised: [Figure]

Original: models for downstream tasks.
Revised: models for downstream tasks.

Original: For transformer-based models, visual prompting methods either learn a token that is appended to the input token sequences (Bar et al., 2022) or learn a direct modification to the input images (Bahng et al., 2022) for adaptation, as shown by (d,e) in Figure 2.
Revised: For transformer-based models, visual prompting methods either learn a token that is appended to the input token sequences (Bar et al., 2022) or learn a direct modification to the input images (Bahng et al., 2022) for adaptation, as shown by (d,e) in Figure 2.

Original: In our study, we conduct an extensive analysis over both adaptation methods to better understand adapting large-scale models for zero-shot adversarial robustness.
Revised: In our study, we conduct an extensive analysis over both adaptation methods to better understand adapting large-scale models for zero-shot adversarial robustness.

Original: 3. T EXT -G UIDED A DVERSARIAL C ONTRASTIVE A DVERSARIAL T RAINING
Revised: 3. T EXT -G UIDED A DVERSARIAL C ONTRASTIVE A DVERSARIAL T RAINING

Original: Through our initial experiments, we conjecture that the training objective might play a key role in improving the zero-shot adversarial robustness of the models.
Revised: Through our initial experiments, we conjecture that the training objective might play a key role in improving the zero-shot adversarial robustness of the models.

Original: Radford et al.
Revised: Radford et al.

Original: (2021) indicates that the zero-shot generalization ability of large-scale vision-language models may come from their language supervision.
Revised: (2021) indicates that the zero-shot generalization ability of large-scale vision-language models may come from their language supervision.

Original: For example, CLIP learns a joint visual-and-text feature space, which helps zero-shot generalization at test time.
Revised: For example, CLIP learns a joint visual-and-text feature space, which helps zero-shot generalization at test time.

Original: If we simply finetune the visual encoder with one-hot labels, it may break the joint feature space and harm this zero-shot generalization ability.
Revised: If we simply finetune the visual encoder with one-hot labels, it may break the joint feature space and harm this zero-shot generalization ability.

Original: These observations motivate us to consider using the text information when generating the adversarial examples and also in the training objective during model adaptation.
Revised: These observations motivate us to consider using the text information when generating the adversarial examples and also in the training objective during model adaptation.

Original: We now introduce the Text-guided Contrastive Adversarial (TeCoA) training loss, to effectively incorporate text information.
Revised: We now introduce the Text-guided Contrastive Adversarial (TeCoA) training loss, to effectively incorporate text information.

Original: In contrast to prior contrastive learning which does image-to-image contrastive learning (Jiang et al., 2020; Kim et al., 2020), we consider a cross-modal text-to-image contrastive learning paradigm.
Revised: In contrast to prior contrastive learning which does image-to-image contrastive learning (Jiang et al., 2020; Kim et al., 2020), we consider a cross-modal text-to-image contrastive learning paradigm.

Original: We first generate a set of adversarial examples conditioned on the text inputs which are targeted to fool the model about the correspondence between image features and text embeddings.
Revised: We first generate a set of adversarial examples conditioned on the text inputs which are targeted to fool the model about the correspondence between image features and text embeddings.

Original: TeCoA then tries to minimize the feature distance between the attacked image and the correct corresponding text inputs contrastively (Figure 3).
Revised: TeCoA then tries to minimize the feature distance between the attacked image and the correct corresponding text inputs contrastively (Figure 3).

Original: We provide additional discussion of this text-image contrastive objective in Appendix A.5.
Revised: We provide additional discussion of this text-image contrastive objective in Appendix A.5.

Original: Concretely, given a set of image and text pairs { ( x i , t i ) } , we use the pretrained CLIP model to encode each pair with an image encoder F θ and a text encoder T .
Revised: Concretely, given a set of image and text pairs { ( x i , t i ) } , we use the pretrained CLIP model to encode each pair with an image encoder F θ and a text encoder T .

Original: We then have the following image-to-text contrastive loss function,
Revised: We then have the following image-to-text contrastive loss function,

Original: [Equation]
Revised: [Equation]

Original: where the z ( I ) i =
Revised: where the z ( I ) i =

Original: F θ ( x i ) are the features of the input image, and z ( T ) i
Revised: F θ ( x i ) are the features of the input image, and z ( T ) i

Original: = T ( t i ) are the features from the input text.
Revised: = T ( t i ) are the features from the input text.

Original: We use y ij to indicate which image-text pairs are positive and which are negative; this indicator satisfies y ij = 1 if and only if the examples i = j , and 0 otherwise.
Revised: We use y ij to indicate which image-text pairs are positive and which are negative; this indicator satisfies y ij = 1 if and only if the examples i = j , and 0 otherwise.

Original: τ is a scalar hyper-parameter, and cos denotes the cosine similarity function.
Revised: τ is a scalar hyper-parameter, and cos denotes the cosine similarity function.

Original: Constructing Adversarial Examples for Image-Text Pair.
Revised: Constructing Adversarial Examples for Image-Text Pair.

Original: Instead of maximizing the standard cross-entropy loss, we maximize the image-text contrastive loss given a batch of natural images x and text t :
Revised: Instead of maximizing the standard cross-entropy loss, we maximize the image-text contrastive loss given a batch of natural images x and text t :

Original: [Equation]
Revised: [Equation]

Original: Here, for image x i in the batch, the associated text t i can be its natural label text or constructed via the standard prompts for the zero-shot tasks ( e.g., “a photo of a [LABEL]”).
Revised: Here, for image x i in the batch, the associated text t i can be its natural label text or constructed via the standard prompts for the zero-shot tasks ( e.g., “a photo of a [LABEL]”).

Original: The indicator y ij =
Revised: The indicator y ij =

Original: when image x i has category j as its ground-truth class, and 0 otherwise.
Revised: when image x i has category j as its ground-truth class, and 0 otherwise.

Original: In practice, we find this objective is effective at generating adversarial examples for zero-shot image recognition tasks.
Revised: In practice, we find this objective is effective at generating adversarial examples for zero-shot image recognition tasks.

Original: Text-Guided Contrastive Adversarial Training.
Revised: Text-Guided Contrastive Adversarial Training.

Original: Once we have the adversarial examples, we optimize the parameters θ of the vision encoder F θ to minimize the aforementioned objective (Equa- tion 8) on the generated adversarial examples,
Revised: Once we have the adversarial examples, we optimize the parameters θ of the vision encoder F θ to minimize the aforementioned objective (Equa- tion 8) on the generated adversarial examples,

Original: [Equation]
Revised: [Equation]

Original: Our algorithm iteratively alternates between generating adversarial examples and updating the model via Equation 5.
Revised: Our algorithm iteratively alternates between generating adversarial examples and updating the model via Equation 5.

Original: Since TeCoA uses additional information from the text embeddings to correct the visual features corrupted by the adversarial attacks, it helps the model to retain zero-shot transferability regarding adversarial robustness.
Revised: Since TeCoA uses additional information from the text embeddings to correct the visual features corrupted by the adversarial attacks, it helps the model to retain zero-shot transferability regarding adversarial robustness.

Original: Contrastive Adversarial Training without Text.
Revised: Contrastive Adversarial Training without Text.

Original: To validate whether the gain of zero-shot robustness comes from language supervision or is due to the formulation of the contrastive loss itself, we consider two variants of contrastive losses which do not utilize languages in our experiments.
Revised: To validate whether the gain of zero-shot robustness comes from language supervision or is due to the formulation of the contrastive loss itself, we consider two variants of contrastive losses which do not utilize languages in our experiments.

Original: One is a contrastive adversarial training loss with one-hot labels ( CoAdv. ), where the label information is encoded as a one-hot vector and is used to contrast with the image features.
Revised: One is a contrastive adversarial training loss with one-hot labels ( CoAdv. ), where the label information is encoded as a one-hot vector and is used to contrast with the image features.

Original: Another is based on Jiang et al.
Revised: Another is based on Jiang et al.

Original: (2020), where the model is finetuned on adversarial examples to fool the image-toimage contrastive loss, denoted as ImgCoAdv.
Revised: (2020), where the model is finetuned on adversarial examples to fool the image-toimage contrastive loss, denoted as ImgCoAdv.

Original: More details are presented in Section 4.
Revised: More details are presented in Section 4.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We start by describing our experiment setups and present the experimental results of 16 datasets in Section 4.1.
Revised: We start by describing our experiment setups and present the experimental results of 16 datasets in Section 4.1.

Original: We observe that CLIP adapted with TeCoA achieves an average of 31 points improvement across the datasets over the original CLIP model.
Revised: We observe that CLIP adapted with TeCoA achieves an average of 31 points improvement across the datasets over the original CLIP model.

Original: In Section 4.2, we provide extensive analysis over the design choices involved in our approach and identify that the use of language in
Revised: In Section 4.2, we provide extensive analysis over the design choices involved in our approach and identify that the use of language in

Original: TeCoA largely improves the model’s zero-shot adversarial robustness.
Revised: TeCoA largely improves the model’s zero-shot adversarial robustness.

Original: Datasets.
Revised: Datasets.

Original: We evaluate the zero-shot adversarial robustness conferred by TeCoA trained on ImageNet (Deng et al., 2009a) and report the performance of the models on the ImageNet test set as well as 15 zero-shot test datasets, covering a diverse range of recognition tasks.
Revised: We evaluate the zero-shot adversarial robustness conferred by TeCoA trained on ImageNet (Deng et al., 2009a) and report the performance of the models on the ImageNet test set as well as 15 zero-shot test datasets, covering a diverse range of recognition tasks.

Original: Specifically, we include
Revised: Specifically, we include

Original: CIFAR10, CIFAR100 (Krizhevsky et al., 2009), STL10 (Coates et al., 2011), Caltech101 (Fei-Fei et al., 2004), and Caltech256 (Griffin et al., 2007) for generic classification; OxfordPets
Revised: CIFAR10, CIFAR100 (Krizhevsky et al., 2009), STL10 (Coates et al., 2011), Caltech101 (Fei-Fei et al., 2004), and Caltech256 (Griffin et al., 2007) for generic classification; OxfordPets

Original: (Parkhi et al., 2012), StanfordCars (Krause
Revised: (Parkhi et al., 2012), StanfordCars (Krause

Original: et al., 2013), Food101 (Bossard et al., 2014), Flowers102 (Nilsback & Zisserman, 2008), and FGVCAircraft (Maji et al., 2013) for fine-grained classification;
Revised: et al., 2013), Food101 (Bossard et al., 2014), Flowers102 (Nilsback & Zisserman, 2008), and FGVCAircraft (Maji et al., 2013) for fine-grained classification;

Original: SUN397 (Xiao et al., 2010) for scene recognition; and DTD (Cimpoi et al., 2014) for texture recognition.
Revised: SUN397 (Xiao et al., 2010) for scene recognition; and DTD (Cimpoi et al., 2014) for texture recognition.

Original: Finally, we include three datasets with domain-specialized tasks, PatchCamelyon (PCAM, lymph node tumor detection)
Revised: Finally, we include three datasets with domain-specialized tasks, PatchCamelyon (PCAM, lymph node tumor detection)

Original: (Veeling et al., 2018), HatefulMemes (hatespeech detection)
Revised: (Veeling et al., 2018), HatefulMemes (hatespeech detection)

Original: (Kiela et al., 2020), and EuroSAT (satellite image classification) (Helber et al., 2017).
Revised: (Kiela et al., 2020), and EuroSAT (satellite image classification) (Helber et al., 2017).

Original: Baselines.
Revised: Baselines.

Original: We consider multiple variants to adapt CLIP (Radford et al., 2021).
Revised: We consider multiple variants to adapt CLIP (Radford et al., 2021).

Original: For adaptation methods, we consider visual prompting (VP)
Revised: For adaptation methods, we consider visual prompting (VP)

Original: (Bahng et al., 2022), linear probing (LP) and finetuning (FT) .
Revised: (Bahng et al., 2022), linear probing (LP) and finetuning (FT) .

Original: Since LP involves training a linear readout layer on the target task, it is not zero-shot, but we still evaluate it for reference.
Revised: Since LP involves training a linear readout layer on the target task, it is not zero-shot, but we still evaluate it for reference.

Original: For training losses, we consider
Revised: For training losses, we consider

Original: (1) vanilla cross-entropy loss (CE) ; (2) standard adversarial training loss (Adv.) with the cross-entropy loss; (3) contrastive adversarial training loss (CoAdv.) ; (4) contrastive adversarial training over images (ImgCoAdv.) ; (5) our text-guided contrastive adversarial training (TeCoA) .
Revised: (1) vanilla cross-entropy loss (CE) ; (2) standard adversarial training loss (Adv.) with the cross-entropy loss; (3) contrastive adversarial training loss (CoAdv.) ; (4) contrastive adversarial training over images (ImgCoAdv.) ; (5) our text-guided contrastive adversarial training (TeCoA) .

Original: In our experiments, we name these variants by adaptation(loss) .
Revised: In our experiments, we name these variants by adaptation(loss) .

Original: Detailed formulations and associated training algorithms for each loss can be found in Section A.3.
Revised: Detailed formulations and associated training algorithms for each loss can be found in Section A.3.

Original: Implementation Details.
Revised: Implementation Details.

Original: We use CLIP-B/32 architecture.
Revised: We use CLIP-B/32 architecture.

Original: We optimize the model using an SGD optimizer with momentum 0.9.
Revised: We optimize the model using an SGD optimizer with momentum 0.9.

Original: We train for 10 epochs.
Revised: We train for 10 epochs.

Original: For finetuning, we use a learning rate of 1 e − 5 .
Revised: For finetuning, we use a learning rate of 1 e − 5 .

Original: For visual prompt tuning, we use a learning rate of 40 and a batch size of 256.
Revised: For visual prompt tuning, we use a learning rate of 40 and a batch size of 256.

Original: For prompt tuning on the entire ImageNet dataset, we use token-level prompt with size 200, while for subsets of ImageNet (1K, 5K, and 50K images), we use token-level prompt with size 5 and a smaller batch size of 64.
Revised: For prompt tuning on the entire ImageNet dataset, we use token-level prompt with size 200, while for subsets of ImageNet (1K, 5K, and 50K images), we use token-level prompt with size 5 and a smaller batch size of 64.

Original: Unless specified, during adversarial training, we generate L inf = 1 / 255 bounded attacks using a 2-step PGD attack (Madry et al., 2018) with step size α = 1 / 255 .
Revised: Unless specified, during adversarial training, we generate L inf = 1 / 255 bounded attacks using a 2-step PGD attack (Madry et al., 2018) with step size α = 1 / 255 .

Original: We test the robustness of our model using 100 steps of PGD attack, with step size α = 1 / 255 .
Revised: We test the robustness of our model using 100 steps of PGD attack, with step size α = 1 / 255 .

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 4. E XPERIMENTAL R ESULTS
Revised: 4. E XPERIMENTAL R ESULTS

Original: We first use PGD attack with 100 steps to evaluate the robustness of CLIP models that are adapted on ImageNet.
Revised: We first use PGD attack with 100 steps to evaluate the robustness of CLIP models that are adapted on ImageNet.

Original: In Table 1, we show the robust accuracies for the models on the Imagenet test set and 15 zero-shot recognition tasks ( i.e., the training classes and test classes are non-overlapping).
Revised: In Table 1, we show the robust accuracies for the models on the Imagenet test set and 15 zero-shot recognition tasks ( i.e., the training classes and test classes are non-overlapping).

Original: Each row is the robust accuracy for one method, where we compare the proposed TeCoA with VP and FT against 10 baselines and their variants.
Revised: Each row is the robust accuracy for one method, where we compare the proposed TeCoA with VP and FT against 10 baselines and their variants.

Original: We bold the best results for each dataset (column).
Revised: We bold the best results for each dataset (column).

Original: From Table 1, we can see that most adapted models with adversarial training losses except for
Revised: From Table 1, we can see that most adapted models with adversarial training losses except for

Original: ImgCoAdv.
Revised: ImgCoAdv.

Original: have better robust accuracies compared to the standard cross-entropy loss.
Revised: have better robust accuracies compared to the standard cross-entropy loss.

Original: If using adversarial training, the vanilla FT (Adv.) , which finetunes the entire model with adversarial training, achieves the best adversarial robustness on ImageNet (non zero-shot data, same as the training tasks) while the average robust accuracy is 10.62%, which is only slightly better than the original CLIP (6.57%).
Revised: If using adversarial training, the vanilla FT (Adv.) , which finetunes the entire model with adversarial training, achieves the best adversarial robustness on ImageNet (non zero-shot data, same as the training tasks) while the average robust accuracy is 10.62%, which is only slightly better than the original CLIP (6.57%).

Original: In the meantime, VP(Adv.) achieves much better results, improving the
Revised: In the meantime, VP(Adv.) achieves much better results, improving the

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: accuracy number from 6.57% to 31.84%.
Revised: accuracy number from 6.57% to 31.84%.

Original: This indicates that visual prompting is more powerful than finetuning when coupled with the standard adversarial training.
Revised: This indicates that visual prompting is more powerful than finetuning when coupled with the standard adversarial training.

Original: Within each set of the variants using the same adaptation method
Revised: Within each set of the variants using the same adaptation method

Original: [Figure]
Revised: [Figure]

Original: we compare the effectiveness of different training losses.
Revised: we compare the effectiveness of different training losses.

Original: We notice that both CoAdv.
Revised: We notice that both CoAdv.

Original: and ImgCoAdv.
Revised: and ImgCoAdv.

Original: are much worse than the proposed TeCoA and even the standard adversarial training ( Adv. ).
Revised: are much worse than the proposed TeCoA and even the standard adversarial training ( Adv. ).

Original: This indicates that the formulation of contrastive learning may not necessarily help improve the zero-shot adversarial robustness and the use of the language supervision might.
Revised: This indicates that the formulation of contrastive learning may not necessarily help improve the zero-shot adversarial robustness and the use of the language supervision might.

Original: Overall, we find that adapting CLIP with TeCoA using model finetuning presents the best performance across the datasets, improving the accuracy number from 6.57% to 38.18%, roughlypoints.
Revised: Overall, we find that adapting CLIP with TeCoA using model finetuning presents the best performance across the datasets, improving the accuracy number from 6.57% to 38.18%, roughlypoints.

Original: This might look counter-intuitive as VP significantly outperforms FT under the standard adversarial training without texts.
Revised: This might look counter-intuitive as VP significantly outperforms FT under the standard adversarial training without texts.

Original: One hypothesis is that with sufficient semantic information at test time, finetuning that directly modifies the model parameters may be more expressive than just modifying the inputs given more model parameters are tuned.
Revised: One hypothesis is that with sufficient semantic information at test time, finetuning that directly modifies the model parameters may be more expressive than just modifying the inputs given more model parameters are tuned.

Original: In Table 3 in Appendix, we show the clean accuracy of the same models, which gives similar trend as the robust accuracies.
Revised: In Table 3 in Appendix, we show the clean accuracy of the same models, which gives similar trend as the robust accuracies.

Original: We also show results under ϵ = 1 / 255 and ϵ = 4 / 255 using AutoAttack Croce & Hein (2020) in Tableand Table 5, where our model is still more robust than baselines, up to an average of 36 points.
Revised: We also show results under ϵ = 1 / 255 and ϵ = 4 / 255 using AutoAttack Croce & Hein (2020) in Tableand Table 5, where our model is still more robust than baselines, up to an average of 36 points.

Original: We describe details in Section A.
Revised: We describe details in Section A.

Original: 4. A NALYSIS
Revised: 4. A NALYSIS

Original: Training Set Size is an important factor when adapting large-scale models.
Revised: Training Set Size is an important factor when adapting large-scale models.

Original: In Figure 4, we show the results of adapting CLIP with TeCoA with 1, 5, 50, and 1000 shot per category on ImageNet.
Revised: In Figure 4, we show the results of adapting CLIP with TeCoA with 1, 5, 50, and 1000 shot per category on ImageNet.

Original: We observe that increasing the amount of training data improves robust accuracy.
Revised: We observe that increasing the amount of training data improves robust accuracy.

Original: Moreover, we also find that FT(TeCoA) outperforms the non-adapted CLIP even when the model is adapted with just one shot per class (the blue bars in Figure 4).
Revised: Moreover, we also find that FT(TeCoA) outperforms the non-adapted CLIP even when the model is adapted with just one shot per class (the blue bars in Figure 4).

Original: Effect of Attack Strength.
Revised: Effect of Attack Strength.

Original: To validate whether our method works when the adversarial perturbations become larger, we increase the perturbation bound for our TeCoA adversarial training.
Revised: To validate whether our method works when the adversarial perturbations become larger, we increase the perturbation bound for our TeCoA adversarial training.

Original: Figureshows that, while increasing the attack strength decreases the robust accuracy, our model can still transfer robustness to zero-shot tasks.
Revised: Figureshows that, while increasing the attack strength decreases the robust accuracy, our model can still transfer robustness to zero-shot tasks.

Original: Visual Prompt Designs.
Revised: Visual Prompt Designs.

Original: There are two ways to design visual prompts.
Revised: There are two ways to design visual prompts.

Original: One is to append additional tokens to image tokens and the other is to add small decoration ( i.e., learnable noises) to the in-
Revised: One is to append additional tokens to image tokens and the other is to add small decoration ( i.e., learnable noises) to the in-

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: C
Revised: C

Original: [Table]
Revised: [Table]

Original: put images.
Revised: put images.

Original: From Figure 6a, we can see appending learnable tokens in the input token sequences achieves consistently better performance than just adding the prompt value to the images.
Revised: From Figure 6a, we can see appending learnable tokens in the input token sequences achieves consistently better performance than just adding the prompt value to the images.

Original: Number of Adapted Parameters.
Revised: Number of Adapted Parameters.

Original: The number of parameters that are adapted during training highly affects model performance.
Revised: The number of parameters that are adapted during training highly affects model performance.

Original: VP is light-weight, as it only modifies the inputs, while FT may either adjust part of the model or the entire model.
Revised: VP is light-weight, as it only modifies the inputs, while FT may either adjust part of the model or the entire model.

Original: In Figure 6b, we show the comparison of partially finetuning and visual prompting of CLIP.
Revised: In Figure 6b, we show the comparison of partially finetuning and visual prompting of CLIP.

Original: We can see that with the same amount of parameters adapted, visual prompting is more effective in adapting the large scale model for zeroshot adversarial robustness.
Revised: We can see that with the same amount of parameters adapted, visual prompting is more effective in adapting the large scale model for zeroshot adversarial robustness.

Original: Are Labels Required for Adapting CLIP?
Revised: Are Labels Required for Adapting CLIP?

Original: Unlabeled data is rich.
Revised: Unlabeled data is rich.

Original: We investigate if it’s necessary to use the groundtruth labels from images.
Revised: We investigate if it’s necessary to use the groundtruth labels from images.

Original: We generate the pseudo text labels that CLIP retrieves from clean images, where we show details in Section A.3.5.
Revised: We generate the pseudo text labels that CLIP retrieves from clean images, where we show details in Section A.3.5.

Original: We then run our TeCoA on the pseudo labels in the same way as the labelled data.
Revised: We then run our TeCoA on the pseudo labels in the same way as the labelled data.

Original: We show experimental results in Table 2, where we obtain a similar zero-shot robust accuracy even without labels.
Revised: We show experimental results in Table 2, where we obtain a similar zero-shot robust accuracy even without labels.

Original: Trading off between Robust Accuracy and Clean Accuracy.
Revised: Trading off between Robust Accuracy and Clean Accuracy.

Original: Similar to typical adversarial training, TeCoA also poses a trade-off between the clean accuracy and the robust accuracy Tsipras et al.
Revised: Similar to typical adversarial training, TeCoA also poses a trade-off between the clean accuracy and the robust accuracy Tsipras et al.

Original: Ideally, we want to be able to dynamically adjust this trade-off depending on the desired level of adversarial robustness.
Revised: Ideally, we want to be able to dynamically adjust this trade-off depending on the desired level of adversarial robustness.

Original: Here we can balance this trade-off by using model weights interpolation (Wortsman et al., 2022).
Revised: Here we can balance this trade-off by using model weights interpolation (Wortsman et al., 2022).

Original: In Figure 7, we can see there is a sweet spot in interpolating the model where we improve both the robustness and clean accuracy, marked by the star in the figure.
Revised: In Figure 7, we can see there is a sweet spot in interpolating the model where we improve both the robustness and clean accuracy, marked by the star in the figure.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we provided a holistic study of the zero-shot adversarial robustness problem of largescale vision-language models.
Revised: In this paper, we provided a holistic study of the zero-shot adversarial robustness problem of largescale vision-language models.

Original: We identified the effects of various adaption methods and training losses when adapting models, and conjectured that existing methods failed to generalize to new tasks due to the lack of the language supervision.
Revised: We identified the effects of various adaption methods and training losses when adapting models, and conjectured that existing methods failed to generalize to new tasks due to the lack of the language supervision.

Original: We proposed a text-guided contrastive adversarial training (TeCoA) which can be used with model finetuning and visual prompting to drastically improve the zero-shot adversarial robustness of CLIP.
Revised: We proposed a text-guided contrastive adversarial training (TeCoA) which can be used with model finetuning and visual prompting to drastically improve the zero-shot adversarial robustness of CLIP.

Original: Extensive experimental evaluation showed the effectiveness of TeCoA and the detailed analyses provide useful lessons for adapting large-scale models to improve their zero-shot adversarial robustness, shedding light on this important problem
Revised: Extensive experimental evaluation showed the effectiveness of TeCoA and the detailed analyses provide useful lessons for adapting large-scale models to improve their zero-shot adversarial robustness, shedding light on this important problem

Original: A A PPENDIX A. E XPERIMENTS A.1. Z ERO - SHOT C LEAN A CCURACY OF OUR A DAPTED M ODEL
Revised: A A PPENDIX A. E XPERIMENTS A.1. Z ERO - SHOT C LEAN A CCURACY OF OUR A DAPTED M ODEL

Original: We show the results for accuracy on clean images in Table 3.
Revised: We show the results for accuracy on clean images in Table 3.

Original: A. A UTO A TTACK E XPERIMENT
Revised: A. A UTO A TTACK E XPERIMENT

Original: We also consider a stronger attack AutoAttack Croce & Hein (2020) in our evaluation.
Revised: We also consider a stronger attack AutoAttack Croce & Hein (2020) in our evaluation.

Original: Since our method uses adversarial training and does not rely on the obfuscated gradient, we use two APGD variants, APGD-CE and APGD-DLR, in AutoAttack to evaluate.
Revised: Since our method uses adversarial training and does not rely on the obfuscated gradient, we use two APGD variants, APGD-CE and APGD-DLR, in AutoAttack to evaluate.

Original: We show robust accuracy under perturbation bound ϵ = 1 / 255 in Table 4 and robust accuracy under perturbation bound ϵ
Revised: We show robust accuracy under perturbation bound ϵ = 1 / 255 in Table 4 and robust accuracy under perturbation bound ϵ

Original: = 

Revised: =

Original: /in
Revised: /in

Original: Table 5.
Revised: Table 5.

Original: For both perturbation bounds, our method achieves higher robust accuracy than vanilla
Revised: For both perturbation bounds, our method achieves higher robust accuracy than vanilla

Original: CLIP by up to 36 points on average.
Revised: CLIP by up to 36 points on average.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: I
Revised: I

Original: [Table]
Revised: [Table]

Original: a
Revised: a

Original: [Table]
Revised: [Table]

Original: e
Revised: e

Original: N e t
Revised: N e t

Original: C
Revised: C

Original: I F
Revised: I F

Original: A
Revised: A

Original: [Table]
Revised: [Table]

Original: S TLC
Revised: S TLC

Original: I F A
Revised: I F A

Original: R
Revised: R

Original: S UN
Revised: S UN

Original: S t a n f o r d C a r s
Revised: S t a n f o r d C a r s

Original: F ood
Revised: F ood

Original: O x f o r d
Revised: O x f o r d

Original: P e t
Revised: P e t

Original: F l o w e r
Revised: F l o w e r

Original: D
Revised: D

Original: [Table]
Revised: [Table]

Original: D
Revised: D

Original: E U R O
Revised: E U R O

Original: S A T
Revised: S A T

Original: F GV
Revised: F GV

Original: C
Revised: C

Original: H a t e M e m e s
Revised: H a t e M e m e s

Original: P C A
Revised: P C A

Original: M
Revised: M

Original: C a lt ec h C a lt ec h
Revised: C a lt ec h C a lt ec h

Original: A v er ag e
Revised: A v er ag e

Original: [Table]
Revised: [Table]

Original: Notably, on ϵ = 1 / 255 , even evaluated under the stronger AutoAttack, the robustness accuracy of FT (TeCoA) is 37.32, which is still higher than all other baselines methods from Table 1 in robust accuracy, even though those baselines are evaluated under a weaker PGD100 attack .
Revised: Notably, on ϵ = 1 / 255 , even evaluated under the stronger AutoAttack, the robustness accuracy of FT (TeCoA) is 37.32, which is still higher than all other baselines methods from Table 1 in robust accuracy, even though those baselines are evaluated under a weaker PGD100 attack .

Original: A larger perturbation bound ϵ = 4 / 255 makes the attack stronger, where our method still improves robustness by an average of 9 points.
Revised: A larger perturbation bound ϵ = 4 / 255 makes the attack stronger, where our method still improves robustness by an average of 9 points.

Original: In addition, while the AutoAttack significantly reduces the robust accuracy of CLIP from 6.57 to 0.85, it only slightly decreases our TeCoA’s robust accuracy: 2.5 points for visual prompt tuning and 0.65 for finetuning model (see Table 1 and Table 4).
Revised: In addition, while the AutoAttack significantly reduces the robust accuracy of CLIP from 6.57 to 0.85, it only slightly decreases our TeCoA’s robust accuracy: 2.5 points for visual prompt tuning and 0.65 for finetuning model (see Table 1 and Table 4).

Original: One reason for AutoAttack to be so effective in attacking vanilla CLIP than PGD100 is because it uses a fractional attack vector, which is not rounded by 1/255 during the inference.
Revised: One reason for AutoAttack to be so effective in attacking vanilla CLIP than PGD100 is because it uses a fractional attack vector, which is not rounded by 1/255 during the inference.

Original: Images are often encoded via integer from 0 to 255, which allows only attack at the integer level.
Revised: Images are often encoded via integer from 0 to 255, which allows only attack at the integer level.

Original: In the main paper, we use PGD attacks with step size 1 (if the image ranges from 0 to 1, then step size is proportionally 1/255) for 100 steps.
Revised: In the main paper, we use PGD attacks with step size 1 (if the image ranges from 0 to 1, then step size is proportionally 1/255) for 100 steps.

Original: Since there is no fractional attack value, the attack space is constrained and less effective.
Revised: Since there is no fractional attack value, the attack space is constrained and less effective.

Original: This is because the standard image inputs have value resolutions that should be larger than 0.5, and any values smaller than this would be rounded when encoding the images.
Revised: This is because the standard image inputs have value resolutions that should be larger than 0.5, and any values smaller than this would be rounded when encoding the images.

Original: If we ignore the fact that images will be encoded in integers between 0 to 255, then we can have stronger attacks by exploring the fraction values.
Revised: If we ignore the fact that images will be encoded in integers between 0 to 255, then we can have stronger attacks by exploring the fraction values.

Original: Since the AutoAttack automatically reduces the attack step size when loss oscillates, it explores the fraction space and is more effective in the attack.
Revised: Since the AutoAttack automatically reduces the attack step size when loss oscillates, it explores the fraction space and is more effective in the attack.

Original: [Table]
Revised: [Table]

Original: T RAINING L OSSES AND A LGORITHMS
Revised: T RAINING L OSSES AND A LGORITHMS

Original: We give formulations for the different training algorithms considered in our experiments.
Revised: We give formulations for the different training algorithms considered in our experiments.

Original: Throughout this section, let F θ denote an image encoder parameterized by θ , and let T denote a frozen text encoder.
Revised: Throughout this section, let F θ denote an image encoder parameterized by θ , and let T denote a frozen text encoder.

Original: Let D denote a dataset containing pairs ( x, y ) of images and their respective one-hot labels.
Revised: Let D denote a dataset containing pairs ( x, y ) of images and their respective one-hot labels.

Original: A.3. S TANDARD A DVERSARIAL T RAINING WITH C ROSS - ENTROPY L OSS (A DV .)
Revised: A.3. S TANDARD A DVERSARIAL T RAINING WITH C ROSS - ENTROPY L OSS (A DV .)

Original: The standard adversarial training paradigm.
Revised: The standard adversarial training paradigm.

Original: We initialize a learnable linear layer C ϕ , and append it to F θ .
Revised: We initialize a learnable linear layer C ϕ , and append it to F θ .

Original: The classification loss is L ( C ϕ ( F θ ( x a )) , y ) defined in cross-entropy loss.
Revised: The classification loss is L ( C ϕ ( F θ ( x a )) , y ) defined in cross-entropy loss.

Original: We first train C ϕ on standard images.
Revised: We first train C ϕ on standard images.

Original: Then, given a natural image x with one-hot label y , we generate an attacked image x a by maximizing the loss L ( C ϕ ( F θ ( x a )) , y ) .
Revised: Then, given a natural image x with one-hot label y , we generate an attacked image x a by maximizing the loss L ( C ϕ ( F θ ( x a )) , y ) .

Original: We then update θ to minimize the loss L ( C ϕ ( F θ ( x a )) , y ) .
Revised: We then update θ to minimize the loss L ( C ϕ ( F θ ( x a )) , y ) .

Original: We describe our algorithm in Algorithm 1.
Revised: We describe our algorithm in Algorithm 1.

Original: Algorithm
Revised: Algorithm

Original: Input: Dataset D , learnable parameter θ , model F , parameter of projector C ϕ for all iter ∈ preset number of training epochs do for all x , y ∈ minibatch do x a = argmax x a L ( C ϕ ( F θ ( x a )) , y )
Revised: Input: Dataset D , learnable parameter θ , model F , parameter of projector C ϕ for all iter ∈ preset number of training epochs do for all x , y ∈ minibatch do x a = argmax x a L ( C ϕ ( F θ ( x a )) , y )

Original: ▷ Generating adversarial attacks θ = θ − ∇ θ L ( C ϕ ( F θ ( x a )) , y )
Revised: ▷ Generating adversarial attacks θ = θ − ∇ θ L ( C ϕ ( F θ ( x a )) , y )

Original: ▷ Training on generated adversarial examples end for end for
Revised: ▷ Training on generated adversarial examples end for end for

Original: A.3. C ONTRASTIVE A DVERSARIAL T RAINING L OSS (C O A DV .)
Revised: A.3. C ONTRASTIVE A DVERSARIAL T RAINING L OSS (C O A DV .)

Original: We study how much the contrastive learning objective contributes to the zero-shot robustness gain.
Revised: We study how much the contrastive learning objective contributes to the zero-shot robustness gain.

Original: Instead of using one-hot label y and cross-entropy loss in our objective, we create a dictionary of embeddings E by random initialization, where each embedding e i denotes the code representation for the category
Revised: Instead of using one-hot label y and cross-entropy loss in our objective, we create a dictionary of embeddings E by random initialization, where each embedding e i denotes the code representation for the category

Original: y i .
Revised: y i .

Original: We optimize the following contrastive learning loss:
Revised: We optimize the following contrastive learning loss:

Original: [Equation]
Revised: [Equation]

Original: where the z ( I ) i =
Revised: where the z ( I ) i =

Original: F θ ( x i ) are the features of the input image, and e j are the code representation from the dictionary.
Revised: F θ ( x i ) are the features of the input image, and e j are the code representation from the dictionary.

Original: We use y ij to indicate which image-code pairs are positive and which are negative; this indicator satisfies y ij = 1 if and only if the examples i = j , and 0 otherwise.
Revised: We use y ij to indicate which image-code pairs are positive and which are negative; this indicator satisfies y ij = 1 if and only if the examples i = j , and 0 otherwise.

Original: τ is a scalar hyper-parameter, and cos denotes the cosine similarity function.
Revised: τ is a scalar hyper-parameter, and cos denotes the cosine similarity function.

Original: We describe our algorithm in Algorithm 2.
Revised: We describe our algorithm in Algorithm 2.

Original: Input: Dataset D , learnable parameter θ , model F for all iter ∈ preset number of training epochs do for all x , y ∈ minibatch do x
Revised: Input: Dataset D , learnable parameter θ , model F for all iter ∈ preset number of training epochs do for all x , y ∈ minibatch do x

Original: [Equation]
Revised: [Equation]

Original: = argmax x a L s ( x , E , y )
Revised: = argmax x a L s ( x , E , y )

Original: ▷ Generating adversarial attacks for contrastive loss θ = θ − ∇ θ L s ( x , E , y )
Revised: ▷ Generating adversarial attacks for contrastive loss θ = θ − ∇ θ L s ( x , E , y )

Original: ▷ Contrastive learning on generated adversarial examples end for end for
Revised: ▷ Contrastive learning on generated adversarial examples end for end for

Original: A.3. C ONTRASTIVE A DVERSARIAL T RAINING OVER I MAGES (I MG C O A DV .)
Revised: A.3. C ONTRASTIVE A DVERSARIAL T RAINING OVER I MAGES (I MG C O A DV .)

Original: Prior work Jiang et al.
Revised: Prior work Jiang et al.

Original: (2020) uses image-only contrastive adversarial learning to obtain robustness.
Revised: (2020) uses image-only contrastive adversarial learning to obtain robustness.

Original: We adapt this method as a baseline to study whether using the knowledge from only images — not language — can achieve zero-shot robustness.
Revised: We adapt this method as a baseline to study whether using the knowledge from only images — not language — can achieve zero-shot robustness.

Original: For each image x i , we create a transformationed x j , and form the image pair ( x i , x j ) .
Revised: For each image x i , we create a transformationed x j , and form the image pair ( x i , x j ) .

Original: We use the same visual encoder to embed the images x i and x j to obtain the features z i and z j .
Revised: We use the same visual encoder to embed the images x i and x j to obtain the features z i and z j .

Original: We then construct the following contrastive learning loss:
Revised: We then construct the following contrastive learning loss:

Original: [Equation]
Revised: [Equation]

Original: where the z i
Revised: where the z i

Original: =
Revised: =

Original: F θ ( x i ) are the features of the input image.
Revised: F θ ( x i ) are the features of the input image.

Original: We use y ij to indicate which image pairs are positive and which are negative; this indicator satisfies y ij = 1 if and only if the images x i and x j are augmented from the same instance, and 0 otherwise.
Revised: We use y ij to indicate which image pairs are positive and which are negative; this indicator satisfies y ij = 1 if and only if the images x i and x j are augmented from the same instance, and 0 otherwise.

Original: τ is a scalar hyper-parameter, and cos denotes the cosine similarity function.
Revised: τ is a scalar hyper-parameter, and cos denotes the cosine similarity function.

Original: Let z ai = F θ ( x ai ) , where x ai denotes the generated adversarial examples.
Revised: Let z ai = F θ ( x ai ) , where x ai denotes the generated adversarial examples.

Original: Then we can obtain the adversarial examples via:
Revised: Then we can obtain the adversarial examples via:

Original: [Equation]
Revised: [Equation]

Original: i , x j , y ) =
Revised: i , x j , y ) =

Original: argmax x ai − E i,j (cid:20) y ij log exp(cos( z ai , z j ) /τ ) (cid:80) k exp(cos( z ai , z j ) /τ ) (cid:21) , (8)
Revised: argmax x ai − E i,j (cid:20) y ij log exp(cos( z ai , z j ) /τ ) (cid:80) k exp(cos( z ai , z j ) /τ ) (cid:21) , (8)

Original: Once we generate the adversarial images, we conduct contrastive learning on adversarial images and the paired clean images using Equation 7.
Revised: Once we generate the adversarial images, we conduct contrastive learning on adversarial images and the paired clean images using Equation 7.

Original: Algorithm
Revised: Algorithm

Original: Input: Dataset D , learnable parameter θ , model F for all iter ∈ preset number of training epochs do for all x ∈ minibatch do x
Revised: Input: Dataset D , learnable parameter θ , model F for all iter ∈ preset number of training epochs do for all x ∈ minibatch do x

Original: [Equation]
Revised: [Equation]

Original: = argmax
Revised: = argmax

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ▷ Generating adversarial attacks for contrastive loss θ = θ
Revised: ▷ Generating adversarial attacks for contrastive loss θ = θ

Original: [Equation]
Revised: [Equation]

Original: θ L s
Revised: θ L s

Original: [Equation]
Revised: [Equation]

Original: x ai , x
Revised: x ai , x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ▷ Contrastive learning on generated adversarial examples end for end for
Revised: ▷ Contrastive learning on generated adversarial examples end for end for

Original: We introduce our algorithm in Algorithm 3.
Revised: We introduce our algorithm in Algorithm 3.

Original: A.3. T EXT -G UIDED C ONTRASTIVE A DVERSARIAL T RAINING (T E C O A)
Revised: A.3. T EXT -G UIDED C ONTRASTIVE A DVERSARIAL T RAINING (T E C O A)

Original: We describe the TeCoA training algorithm in Algorithm 4.
Revised: We describe the TeCoA training algorithm in Algorithm 4.

Original: We denote the learnable parameters to be θ .
Revised: We denote the learnable parameters to be θ .

Original: For the visual prompt tuning, θ is only the prompt vector.
Revised: For the visual prompt tuning, θ is only the prompt vector.

Original: For the finetuning method, θ is the parameter of the whole model.
Revised: For the finetuning method, θ is the parameter of the whole model.

Original: A.3. T E C O A L EARNING ON U NLABELED D ATA
Revised: A.3. T E C O A L EARNING ON U NLABELED D ATA

Original: Given an unlabeled image, we first provide a list of text using the possible category names:
Revised: Given an unlabeled image, we first provide a list of text using the possible category names:

Original: A photo of a {Category Name}.
Revised: A photo of a {Category Name}.

Original: Input: Dataset D , learnable parameter θ , model F , text t for all iter ∈ preset number of training epochs do for all x , y ∈ minibatch do x a = argmax x L s ( x , t , y )
Revised: Input: Dataset D , learnable parameter θ , model F , text t for all iter ∈ preset number of training epochs do for all x , y ∈ minibatch do x a = argmax x L s ( x , t , y )

Original: ▷ Generating adversarial attacks for contrastive loss θ = θ − ∇ θ L s ( x a , t , y )
Revised: ▷ Generating adversarial attacks for contrastive loss θ = θ − ∇ θ L s ( x a , t , y )

Original: ▷ Contrastive learning on generated adversarial examples end for end for
Revised: ▷ Contrastive learning on generated adversarial examples end for end for

Original: Since the unlabeled images are not attacked, CLIP can retrieve the nearest text embedding from the image embedding and use the text as the pseudo label for the image.
Revised: Since the unlabeled images are not attacked, CLIP can retrieve the nearest text embedding from the image embedding and use the text as the pseudo label for the image.

Original: We then conduct the TeCoA training on the images and their pseudo text label.
Revised: We then conduct the TeCoA training on the images and their pseudo text label.

Original: We describe the algorithm below in Algorithm 5. 
Revised: We describe the algorithm below in Algorithm 5. Algorithm

Original: Input for all iter ∈ preset number of training epochs do for all x ∈ minibatch B = { x 1 , . . .
Revised: Input: Dataset D without , learnable parameter θ , F , for all iter ∈ preset number of training epochs do for all x ∈ minibatch B = { x 1 , . . .

Original: , x m } do y = argmin
Revised: , x m } do y = argmin

Original: y L s ( x , t , y )
Revised: y L s ( x , t , y )

Original: ▷ Finding pseudo label for the clean images using CLIP x a = argmax x
Revised: ▷ Finding pseudo label for the clean images using CLIP x a = argmax x

Original: L s ( x , t , y )
Revised: L s ( x , t , y )

Original: ▷ Generating adversarial attacks for contrastive loss θ = θ − ∇ θ L s ( x a , t , y )
Revised: ▷ Generating adversarial attacks for contrastive loss θ = θ − ∇ θ L s ( x a , t , y )

Original: ▷ Contrastive learning on generated adversarial examples end for end for
Revised: ▷ Contrastive learning on generated adversarial examples end for end for

Original: A. D ISCUSSION FOR R ESULTS
Revised: A. D ISCUSSION FOR R ESULTS

Original: In Table 1, our TeCoA performs better than existing methods except for LP(CE) and VPT(Adv.)
Revised: In Table 1, our TeCoA performs better than existing methods except for LP(CE) and VPT(Adv.)

Original: on
Revised: on

Original: HateMemes and PCAM datasets.
Revised: HateMemes and PCAM datasets.

Original: This is because the HateMemes dataset is a binary classification task for detecting hateful speech, which is a very different domain from the ImageNet classification.
Revised: This is because the HateMemes dataset is a binary classification task for detecting hateful speech, which is a very different domain from the ImageNet classification.

Original: Since both LP(CE) and VPT (adv) only adapt a small number of parameters on the ImageNet set, the resulting model may overfit less to the Image recognition task, and just perform random guessing.
Revised: Since both LP(CE) and VPT (adv) only adapt a small number of parameters on the ImageNet set, the resulting model may overfit less to the Image recognition task, and just perform random guessing.

Original: Note that the 54% accuracy is close to random guessing 50%.
Revised: Note that the 54% accuracy is close to random guessing 50%.

Original: In addition, PCAM is a binary classification for lymph nodes (medical image, https://github.com/basveeling/pcam), which is also a very different domain from ImageNet.
Revised: In addition, PCAM is a binary classification for lymph nodes (medical image, https://github.com/basveeling/pcam), which is also a very different domain from ImageNet.

Original: Similar to HateMemes, adapting fewer parameters makes the model learn less and overfit less, where the 52.5% accuracy is close to random guessing 50%.
Revised: Similar to HateMemes, adapting fewer parameters makes the model learn less and overfit less, where the 52.5% accuracy is close to random guessing 50%.

Original: Thus, both datasets remain a big challenge for all existing zero-shot robust classification tasks.
Revised: Thus, both datasets remain a big challenge for all existing zero-shot robust classification tasks.

Original: A. D ISCUSSION FOR T E C O A L OSS
Revised: A. D ISCUSSION FOR T E C O A L OSS

Original: In the main paper, we interpret our loss through the image-text contrastive objective, which first conducts a matrix multiplication between the image embedding and language embedding, and then applies a cross-entropy loss on the output.
Revised: In the main paper, we interpret our loss through the image-text contrastive objective, which first conducts a matrix multiplication between the image embedding and language embedding, and then applies a cross-entropy loss on the output.

Original: Since the text embedding is fixed, this embedding can be treated as a layer of linear classifier, whose weights are obtained from the language embedding.
Revised: Since the text embedding is fixed, this embedding can be treated as a layer of linear classifier, whose weights are obtained from the language embedding.

Original: This image-text contrastive objective can also be interpreted as using cross-entropy loss on a fixed readout layer that is initialized with the right language knowledge.
Revised: This image-text contrastive objective can also be interpreted as using cross-entropy loss on a fixed readout layer that is initialized with the right language knowledge.

Original: This further validates the importance of language information for zero-shot adversarial robustness.
Revised: This further validates the importance of language information for zero-shot adversarial robustness.

Revised: A. A DAPTATION M ETHOD F ORMULATION

Revised: Token-Level Visual Prompts.

Revised: Token-level visual prompts adapt transformer-based models by appending tokens to the input token sequence.

Revised: This is the most effective prompt method in our experiments, where we use this by default unless specified.

Revised: Our visual prompts append additional tokens

Revised: P k to the input sequence x of the vision transformer:

Revised: [Equation]

Revised: The remaining transformer parameters and computations are kept the same as the original.

Revised: Image-Level Visual Prompts.

Revised: Image-level visual prompts adapt transformer models by adding prompt to the input pixels.

Revised: This is less effective way of adding prompt, as discussed in Figure 6a.

Revised: Let the prompt token be P , and input image be x the visual prompt is added to the input image:

Revised: [Equation]

Revised: The remaining transformer parameters and computations are kept the same as the original.

Revised: Finetuning.

Revised: This is the standard way of adapting a model, where all parameters of the model is updated with relatively small learning rate.

Revised: In our experiments, we find learning rate of 1 e −achieves the best performance.