O N I MPROVING A DVERSARIAL T RANSFERABILITY OF V ISION T RANSFORMERS
A BSTRACT
Vision transformers (ViTs) process input images as sequences of patches via selfattention; a radically different architecture than convolutional neural networks (CNNs).
This makes it interesting to study the adversarial feature space of ViT models and their transferability.
In particular, we observe that adversarial patterns found via conventional adversarial attacks show very low black-box transferability even for large ViT models.
We show that this phenomenon is only due to the sub-optimal attack procedures that do not leverage the true representation potential of ViTs.
A deep ViT is composed of multiple blocks, with a consistent architecture comprising of self-attention and feed-forward layers, where each block is capable of independently producing a class token.
Formulating an attack using only the last class token (conventional approach) does not directly leverage the discriminative information stored in the earlier tokens, leading to poor adversarial transferability of ViTs.
Using the compositional nature of ViT models, we enhance transferability of existing attacks by introducing two novel strategies speciﬁc to the architecture of ViT models.
(i) Self-Ensemble: We propose a method to ﬁnd multiple discriminative pathways by dissecting a single ViT model into an ensemble of networks.
This allows explicitly utilizing class-speciﬁc information at each ViT block.
(ii)
Token Reﬁnement: We then propose to reﬁne the tokens to further enhance the discriminative capacity at each block of ViT.
Our token reﬁnement systematically combines the class tokens with structural information preserved within the patch tokens.
An adversarial attack when applied to such reﬁned tokens within the en- semble of classiﬁers found in a single vision transformer has signiﬁcantly higher transferability and thereby brings out the true generalization potential of the ViT’s adversarial space.
Code: https://t.ly/hBbW .
I NTRODUCTION
Transformers compose a family of neural network architectures based on the self-attention mechanism, originally applied in natural language processing tasks achieving state-of-the-art performance (Vaswani et al., 2017; Devlin et al., 2018; Brown et al., 2020).
The transformer design has been subsequently adopted for vision tasks (Dosovitskiy et al., 2020), giving rise to a number of successful vision transformer (ViT) models (Touvron et al., 2020; Yuan et al., 2021;
Khan et al., 2021).
Due to the lack of explicit inductive biases in their design, ViTs are inherently different from convolutional neural networks (CNNs) that encode biases e.g., spatial connectivity and translation equivariance.
ViTs process an image as a sequence of patches which are reﬁned through a series of self-attention mechanisms (transformer blocks), allowing the network to learn relationships between any individual parts of the input image.
Such processing allows wide receptive ﬁelds which can model global context as opposed to the limited receptive ﬁelds of CNNs.
These signiﬁcant differences between ViTs and
CNNs give rise to a range of intriguing characteristics unique to ViTs (Caron et al., 2021;
Tuli et al., 2021; Mao et al., 2021; Paul & Chen, 2021; Naseer et al., 2021b).
Adversarial attacks pose a major hindrance to the successful deployment of deep neural networks in real-world applications.
Recent success of ViTs means that adversarial properties of ViT models also become an important research topic.
A few recent works explore adversarial robustness of
ViTs (Shao et al., 2021; Mahmood et al., 2021; Bhojanapalli et al., 2021) in different attack settings.
Surprisingly, these works show that large ViT models exhibit lower transferability in black-box attack setting, despite their higher parameter capacity, stronger performance on clean images, and better generalization (Shao et al., 2021;
Mahmood et al., 2021).
This ﬁnding seems to indicate that as ViT performance improves, its adversarial feature space gets weaker.
In this work, we investigate whether the weak transferability of adversarial patterns from high-performing ViT models, as reported in recent works (Shao et al., 2021; Mahmood et al., 2021; Bhojanapalli et al., 2021), is a result of weak features or a weak attack.
To this end, we introduce a highly transferable attack approach that augments the current adversarial attacks and increase their transferability from ViTs to the unknown models.
Our proposed transferable attack leverages two key concepts, multiple discriminative pathways and token reﬁnement, which exploit unique characteristics of ViT models.
Our approach is motivated by the modular nature of ViTs (Touvron et al., 2020; Yuan et al., 2021; Mao et al., 2021): they process a sequence of input image patches repeatedly using multiple multi-headed self-attention layers (transformer blocks)
(Vaswani et al., 2017).
We refer to the representation of patches at each transformer block as patch tokens .
An additional randomly initialized vector ( class token 1 ) is also appended to the set of patch tokens along the network depth to distill discriminative information across patches.
The collective set of tokens is passed through the multiple transformer blocks followed by passing of the class token through a linear classiﬁer (head) which is used to make the ﬁnal prediction.
The class token interacts with the patch tokens within each block and is trained gradually across the blocks until it is ﬁnally utilized by the linear classiﬁer head to obtain class-speciﬁc logit values.
The class token can be viewed as extracting information useful for the ﬁnal prediction from the set of patch tokens at each block.
Given the role of the class token in ViT models, we observe that class tokens can be extracted from the output of each block and each such token can be used to obtain a class-speciﬁc logit output using the ﬁnal classiﬁer of a pretrained model.
This leads us to the proposed self-ensemble of models within a single transformer (Fig. 1).
We show that attacking such a self-ensemble (Sec. 3) containing multiple discriminative pathways signiﬁcantly improves adversarial transferability from ViT models, and in particular from the large ViTs.
Going one step further, we study if the class information extracted from different intermediate ViT blocks (of the self-ensemble) can be enhanced to improve adversarial transferability.
To this end, we introduce a novel token reﬁnement module directed at enhancing these multiple discriminative pathways.
The token reﬁnement module strives to reﬁne the information contained in the output of each transformer block (within a single ViT model) and aligns the class tokens produced by the intermediate blocks with the ﬁnal classiﬁer in order to maximize the discriminative power of intermediate blocks.
Our token reﬁnement exploits the structural information stored in the patch tokens and fuses it with the class token to maximize the discriminative performance of each block.
Both the reﬁned tokens and self-ensemble ideas are combined to design an adversarial attack that is shown to signiﬁcantly boost the transferability of adversarial examples, thereby bringing out the true generalization of ViTs’ adversarial space.
Through our extensive experimentation, we empirically demonstrate favorable transfer rates across different model families (convolutional and transformer) as well as different vision tasks (classiﬁcation, detection and segmentation).
B ACKGROUND AND R ELATED W ORK
Adversarial Attack Modeling: Adversarial attack methods can be broadly categorized into two categories, white-box attacks and black-box attacks.
While the white-box attack setting provides the attacker full access to the parameters of the target model, the black-box setting prevents the attacker from accessing the target model and is therefore a harder setting to study adversarial transferability.
White-box Attack: Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2014) and Projected
Gradient Descent (PGD) (Madry et al., 2018) are two initially proposed white-box attack methods.
FGSM corrupts the clean image sample by taking a single step within a small distance (perturbation budget (cid:15) ) along the objective function’s gradient direction.
PGD corrupts the clean sample for multiple steps with a smaller step size, projecting the generated adversarial example onto the (cid:15) -sphere around the clean sample after each step.
Other state-of-the-art white-box attack methods include
Jacobian-based saliency map attack (Papernot et al., 2016), Sparse attack (Modas et al., 2019), Onepixel attack (Su et al., 2019), Carlini and Wagner optimization (Carlini & Wagner, 2017), Elastic-net (Chen et al., 2018), Diversiﬁed sampling (Tashiro et al., 2020), and more recently Auto-attack (Croce & Hein, 2020b).
We apply white-box attacks on surrogate models to ﬁnd perturbations that are then transferred to black-box target models.
Black-box Attack and Transferability: Black-box attacks generally involve attacking a source model to craft adversarial signals which are then applied on the target models.
While gradient estimation methods that estimate the gradients of the target model using black-box optimization methods such as Finite Differences (FD) (Chen et al., 2017;
Bhagoji et al., 2018) or Natural Evolution
Strategies (NES) (Ilyas et al., 2018; Jiang et al., 2019) exist, these methods are dependent on multiple queries to the target model which is not practical in most real-world scenarios.
In the case of adversarial signal generation using source models, it is possible to directly adopt white-box methods.
In our work, we adopt FGSM and PGD in such a manner.
Methods like (Dong et al., 2018) incorporate a momentum term into the gradient to boost the transferability of existing white-box attacks, building attacks named MIM.
In similar spirit, different directions are explored in literature to boost transferability of adversarial examples; a) Enhanced Momentum:
Lin et al .
(Lin et al., 2019) and Wang et al .
(Wang & He, 2021) improve momentum by using Nesterov momentum and variance tuning respectively during attack iterations, b) Augmentations: Xie et al .
(Xie et al., 2019) showed that applying differentiable stochastic transformations can bring diversity to the gradients and improve transferability of the existing attacks, c)
Exploiting Features: Multiple suggestions are proposed in the literature to leverage the feature space for adversarial attack as well.
For example, Zhou et al .
(Zhou et al., 2018) incorporate the feature distortion loss during optimization.
Similarly, (Inkawhich et al., 2020b;a; Huang et al., 2019) also exploit intermediate layers to enhance transferability.
However, combining the intermediate feature response with ﬁnal classiﬁcation loss is non-trivial as it might require optimization to ﬁnd the best performing layers (Inkawhich et al., 2020b;a), and d) Generative
Approach: Orthogonal to iterative attacks, generative methods (Poursaeed et al., 2018; Naseer et al., 2019; 2021a) train an autoencoder against the white-box model.
In particular, Naseer et al .
show that transferability of an adversarial generator can be increased with relativistic cross-entropy (Naseer et al., 2019) and augmentations (Naseer et al., 2021a).
Ours is the ﬁrst work to address limited transferability of ViT models.
The Role of Network Architecture:
Recent works exploit architectural characteristics of networks to improve the transferability of attacks.
While Wu et al.
(2020) exploit skip connections of models like ResNets and DenseNets to improve black-box attacks, Guo et al.
(2020) build on similar ideas focused on the linearity of models.
Our work similarly focuses on unique architectural characteristics of ViT models to generate more transferable adversarial perturbations with the existing white-box attacks.
Robustness of ViTs: Adversarial attacks on ViT models are relatively unexplored.
Shao et al.
(2021) and Bhojanapalli et al.
(2021) investigate adversarial attacks and robustness of ViT models studying various white-box and black-box attack techniques.
The transferability of perturbations from ViT models is thoroughly explored in (Mahmood et al., 2021) and they conclude that ViT models do not transfer well to CNNs, whereas we propose a methodology to solve this shortcoming.
Moreover, Mahmood et al.
(2021) explores the idea of an ensemble of CNN and ViT models to improve the transferability of attacks.
Our proposed ensemble approach explores a different direction by converting a single ViT model into a collection of models (self-ensemble) to improve attack transferability.
In essence, our proposed method can be integrated with existing attack approaches to take full advantage of the ViTs’ learned features and generate transferable adversaries.
E NHANCING A DVERSARIAL T RANSFERABILITY OF V I T S
Preliminaries: Given a clean input image sample x with a label y , a source ViT model F and a target model M which is under-attack, the goal of an adversarial attack is generating an adversarial signal, x (cid:48) , using the information encoded within F , which can potentially change the target network’s prediction ( M ( x (cid:48) ) argmax (cid:54) =
y ).
A set of boundary conditions are also imposed on the adversarial signal to control the level of distortion in relation to the original sample, i.e., (cid:107) x − x (cid:48) (cid:107) p < (cid:15) , for a small perturbation budget (cid:15) and a p -norm, often set to inﬁnity norm ( (cid:96) ∞ ).
Motivation: The recent ﬁndings (Shao et al., 2021; Mahmood et al., 2021) demonstrate low black-box transferability of ViTs despite their higher parametric complexity and better feature generalization.
Motivated by this behaviour, we set-up a simple experiment of our own to study the adversarial transferability of ViTs (see Fig. 2).
We note that transferability of adversarial examples found via momentum iterative fast gradient sign method (Dong et al., 2018) (MIM) at (cid:96) ∞ ≤ 16 on DeiT (Touvron et al., 2020) does not increase with model capacity.
In fact, adversarial transferability from DeiT base model (DeiT-B) on ResNet152 and large vision transformer (ViT-L (Dosovitskiy et al., 2020)) is lower than DeiT tiny model (DeiT-T).
This is besides the fact that DeiT-B has richer representations and around 17 × more parameters than DeiT-T.
We investigate if this behavior is inherent to ViTs or merely due to a sub-optimal attack mechanism.
To this end, we exploit unique architectural characteristics of ViTs to ﬁrst ﬁnd an ensemble of networks within a single pretrained
ViT model (self-ensemble, right Fig. 1).
The class token produced by each self-attention block is processed by the ﬁnal local norm and classiﬁcation MLP-head to reﬁne class-speciﬁc information (Fig. 2).
In other words, our MIM E and MIM RE variants attack class information stored in the class tokens produced by all the self-attention blocks within the model and optimize for the adversarial example (Sec. 3.1 and 3.2).
Exploring the adversarial space of such multiple discriminative pathways in a self-ensemble generates highly transferable adversarial examples, as we show next.
3. S ELF -E NSEMBLE : D ISCRIMINATIVE P ATHWAYS OF V ISION T RANSFORMER
A ViT model (Dosovitskiy et al., 2020; Touvron et al., 2020), F , with n transformer blocks can be deﬁned as F = ( f 1 ◦ f 2 ◦ f 3 ◦ . . .
f n ) ◦ g , where f i represents a single ViT block comprising of multi-head self-attention and feed-forward layers and g is the ﬁnal classiﬁcation head.
To avoid notation clutter, we assume that g consists of the ﬁnal local norm and MLP-head (Touvron et al., 2020; Dosovitskiy et al., 2020).
Self-attention layer within the vision transformer model takes a sequence of m image patches as input and outputs the processed patches.
We will refer to the representations associated with the sequence of image patches as patch tokens, P t ∈ R m × d (where d is the dimensionality of each patch representation).
Attention in ViT layers is driven by minimizing the empirical risk during training.
In the case of classiﬁcation, patch tokens are further appended with the class token
( Q t ∈ R 1 × d ).
These patch and class tokens are reﬁned across multiple blocks ( f i ) and attention in these layers is guided such that the most discriminative information from patch tokens is preserved within the class token.
The ﬁnal class token is then projected to the number of classes by the classiﬁer, g .
Due to the availability of class token at each transformer block, we can create an ensemble of classiﬁers by learning a shared classiﬁcation head at each block along the ViT hierarchy.
This provides us an ensemble of n classiﬁers from a single ViT, termed as the self-ensemble :
[Equation]
g, where k = 1 , 2 , . . n.
[Equation]
We note that the multiple classiﬁers thus formed hold signiﬁcant discriminative information.
This is validated by studying the classiﬁcation performance of each classiﬁer (Eq. 1) in terms of top- (%) accuracy on ImageNet validation set, as demonstrated in Fig.
Note that multiple intermediate layers perform well on the task, especially towards the end of the ViT processing hierarchy.
For an input image x with label y , an adversarial attack can now be optimized for the ViT’s selfensemble by maximizing the loss at each ViT block.
However, we observe that initial blocks (1-6) for all considered DeiT models do not contain useful discriminative information as their classiﬁcation accuracy is almost zero (Fig. 3).
During the training of ViT models (Touvron et al., 2020; Yuan et al., 2021; Mao et al., 2021), parameters are updated based on the last class token only, which means that the intermediate tokens are not directly aligned with the ﬁnal classiﬁcation head, g in our self-ensemble approach (Fig. 3) leading to a moderate classiﬁcation performance.
To resolve this, we introduce a token reﬁnement strategy to align the class tokens with the ﬁnal classiﬁer, g , and boost their discriminative ability, which in turn helps improve attack transferability.
3. T OKEN R EFINEMENT
As mentioned above, the multiple discriminative pathways within a ViT give rise to an ensemble of classiﬁers (Eq. 1).
However, the class token produced by each attention layer is being processed by the ﬁnal classiﬁer, g .
This puts an upper bound on classiﬁcation accuracy for each token which is lower than or equal to the accuracy of the ﬁnal class token.
Our objective is to push the accuracy of the class tokens in intermediate blocks towards the upper bound as deﬁned by the last token.
For this purpose, we introduced a token reﬁnement module to ﬁne-tune the class tokens.
Our proposed token reﬁnement module is illustrated in Fig.
It acts as an intermediate layer inserted between the outputs of each block (after the shared norm layer) and the shared classiﬁer head.
Revisiting our baseline ensemble method (Fig. 1), we note that the shared classiﬁer head contains weights directly trained only on the outputs of the last transformer block.
While the class tokens of previous layers may be indirectly optimized to align with the ﬁnal classiﬁer, there exists a potential for misalignment of these features with the classiﬁer: the pretrained classiﬁer (containing weights compatible with the last layer class token) may not extract all the useful information from the previous layers.
Our proposed module aims to solve this misalignment by reﬁning the class tokens in a way such that the shared (pretrained) classiﬁer head is able to extract all discriminative information contained within the class tokens of each block.
Moreover, intermediate patch tokens may contain additional information that is not at all utilized by the class tokens of those blocks, which would also be addressed by our proposed block.
Therefore, we extract both patch tokens and the class token from each block and process them for reﬁnement, as explained next.
− Patch Token Reﬁnement: One of the inputs to the token reﬁnement module is the set of patch tokens output from each block.
We ﬁrst rearrange these patch tokens to regain their spatial relationships.
The aim of this component within the reﬁnement module is to extract information relevant to spatial structure contained within the intermediate patch tokens.
We believe that signiﬁcant discriminative information is contained within these patches.
The obtained rearranged patch tokens are passed through a convolution block (standard ResNet block containing a skip connection) to obtain a spatially aware feature map, which is then average pooled to obtain a single feature vector (of same dimension as the class token).
This feature vector is expected to extract all spatial information from patch tokens.
− Class Token Reﬁnement: By reﬁning the class tokens of each block, we aim to remove any misalignment between the existing class tokens and the shared (pretrained) classiﬁer head.
Also, given how the class token does not contain a spatial structure, we simply use a linear layer to reﬁne it.
We hypothesize that reﬁned class token at each block would be much more aligned with the shared classiﬁer head allowing it to extract all discriminative information contained within those tokens.
− Merging Patch and Class Token:
We obtain the reﬁned class token and the patch feature vector (reﬁned output of patch tokens) and sum them together to obtain a merged token.
While we tested multiple approaches for merging, simply summing them proved sufﬁcient.
− Training:
Given a ViT model containing k transformer blocks, we plugin k instances of our token reﬁnement module to the output of each block as illustrated in Figure 4.
We obtain the pretrained model, freeze all existing weights, and train only the k token reﬁnement modules for only a single epoch on ImageNet training set.
We used SGD optimizer with learning rate set to 0.001.
Training ﬁnishes in less than one day on a single GPU-V100 even for a large ViT model such as DeiT-B.
As expected, the trained token reﬁnement module leads to increased discriminability of the class tokens, which we illustrate in Figure 5.
Note how this leads to signiﬁcant boosting of discriminative power especially in the earlier blocks, solving the misalignment problem.
We build on this enhanced discriminability of the ensemble members towards better transferability, as explained next.
3. A DVERSARIAL T RANSFER
Our modiﬁcations to ViT models with respect to multiple discriminative pathways and token reﬁnement are exploited in relation to adversarial transfer.
We consider black-box attack perturbations that are generated using a source (surrogate) ViT model.
The source model is only pretrained on ImageNet, modiﬁed according to our proposed approach and is subsequently ﬁne-tuned to update only the token reﬁnement module for a single epoch.
We experiment with multiple white-box attacks, generating the adversarial examples using a joint loss over the outputs of each block.
The transferability of adversarial examples is tested on a range of CNN and ViT models.
Given input sample x and its label y , the adversarial object for our self-ensemble (Eq. 1) for the untargeted attack is deﬁned as,
[Table] [Equation]
[Equation]
where [[ · ]] is an indicator function.
In the case of target attack, the attacker optimizes the above objective towards a speciﬁc target class instead of an arbitrary misclassiﬁcation
[Table]
E XPERIMENTS
We conduct thorough experimentation on a range of standard attack methods to establish the performance boosts obtained through our proposed transferability approach.
We create (cid:96) ∞ adversarial attacks with (cid:15) ≤ 16 and observe their transferability by using the following protocols:
Source (white-box) models: We mainly study three vision transformers from DeiT (Touvron et al., 2020) family due to their data efﬁciency.
Speciﬁcally, the source models are Deit-T, DeiT-S, and
DeiT-B (with 5, 22, and 86 million parameters, respectively).
They are trained without CNN distillation.
Adversarial examples are created on these models using an existing white-box attack (e.g., FGSM (Goodfellow et al., 2014), PGD (Madry et al., 2018) and MIM (Dong et al., 2018)) and then transferred to the black-box target models.
Target (black-box) models: We test the black-box transferability across several vision tasks including classiﬁcation, detection and segmentation.
We consider convolutional networks including BiT-ResNet50 (BiT50) (Beyer et al., 2021), ResNet152 (Res152)
(He et al., 2016), Wide-ResNet-50- (WRN) (Zagoruyko & Komodakis, 2016), DenseNet201 (DN201) (Huang et al., 2017) and other ViT models including Token-to-Token transformer (T2T) (Yuan et al., 2021), Transformer in Transformer (TnT) (Mao et al., 2021), DINO (Caron et al., 2021), and Detection Transformer (DETR) (Carion et al., 2020) as the black-box target models.
Datasets: We use ImageNet training set to ﬁne tune our proposed token reﬁnement modules.
For evaluating robustness, we selected 5k samples from ImageNet validation set such that 5 random samples from each class that are correctly classiﬁed by ResNet50 and ViT small (ViT-S)
(Dosovitskiy et al., 2020) are present.
In addition, we conduct experiments on COCO (Lin et al., 2014)
(5k images) and PASCAL-VOC12 (Everingham al
[Table]
Evaluation Metrics: We report fooling rate (percentage of samples for which the predicted label is ﬂipped after adding adversarial perturbations) to evaluate classiﬁcation.
In the case of object detection, we report the decrease in mean average precision (mAP) and for automatic segmentation, we use the popular Jaccard Index.
Given the pixel masks for the prediction and the ground-truth, it calculates the ratio between the pixels belonging to intersection and the union of both masks.
Baseline Attacks: We show consistent improvements for single step fast gradient sign method (FGSM)
(Goodfellow et al., 2014) as well as iterative attacks including PGD (Madry et al., 2018),
MIM (Dong et al., 2018) and input diversity (transformation to the inputs) (DIM) (Xie et al., 2019) attacks.
Iterative attacks ran for 10 iterations and we set transformation probability for DIM to default 0.7 (Xie et al., 2019).
Our approach is not limited to speciﬁc attack settings, but existing attacks can simply be adopted to our self-ensemble ViTs with reﬁned tokens.
Refer to appendices A-J for extensive analysis with more ViT designs, attacks, datasets (CIFAR10 & Flowers), computational cost comparison, and latent space visualization of our reﬁned token.
4. C LASSIFICATION
In this section, we discuss the experimental results on adversarial transferability across black-box classiﬁcation models.
For a given attack method ‘Attack’ , we refer ‘Attack E ’ and ‘Attack RE ’ as self-ensemble and self-ensemble with reﬁned tokens, respectively, which are the two variants of our approach.
We observe that adversarial transferability from ViT models to CNNs is only moderate for conventional attacks (Tables 1 & 2).
For example, perturbations found via iterative attacks from
DeiT-B to Res152 has even lower transfer than VGG19 bn .
However, the same attacks when applied using our proposed ensemble strategy (Eq. 1) with reﬁned tokens consistently showed improved transferability to other convolutional as well as transformer based models.
We observe that models without inductive biases that share architecture similarities show higher transfer rate of adversarial perturbations among them (e.g., from DeiT to ViT (Dosovitskiy et al., 2020)).
We further observe that models trained with the same mechanism but lower parameters are more vulnerable to black-box attacks.
For example, ViT-S and T2T-T are more vulnerable than their larger counterparts, ViT-L and T2T-24.
Also models trained with better strategies that lead to higher generalizability are less vulnerable to black-box attacks e.g., BiT50 is more robust than ResNet152 (Tables 1 and 2).
[Table]
[Table]
[Figure]
The strength of our method is also evident by blockwise fooling rate in white-box setting (Fig. 6). It is noteworthy how MIM fails to fool the initial blocks of ViT, while our approach allows the attack to be as effective in the intermediate blocks as for the last class token.
This ultimately allows us to fully exploit ViT’s adversarial space leading to high transfer rates for adversarial perturbations. 4. C ROSS -T ASK T RANSFERABILITY
Self-attention is the core component of transformer architecture regardless of the task; classiﬁcation (Dosovitskiy et al., 2020; Touvron et al., 2020; Yuan et al., 2021; Mao et al., 2021), object detection (Carion et al., 2020), or unsupervised segmentation (Caron et al., 2021).
We explore the effectiveness of our proposed method on two additional tasks: object detection (DETR) (Carion et al., 2020) and segmentation (DINO) (Caron et al., 2021).
We select these methods considering the use of transformer modules employing the self-attention mechanism within their architectures.
While the task of object detection contains multiple labels per image and involves bounding box regression, the unsupervised model DINO is trained in a self-supervised manner with no traditional image-level labels.
Moreover, DINO uses attention maps of a ViT model to generate pixel-level segmentations, which means adversaries must disrupt the entire attention mechanism to degrade its performance.
We generate adversarial signals on source models with a classiﬁcation objective using their initial predictions as the label.
In evaluating attacks on detection and segmentation tasks at their optimal setting, the source ViT need to process images of different sizes (e.g., over 896 × 896 pix for DETR).
To cater for this, we process images in parts (refer appendix G) which allows generation of stronger adversaries.
The performance degradation of DETR and DINO on generated adversaries are summarised in Tables 3 & 4.
For DETR, we obtain clear improvements.
In the more robust DINO model, our transferability increases well with the source model capacity as compared to the baseline.
C ONCLUSION
We identify a key shortcoming in the current state-of-the-art approaches for adversarial transferability of vision transformers (ViTs) and show the potential for much strong attack mechanisms that would exploit the architectural characteristics of ViTs.
Our proposed novel approach involving multiple discriminative pathways and token reﬁnement is able to ﬁll in these gaps, achieving signiﬁcant performance boosts when applied over a range of state-of-the-art attack methods.
Reproducibility Statement : Our method simply augments the existing attack approaches and we used open source implementations.
We highlight the steps to reproduce all of the results presented in our paper, a) Attacks: We used open source implementation of patchwise attack (Gao et al., 2020) and Auto-Attack (Croce & Hein, 2020b) (refer Appendix B) with default setting.
Wherever necessary, we clearly mention attack parameters, e.g., iterations for PGD (Madry et al., 2018),
MIM (Dong et al., 2018) and DIM (Xie et al., 2019) in section 4 (Experiments: Baseline
Attack).
Similarly, transformation probability for DIM is set to the default value provided by the corresponding authors that is 0.7, b) Refined Tokens: We ﬁne tuned class tokens for the pretrained source models (used to create perturbations) using open source code base ( https: //github.com/pytorch/examples/tree/master/imagenet ).
We provided training details for ﬁne tuning in section 3.2.
Further, we will publicly release all the models with reﬁned tokens, c) Cross-Task Attack Implementation: We provided details in section 4.2 and pseudo code in appendix G for cross-task transferability (from classiﬁcation to segmentation and detection), and d) Dataset: We describe the procedure of selecting subset (5k) samples from
ImageNet val.
set in section 4.
We will also release indices of these samples to reproduce the results.
Ethics Statement : Since our work focuses on improving adversarial attacks on models, in the short run our work can assist various parties with malicious intents of disrupting real-world deployed deep learning systems dependent on ViTs.
However, irrespective of our work, the possibility of such threats emerging exists.
We believe that in the long run, works such as ours will support further research on building more robust deep learning models that can withstand the kind of attacks we propose, thus negating the short term risks.
Furthermore, a majority of the models used are pre-trained on ImageNet (ILSVRC’12).
We also conduct our evaluations using this dataset.
The version of ImageNet used contains multiple biases that portray unreasonable social stereotypes.
The data contained is mostly limited to the Western world, and encodes multiple gender / ethnicity stereotypes Yang et al.
(2020) while also posing privacy risks due to unblurred human faces.
In future, we hope to use the more recent version of ImageNet Yang et al.
(2021) which could address some of these issues
Appendix
Adversarial perturbations found with ensemble of models are shown to be more transferable (Dong et al., 2018; Naseer et al., 2021a).
In appendix A, we demonstrate the effectiveness of our selfensemble to boost adversarial transferability within ensemble of different models.
Our approach augments and enhances the transferability of the existing attack.
We further demonstrate this with recent attacks, Patchwise (Gao et al., 2020) and Auto-Attack (Croce & Hein, 2020b) in appendix B. Auto-Attack is a strong white-box attack which is a combination of PGD with novel losses and other attacks such as boundary-based, FAB (Croce & Hein, 2020a) and, query based Square attack (Andriushchenko et al., 2020).
This highlights that our method can be used as plug-and-play with existing attack methods.
We then evaluate adversarial transferability of Auto-Attack and PGD ( iterations) at (cid:15) 4, 8, and 16 in appendix C.
Our self-ensemble with reﬁned tokens approach consistently performs better with these attack settings as well.
In appendix D, we extended our approach to other dataset including CIFAR10 (Krizhevsky et al., 2009) and Flowers (Nilsback & Zisserman, 2008) datasets.
We highlight vulnerability of Swin transformer (Liu et al., 2021) against our approach in appendix E. We showed the results on full ImageNet validation set (50k samples) in appendix F.
This demonstrates the effectiveness of our method regardless of dataset or task.
We discuss the generation of adversarial samples for images of sizes that are different to the source ViT models’ input size (e.g., greater than 224) in appendix G.
We provide computational cost analysis in Appendix H and visualize the latent space of reﬁned tokens in Appendix H.1.
Finally, we extended our approach to diverse ViT designs (Wu et al., 2021;
Tolstikhin et al., 2021) and CNN in Appendices I and J.
A S ELF - ENSEMBLE WITHIN E NSEMBLE
We created an ensemble of pre-trained Deit models (Touvron et al., 2020) including Deit-T, DeitS and DeiT-B. These models have 12 blocks (layers) and differ in patch embedding size.
These models are trained in a similar fashion without distillation from CNN.
As expected, adversarial transferability improves with an ensemble of models (Table 5).
We also note that unlike such multiple-model ensemble approaches, our self-ensemble attack achieves performance improvements with minimal increase in computational complexity of the attack that is from a single ViT. Our self-ensemble extended three classiﬁers ensemble to an ensemble of 36 classiﬁers.
Such multi-model ensemble combined with our proposed self-ensemble with reﬁned tokens approach leads to clear attack improvements in terms of transferability (refer Table 5)
[Table]
[Table]
[Table]
+S+B]
[Table]
B S ELF - ENSEMBLE FOR MORE A TTACKS : P ATCHWISE AND A UTO -A TTACK
We conducted additional experiments with recent methods such as Patch-wise attack Gao et al.
and Auto-Attack Croce & Hein (2020b) to show case that our approach does not need special setup and can be used as plug-and-play with the existing attacks.
Patch-wise Gao et al.
(2020) is a recent black-box attack while Auto-Attack is a combination of PGD with novel losses and other attacks such as boundary-based, FAB (Croce & Hein, 2020a) and, query based Square attack (Andriushchenko et al., 2020).
In each case, attack transferability is increased signiﬁcantly using our approach of self-ensemble with reﬁned tokens (refer Table 6).
[Table]
C S ELF - ENSEMBLE FOR DIFFERENT (cid:15) : 4, 8,
We evaluate if adversarial transferability boost provided by our method also hold for large number of attack iterations such PGD with 100 iterations and Auto-Attack which is based on multiple random restarts, thousands of queries, etc.
We further evaluate adversarial transfer at various perturbation budgets, (cid:15) such as 4, 8, and 16.
Our self-ensemble with reﬁned tokens approach consistently improves black-box adversarial transfer under such attack setting, thus making them not only powerful in white-box but also in black-box setting.
Results are presented in Table 7.
D S ELF - ENSEMBLE FOR MORE D ATASETS : CIFAR10 AND F LOWERS
We extended our approach to CIFAR10 and Flowers datasets.
Since ViT training from scratch on small datasets is unstable Touvron et al.
; Yuan et al.
(2021), we ﬁne-tuned ViTs trained on
ImageNet on CIFAR10 and Flowers.
Transferability of adversarial attacks increases with notable gains by using our approach based on self-ensemble and reﬁned tokens (refer Tables 8, 9, 10 and 11).
[Table] [Table]
E V ULNERABILITY OF S WIN T RANSFORMER
We show how the black-box vulnerability of ViT architecture without explicit class token (Swin transformer (Liu et al., 2021)) increases against our approach (refer Table 12).
F F ULL I MAGE N ET VALIDATION S ET
In line with most existing attack methods, we use subset of ImageNet (5k samples, 5 samples from each class) for all our experiments.
Our subset from ImageNet is balanced as we sample from each class.
We will release the image-indices of ours subset publicly to reproduce the results.
Additionally, we re-run our best performing attacks (MIM and DIM) on the whole ImageNet val.
set (50k samples) to validate the merits of our approach ( refer Table 13).
Our approach shows considerable improvements in attack transferability regardless the dataset size.
[Table]
[Table] [Table]
[Table]
G C ROSS -T ASK T RANSFERABILITY WITH L ARGER I MAGES (>224)
We generate adversarial signals on source models with a classiﬁcation objective using their initial predictions as the label (Algorithm 1).
In the case of ViT models trained on 224x224 images, we rescale all larger images to a multiple of 224x224, split them into smaller 224x224 image portions, obtain the prediction of the source (surrogate) model for each smaller image portion, and apply the attack separately for each image portion using the prediction as the label for the adversarial objective function (Algorithm 2).
For example, in the case of DETR, we use an image size of 896 x 896, split
[Table] [Table]
it into 16 portions of size 224x224, and use these to individually generate adversarial signals which we later combine to build a joint 896x896 sized adversary.
0 for samples, _ in (dataloader):orig_shape =
samples.shape #
with torch.no_grad():clean_out = src_model(samples)label = clean out argmax(dim=-1).detach() adv = generate_adversary(src_model, samples,
[Table] [Table]
We run a generic attack on crosstasks where a single image level label is not available.
The source model is used to generate a label (its original prediction) which is used as the target for the whitebox attack.
The generated adversarial signal is then applied onto the target task.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure]
[Table]
We run a modiﬁed version of the attack for cross-task experiments requiring speciﬁc input sizes not compatible with the
ViT models’ input size.
We split larger images into smaller image portions, generate adversarial signals separately for each portion, and combine these to obtain a joint adversarial signal relevant to the entire image.
For each image portion, we use the source model to generate a label (its prediction for that image portion), and use it as the target when generating the adversarial signal.
H C OMPUTATION C OST AND I NFERENCE S PEED A NALYSIS
In this section, we analyze the parameter complexity and computational cost of our proposed approach.
The basic version of our self-ensemble (Attack E ) only uses an off the shelf pretrained ViT without adding any additional parameters.
However, we introduce additional learnable parameters with token reﬁnement module to solve misalignment problem between the tokens produced by an intermediate
[Table] [Table]
block and the ﬁnal classiﬁer of a vision transformer (Sec. 3.2).
Our reﬁnement module processes class and patch tokens using MLP (linear layer) and Convolutional block (Fig.
8) and its parameter complexity is dependent on the embedding dimension of these pretrained tokens.
For example,
DeiT-S produces tokens with embedding dimension of size R 384 and our reﬁnement module adds 1.47 million parameters to its sub-model within the self-ensemble trained with reﬁned tokens.
Fine tuning with additional parameters provides notable gains in recognition accuracy.
Our approach increases top-1 (%) accuracy (averaged across self-ensemble) by 12.43, 15.21, and 16.70 for Deit-T,
Deit-S, and DeiT-B, respectively.
Similar trend holds for the convolutional vision transformer (Wu et al., 2021) and
MLP-Mixer (Tolstikhin et al., 2021) as well (Fig. 10).
Further, we analyze inference speed for different attacks in Table 14.
The computational cost difference between the attack on a conventional model and the attack on our self-ensemble (without reﬁnement module) is only marginal.
As expected, attacking self-ensemble with reﬁnement module is slightly more expensive, however, the notable difference is only with very large models such as
DeiT-B.
In fact, the increase in computational cost is a function of the original complexity of the pre-trained ViT model e.g., DeiT-B with an original parametric complexity of 86 million generates high-dimensional tokens R 784 , which leads to higher compute cost in our reﬁnement module.
H. L ATENT S PACE OF R EFINED T OKENS
We visualize the latent space of our reﬁned tokens in Fig.
We randomly selected 10 classes from
ImageNet validation set distributed across the entire dataset.
We extracted class tokens with and without reﬁnement from the intermediate blocks (5,6,7,8) of Deit-T, Deit-S, and Deit-B. Our reﬁned tokens have lower intra-class variations i.e., feature representations of samples from the same class are clustered together.
Furthermore, the reﬁned tokens have better inter-class separation than the original tokens.
This indicates that reﬁnement minimizes the misalignment between the ﬁnal classiﬁer and intermediate class tokens, which leads to more disentangled representations.
Attacking such disentangled representations across the self-ensemble allows us to ﬁnd better adversarial direction that leads to more powerful attacks.
I S ELF - ENSEMBLE FOR H YBRID V I T AND MLP-M IXER
In this section, we apply our proposed approach to a diverse set of ViT models including a hybrid vision transformer (convolutional vision transformer (CvT) (Wu et al., 2021)) as well as a MLP-Mixer (Tolstikhin et al., 2021).
CvT design incorporates convolutional properties into the ViT architecture.
We apply our approach on two CvT models, CvT-Tiny (CvT-T) and CvT-Small (CvT-S).
We create 10 and 16 models within self-ensemble of CvT-T and CvT-S, respectively.
On the other hand,
MLP-Mixer thrives on a simplistic architecture design.
It does not have self-attention layers or class token.
We use an average of patch tokens as class token in this case and create 12 models within self-ensemble of MLP-Mixer-Base (Mixer-B).
We train reﬁnement module for each of these models using the same setup as described in Sec. 3.2.
The reﬁned class tokens show clear improvements in top-1 (%) accuracy on ImageNet validation set (Fig. 10).
Similarly attacking self-ensemble with reﬁned tokens has non-trivial boost in attack transferability (Table 15).
The successful application of our approach to such diverse ViT designs highlights the generality of our method.
[Table]
J C AN S ELF - ENSEMBLE IMPROVE T RANSFERABILITY FROM CNN?
Our proposed idea of the self-ensemble can be applied to a CNN model (e.g., ResNet50) with an average pooling operation over the intermediate layer outputs.
However, due to the varying channel dimension of feature maps across a pretrained CNN , building an ensemble with a shared classiﬁer is not as straight-forward as in ViT (where equidimensional class token is available at each level).
For example, ResNet50 has four intermediate blocks that produce feature maps of sizes R 256 × 56 × 56 , R 512 × 28 × 28 , R 1024 × 14 × 14 , and R 2048 × 7 × 7 , respectively.
Then, the ﬁnal classiﬁer processes the average pooled features from the last block.
Since there is a mismatch between feature dimensions of the intermediate layers and the ﬁnal classiﬁer, therefore applying the basic version of our self-ensemble approach (Attack E ) is not possible for the pretrained ResNet.
An intermediate layer is essential to project intermediate feature vectors to the same dimension as the ﬁnal feature vector in the case of CNNs.
Therefore, for the reﬁned embedding variant of our approach, we ﬁnetune ResNet50 features to create self-ensemble with the procedure described in Sec. 3.2.
We report the relative improvements of self-ensemble with reﬁned features w.r.t the original (single) model and compare ResNet50 with Deit-S (Fig. 11).
Both these off-the-shelf models,
ResNet50 (25 million parameters) and Deit-S (22 million parameters), are comparable in terms of their computational complexity.
We observe that feature reﬁnement also helps to boost adversarial transferability from the ResNet50 model.
However, adversarial transferability improvement of our self-ensemble with token reﬁnement on ViTs is considerably better than for the case of ResNet50.
These results suggest that the proposed approach is well-suited for improving adversarial transferability of ViT models.