Understanding and Improving Robustness of Vision Transformers through patch-based Negative Augmentation
Abstract
We investigate the robustness of vision transformers (ViTs) through the lens of their special patch-based architectural structure, i.e., they process an image as a sequence of image patches.
We ﬁnd that ViTs are surprisingly insensitive to patch-based transformations, even when the transformation largely destroys the original semantics and makes the image unrecognizable by humans.
This indicates that ViTs heavily use features that survived such transformations but are generally not indicative of the semantic class to humans.
Further investigations show that these features are useful but non-robust , as ViTs trained on them can achieve high in-distribution accuracy, but break down under distribution shifts.
Based on this understanding, we use the images transformed with our patch-based operations as negatively augmented views and offer losses to regularize the training away from using non-robust features.
This is a complementary view to existing research that mostly focuses on augmenting inputs with semantic-preserving transformations to enforce models’ invariance.
We show that patch-based negative augmentation consistently improves robustness of ViTs across a wide set of ImageNet based robustness benchmarks.
Furthermore, we ﬁnd our patch-based negative augmentation are complementary to traditional (positive) data augmentation, and together boost the performance further 1 .
Introduction
Building vision models that are robust, i e., that are highly accurate even on unexpected and out-ofdistribution images, is increasingly a requirement to trusting vision models and a strong benchmark for progress in the ﬁeld.
Recently, Vision Transformers (ViTs, Dosovitskiy et al.
(2021)) achieved a great sucess in image classiﬁcation.
With the new promise of vision transformers, it is critical to understand their properties and in particular their robustness.
Recent early studies (Naseer et al., 2021; Paul & Chen, 2021; Bhojanapalli et al., 2021) have found ViTs be more robust than ConvNets in some scenarios with the hypothesis that the non-local attention based interactions enabled ViTs to capture more global and semantic features.
In contrast, we add to this line of research showing a different side of the challenge: we ﬁnd ViTs are still vulnerable to relying on non-robust features impeding out-of-distribution performance.
In this paper, we ﬁrst demonstrate ViTs rely on speciﬁc non-robust features and then show how to reduce the reliance on these non-robust features, enabling improved out-of-distribution performance.
Visualization of patch-based transformations.
On the top of each image, we display the predicted conﬁdence score of ViT-B/16 pretrained on ImageNet-21k and ﬁnetuned on ImageNet-1k.
To understand the robustness properties of ViTs, we start with the architectural traits of ViTs –
ViTs operate on non-overlapping image patches and allow long range interaction between patches even in lower layers.
It is hypothesized in recent studies (Naseer et al., 2021;
Paul & Chen, 2021; Bhojanapalli et al., 2021) that the non-local attention based interactions contribute to better robustness of ViTs than ConvNets.
To study the ability of ViTs to integrate global semantics structures across patches, we design and apply patch-based image transformations, such as random patch rotation, shufﬂing, and background-inﬁlling (Figure 1).
Those transformations destroy the spatial relationship between patches and corrupted the global semantics, and the resultant images are often visually unrecognizable.
However, we ﬁnd that ViTs are surprisingly insensitive to these transformations and can make highly accurate predictions on these transformed images.
This suggests that ViTs use features that survive such transformations but are generally not indicative of the semantic class to humans.
Going one step further, we ﬁnd that those features are useful but not robust, as ViTs trained on them achieved high in-distribution accuracy, but suffered signiﬁcantly on robustness benchmarks.
With this understanding of ViTs’ reliance on non-robust features captured by patch-based transformations, we still must answer: how can we train ViTs to improve out-of-distribution performance and not sacriﬁce in-distribution accuracy?
A majority of past robust training algorithms encourage the smoothness of model predictions on augmented images with semantic preserving transformations (Hendrycks et al., 2020b; Cubuk et al., 2019).
However, the patch-based transformations deliberately destroy the semantic meaning and only leave non-robust features.
Taking inspiration from recent research on generative modeling (Sinha et al., 2020), we propose a family of robust training algorithms based on patch-based negative augmentations that regularize the training from relying on non-robust features surviving patch-based transformations.
Through extensive evaluation on a wide set of ImageNet-based benchmarks, we ﬁnd that our methods consistently improve the robustness of the trained ViTs.
Furthermore, our patch-based negative augmentation can be combined with the traditional (positive) data augmentation to boost the performance further.
Understanding Robustness of Vision Transformers
To investigate if ViT has successfully taken advantage of the long range interactions between patches, we design a series of patch-based transformations which signiﬁcantly destroys the global structure of images.
The patch-based transformations (see Fig. 1) are:
• Patch-based Shufﬂe (P-Shufﬂe) : we randomly shufﬂe the input image patches to change their positions 2 .
• Patch-based Rotate (P-Rotate) : we randomly select a rotation degree from the set Ω = { 0 ◦ , 90 ◦ , 180 ◦ , 270 ◦ } and rotate each image patch independently. • Patch-based Inﬁll (P-Inﬁll) : we replace the image patches in the center region of an image with the patches on the image boundary 3 .
Each patch-based transformation is performed to a single image.
We make sure the patch size of our patch-based transformation is a multiple of the input image patch of ViT so that the content within each patch is well-maintained.
For P-inﬁll, we use “replace rate” to denote the ratio of replaced patches in the center over the total number of patches in an image.
Examples of transformed images are shown in Fig.
1 (see Appendix E for more examples).
In most cases, it is challenging to recognize the semantic classes after those transformations.
Do ViTs rely on features not indicative of the semantic classes to humans?
To validate if ViTs behave similarly as humans on these patch-based transformed images, we evaluate ViT models (Dosovitskiy et al., 2021) on these patch-based transformed image.
Speciﬁcally, we apply each patch-based transformation to ImageNet-1k validation set and report the test accuracy of each ViT on the transformed images.
The test accuracy is computed by using the semantic class of the corresponding original image as the ground-truth.
As shown in Figure 2, the accuracy achieved by ViTs are signiﬁcantly higher than random guessing (0.1 % ).
In addition, as shown in Figure 1, ViT gives these patch-based transformed images a very high-conﬁdent prediction even when the transformation largely destroys the semantics and make the image unrecognizable by humans.
Do features preserved in patch-based transformations impede robustness?
Taking one step further, we want to know if the features preserved by simple patch-based transformations result in robustness issues.
To this end, we train a vision transformer, e.g., ViT-B/16 4 , on patch-based transformed images with original semantic class assigned as their ground-truth.
Note that all the training images are patch-based transformed images.
In this way, we force the model to fully exploit the features preserved in patch-based transformations.
Then, we test the model on ImageNet-1k validation set and three robustness benchmarks, ImageNet-A (Hendrycks et al., 2019), ImageNet-C (Hendrycks &
Dietterich, 2019), ImageNet-R (Hendrycks et al., 2020a) without any transformation .
When we compare the accuracy between the baseline model and models trained on patch-based transformations (i.e., the difference between the blue bar and one of the red/green/orange bars in
Figure 3), we ﬁnd that ViTs’ in-distribution accuracy drops only slightly, but the robustness drop is signiﬁcant when models are trained on these patch-based transformations.
This strongly suggests that the features preserved in patch-based transformations are useful for high-accurate in-distribution prediction but are non-robust under distributional shifts.
Improving the Robustness of Vision Transformers
Based on the key observations that the patch-based transformations encode features that contribute to the non-robustness of ViTs, we propose a negative augmentation procedure to regularize ViTs from relying on such features.
Speciﬁcally, given a clean image x , we generate its negative view, denoted as ˜ x , by applying a patch-based transformation to x .
We call it negative augmentation, in contrast with the standard (positive) augmentation that are semantic preserving.
Let L ce
( B ; θ ) represent the cross-entropy loss function used to train a vision transformer with parameters θ , where
B is a minibatch of clean examples.
The loss on negative views L neg ( B , ˜ B ; θ ) can be easily added to the cross-entropy loss L ce ( B ; θ ) via L
[Equation]
where λ is a coefﬁcient balancing the importance between clean data as well as negative augmentation.
We introduce three different losses on negative views to leverage patch-based negative augmentation through label, logit and representation space respectively.
Label space: uniform loss Many existing data augmentation techniques (Cubuk et al., 2019, 2020;
Hendrycks et al., 2020b) use one-hot labels for semantic-preserving augmented data to enforce the invariance of the model prediction.
In contrast, the semantic classes of our generated patch-based negative augmented data are visually unrecognizable, as shown in Figure 1.
Therefore, we propose to use uniform labels instead for those negative augmentations.
Speciﬁcally, the loss function on negative views that we optimize at each training step can be formulated as:
[Equation]
where ˜ y denotes the uniform distribution:
˜
y k = 1 K where K is the total number of classes.
f ( x ; θ ) denotes the function mapping the input image into the logit space.
Logit space: (cid:96) 2 Loss
An alternative to pre-assuming labels for negative augmentation is to add the constraints on the logit space (or the space of predicted probability).
Inspired by existing work (Kannan et al., 2018; Zhang et al., 2019; Hendrycks et al., 2020b) which provides an extra regularization term encouraging similar logits between clean and “positive” augmented counterparts, we instead encourage the logits of clean examples and their corresponding negative augmentations to be far away.
In this way, we prevent the model from relying on the non-robust features preserved in negative views.
Speciﬁcally, we maximize the (cid:96) 2 distance between the predicted probability of clean examples and their corresponding negative views.
The loss on negative views, therefore, can be formulated as:
[Equation]
Here the (cid:96) 2 distance is computed over the predicted probability rather than the logits f ( x ; θ ) because empirically we observe that maximizing the difference of logits can cause numerical instability.
Representation space: contrastive loss
Lastly, we propose to use a contrastive loss (Oord et al., 2018;
Chen et al., 2020a; Khosla et al., 2020) to regularize the training away from using non-robust features.
For an example x i ∈ B , we create a positive set P i ≡ { x j ∈ B \{ x i }| y j = y i } with all the examples in the minibatch B sharing the same class as x i .
The anchor x i is excluded from its positive set P i .
Next, we can generate the negative set composed of two types of negative examples: 1) all the examples in the minibatch B with a different class as x i , 2) the patch-based negatively transformed images ˜ x ∈ ˜ B .
For each anchor x i , we can in total have 2 | B | − | P i | − 1 negative pairs, where | B | is the batch size and | P i | is the cardinality of the positive set P i .
Let the candidate set
Q i ≡ ˜ B ∪ B \{ x i } , the loss function can be expressed as:
[Equation]
where τ is the temperature and sim ( x i , x j )
=
[Equation]
computes the cosine similarity between g ( x i ; θ ) and g ( x j ; θ ) , and g ( x ; θ ) denotes the representation learned by the penultimate [Table] layer of the classiﬁer.
We do not use a learnable projection head 5 as in contrastive representation learning (Chen et al., 2020a,b).
Therefore, no extra network parameters are used for our proposed method and the improvement of robustness can be mainly attributed to patch-based negative augmentations.
When the batch size is larger than the number of classes (which is the case for our ImageNet1K experiments) [Table] it is easy to ﬁnd positive examples from the same class in a mini-batch, so we only extend candidate set Q i with our proposed patch-based negative data augmentations.
When the batch size is far smaller than the number of classes (which is the case for our ImageNet-21K experiments), it can be difﬁcult to ﬁnd two examples from the same classes.
Similar to (Chen et al., 2020a; Khosla et al., 2020), we generate another “positive” view for each image using common data augmentation (e.g., random cropping) so that we can make sure there is at least one positive pair from the same class.
Denoting the set of positively augmented data as B + , the modiﬁed positive set is P i ≡ { x j ∈ B + | y j = y i } , the candidate set Q i is now ˜ B ∪ B + instead of ˜ B ∪ B \{ x i } .
Experimental setup We follow Dosovitskiy et al.
(2021) to ﬁrst pre-train all the models with image size 224 × 224 and then ﬁne-tune the models with a higher resolution 384 × 384 (see Appendix A for details). The top-1 accuracy of the ﬁne-tuned models are reported on ImageNet-1k validation set as well as three robustness benchmarks, ImageNet-A (Hendrycks et al., 2019), ImageNet-C (Hendrycks & Dietterich, 2019) and ImageNet-R (Hendrycks et al., 2020a). For ImageNet-C, the reported accuracy is averaged over 19 corruptions types and 5 different corruption severities.
Complementary to traditional (“positive”) data augmentation To investigate if our proposed patch-based negative augmentation is complementary to “positive” data augmentation, we apply our patch-based negative transformation on top of Rand-Augment (Cubuk et al., 2020) and AugMix (Hendrycks et al., 2020b).
As in Table 1, when patch-based negative augmentations are applied to either Rand-Augment or AugMix, we can consistently improve the robustness of vision transformers across all three robustness benchmarks (please refer to Table 5 in Appendix for a full table with three losses for each patch-based transformation).
This is particularly noteworthy as both Rand-Agument and AugMix are already designed to signiﬁcantly improve the robustness of vision models.
Yet, we see that patch-based negative augmentation provides further robustness beneﬁts.
This suggests that robustness of vision models was not adequately addressed by “positive” data augmentation and that patch-based negative augmentation is complementary to traditional approaches.
Conclusion
Through this research we have found concrete examples of ViTs relying on non-robust features for predictions and shown that this reliance is limiting robustness and out-of-distribution performance.
We believe this opens multiple exciting new lines of research.
First, we believe that the methodological approach developed here is a valuable recipe for further progress.
Through ﬁnding patch-wise, semantic-destroying transformations that ViTs are insensitive to we can identify when models rely on non-robust features, and through incorporating them as negative augmentations during training we can meaningfully reduce reliance on such features.
Second, we believe this shows the potential for further improving the robustness of ViTs.
Through training the model to use such non-robust features less, we have seen we can signiﬁcantly improve the out-of-distribution performance of ViTs, without harming in-distribution accuracy!
A Training Details
We follow (Dosovitskiy et al., 2021) to train each model using Adam (Kingma & Ba, 2015) optimizer with β 1 = 0.9, β 2 = 0.999 for pre-training and SGD with momentum for ﬁne-tuning.
The batch size is set to be 4096 for pre-training and 512 for ﬁne-tuning.
All models are trained withepochs on ImageNet-1k and 90 epochs on ImageNet-21k in the pre-training stage.
In the ﬁne-tuning stage, all models are trained with 20k steps except the models pretrained from ImageNet-1k without
Rand-Augment (Cubuk et al., 2020) or Augmix (Hendrycks et al., 2020b), which we train them with 8k steps.
The learning rate warm-up is set to be 10k steps.
Dropout is used for both pre-training and ﬁne-tuning with dropout rate
[Table]
If the training dataset is ImageNet-1k, we additionally apply gradient clipping at global norm
[Table]
[Table]
[Table]
B Hyper-parameters in Patch-based Negative Augmentation
For the temperature τ used in contrastive loss [Table] we consistently observe that τ = 0 .
5 works better in pre-training stage and τ = 0 .
1 works better in ﬁne-tuning stage.
Therefore, we keep this setting for all the models in our paper.
Since we sweep the coefﬁcient λ in Eqn. 1 from the set { 0 . 5 , 1 . 0 , 1 . 5 } [Table] we observe that for most of the cases [Table]
λ
= 1 .
5 works the best.
In total we have 48 models using loss regularization on negative views in Table [Table] Table [Table]
Table 6 and Table 5. We use λ = 1 . 5 for all of them except those listed in Table [Table] where either λ = 0 .
5 or λ = 1 . 0 works better.
Actually, we ﬁnd our proposed negative augmentation is relatively robust to λ .
Therefore, we suggest using λ = 1 . 5 if readers do not want to sweep for the best value for this hyperparameter.
In Table.
4, we display the hyperparameters in each patch-based transformation that we use for the reported results in this work.
Our algorithms are generally insensitive to these parameters, and we use the same hyperparameter for all the settings investigated in this work.
[Table]
Top-1 accuracy on ImageNet-1k (IN), ImageNet-A (IN-A), ImageNet-C (IN-C) and ImageNet-R (IN-R) of ViT-B/16 pretrained and ﬁne-tuned on ImageNet-1k.
Our proposed patch-based negative augmentation are applied to either Rand-Augment (Cubuk et al., 2020) or AugMix (Hendrycks et al., 2020b).
We display the accuracy of ﬁve different corruption severities on ImageNet-C (IN-C).
[Table]
+ P
[Table]
+ P-Shufﬂe / L
[Table]
22.6 +
[Table] [Table]
/ L 79.
23.8 +
[Table]
-Inﬁll /
[Table]
+
[Table]
Shufﬂe / Contrastive 79.
24.7 +
[Table] [Table]
/ Contrastive 79.
25.4 + P-Inﬁll /
[Table]
AugMix (Hendrycks et al., 2020b) 78.
+ P-Shufﬂe / Uniform 79.
[Table] [Table]
/ Uniform 79.
25.7 + P-Inﬁll / Uniform 79.
+ P-Shufﬂe / L 78.
25.7 +
[Table] [Table]
[Table]
26.0 + P-Inﬁll / L 79.
[Table]
C Related Work
Vision transformers (Dosovitskiy et al., 2021; Touvron et al., 2021) are a family of Transformer models (Vaswani et al., 2017) that directly process visual tokens constructed from image patch embedding.
Unlike convolutional neural networks (LeCun et al., 1989; Krizhevsky et al., 2012; He et al., 2016) that assume locality and translation invariance in their architectures, vision transformers have no such assumptions and are able to exchange information globally, thus having less inductive bias about the input image data.
The signiﬁcant difference in architectures raises questions about their robustness properties.
A few recent studies ﬁnd pretrained vision transformers are at least as robust as the ResNet counterparts (Bhojanapalli et al., 2021), and possibly more robust (Naseer et al., 2021; Paul & Chen, 2021).
Our work studies a speciﬁc aspect of robustness pertaining patch-based visual tokens in ViT, and show it may lead to a generalization gap.
Different from (Naseer et al., 2021) which also shows ViTs are insensitive to patch operations such as shufﬂe and occlusion, we further propose a mitigation strategy to increase robustness of patch-based architectures.
Data augmentation is widely used in computer vision models to improve model performance (Howard, 2013; Szegedy et al., 2015; Cubuk et al., 2020, 2019; Qin et al., 2021).
It has been shown that data
[Table] [Table]
[Table]
augmentation beneﬁts vision transformers more than convolutional networks for relatively small scaled datasets (Touvron et al., 2021).
However, most of the existing data augmentations are “positive” in the sense they assume the class semantic being preserved after the transformation. In this work
[Table] we explore “negative” data augmentation operations based on patches [Table] where we encourage the representations of transformed example to be different from the original ones.
Most related to our work in this direction is the work of Sinha et al.
Although the concept of negative augmentation was proposed in their work [Table] they only apply it for generative and unsupervised modeling.
In contrast, our work focuses on discriminative and supervised modeling [Table] and demonstrate how such negative examples can reveal speciﬁc robustness issues and such augmentation approaches can directly mitigate them, offering robustness improvements under large-scale pretraining settings.
Our work is also related to contrastive learning (Wu et al., 2018;
Hjelm et al., 2019; Oord et al., 2018;
He et al., 2020; Tian et al., 2020).
The increasing number of negative pairs has shown to be important for representation learning in self-supervised contrastive learning (Chen et al., 2020a), where different images serve as negative examples for each other, and supervised contrastive learning (Khosla et al., 2020), where images with different classes are used as negative examples.
Unlike the traditional setting of representation learning, our proposed contrastive loss serves as a regularization term with patch-based negative augmentations as extra negative data points.
D Experiments D.1 Effective in improving robustness
First, we apply our proposed patch-based transformations to a ViT-B/16 model pre-trained and ﬁnetuned on ImageNet-1k.
The extra loss regularization on negative views is used in both pre-training and ﬁne-tuning stages to prevent the model from learning non-robust features preserved in patchbased transformations.
We use “Transformation / Regularization” to denote a pair of patch-based negative augmentation and loss regularization.
For examples, “P-Rotate / Uniform” means that we use P-Rotate to generate the negative views and use uniform loss to regularize the training.
We display the results in Table 8, where we can clearly see that our proposed patch-based negative augmentation effectively improves the in-distribution test accuracy and the out-of-distribution robustness across all
ImageNet-based benchmarks.
We observe that all three loss regularizations effectively leverage the negative views to regularize the training away from using non-robust features, while the contrastive loss works the best.
D.2 Robustness improvements even under larger pre-training datasets
Considering that larger training data can signiﬁcantly improve models’ robustness and achieve stateof-the-art performance, we further investigate if our proposed method can scale up to larger datasets and continues to be necessary and valuable.
To this end, we test if our proposed patch-based negative augmentation still helps robustness even when models are pre-trained on ImageNet-21k (10x larger than ImageNet-1k).
Since we follow (Dosovitskiy et al., 2021) and use a batch size of 4096 in the pretraining stage, which is much less than the 21K classes in ImageNet-21k, it is unlikely to have multiple images of the same class in a mini-batch during pretraining.
As mentioned above, we address this issue by augmenting each image one more time, generating another positive view of the same image, to make sure there is at least two examples from the same class in the mini-batch.
We use P-Shufﬂe as an example to generate negative views and display the results in Table 6 with negative augmentation in both pre-training and ﬁne-tuning stages.
We can clearly see that even when
[Table] [Table]
we greatly increase the size of pre-training dataset (i
[Table]
e., ImageNet-21k is 10x larger than ImageNet1k), our proposed patch-based negative augmentation can still further improve the robustness of ViT.
This demonstrates that our approach is valuable at scale and improves models’ robustness from an angle orthogonal to larger training data
[Table]
D.3 Pre-training vs. Fine-tuning
We further disentangle the effect of patch-based negative data augmentation in pre-training and ﬁne-tuning.
Take P-Shufﬂe as an example, we design experiments to apply negative augmentation 1) only at the ﬁne-tuning stage, 2) only at the pre-training stage, and 3) at both stages.
As shown in
Table 7, compared to the baselines, patch-based negative augmentation can effectively help improve robustness in both stages [Table] and its effect in pre-training is slightly larger than in ﬁne-tuning.
Finally, we found using negative augmentation in both stages during training yields the largest gain.
D.4 When to use Positive data augmentation
As Steiner et al.
(2021) observed that traditional (positive) augmentation can slightly hurt the accuracy of ViT if applied to ﬁne-tuning stage, we compare the accuracy of a ViT-B/16 when positive augmentation (e.g., Rand-Augment (Cubuk et al., 2020)) is only applied to pre-training stage as well as both stages.
As shown in Table 9, ﬁne-tuning without Rand-Augment achieves slightly better performance.
In addition, we also provide the results in Table 10 where we apply positive data augmentation in both stages, our proposed negative augmentation are still complementary to positive ones.
[Table]
[Table]
D.5 Effect of negative augmentation in contrastive loss
Since we consistently observe that contrastive loss regularization works the best across all the settings that we have studied, we want to further investigate the effect of our proposed negative augmentation in contrastive loss.
To this end, we design a stronger baseline by “only” excluding the patch-based negative augmentation in the negative set.
Speciﬁcally, we replace Q ≡ ˜ B ∪ B \{ x i } in Eqn.
4 with Q ≡ B \{ x i } .
We denote this stronger baseline as “Contrastive*” and display the comparison in
Table 11.
We can see that even if we add the patch-based negative augmentation on top of this stronger contrastive baseline, we can still achieve extra improvement across robustness benchmarks.
This further supports the effectiveness of our proposed patch-based negative augmentation in improving models’ robustness.
D.6 Understanding the effects of patch-based Negative Augmentation
Does ViT become more robust w.r.t.
transformed images?
We further evaluate ViTs trained with our robust training algorithms on the patch-based transformed images.
We found all three losses on negative views can successfully reduce the prediction accuracy of ViTs to be close to random guess (0.1 % ) with the original semantic classes as the ground-truth.
In other words, our robust training algorithms make ViTs behave similarly as humans on those patch-based transformed images.
Are texture biases contributing to non-robust features?
Geirhos et al.
(2018) observed that unlike humans, CNNs rely on more local information (e.g., texture) rather than more global information (e.g., shape) to make a classiﬁcation.
Since our patch-based transformations largely destroy the global structure (e.g., shape), we want to investigate if the non-robust features surviving patch-based transformation overlap with local texture biases.
To this end, we evaluate ViT-B/16 trained on patchbased transformations on Conﬂict Stimuli benchmark (Geirhos et al., 2018), and we see that ViTs trained only on patch-based transformation have a 4.9pp to 31.1pp increase on texture bias (Figure 4).
This suggests that the useful but non-robust features preserved in patch-based transformation are indeed overlapped with the local texture bias.
In addition, using our patch-negative augmentation can also to some extent reduce models’ reliance on local texture bias, e.g., we decrease the texture accuracy from 71.7 % to 62.2 % for ViT-B/16 (Table 12).
[Table]
D.7 Ablation Study
Sensitivity analysis We test the sensitivity of our patch-based negative augmentation to various patch sizes in P-Shufﬂe and P-Rotate, and different replace rates in P-Inﬁll.
We ﬁnd that P-Shufﬂe and
P-Rotate are insensitive to patch sizes from { 16 , 32 , 48 , 64 , 96 } for ViT-B/16, and P-Inﬁll is robust to replace rates ranging from 1/3 to 1/2.
The accuracy difference is smaller than 0.5 % on ImageNet-1k as well as ImageNet-A and ImageNet-R.
Therefore, we use the same parameter for all the settings investigated in this work (see Table 4 and Appendix B for details).
Double batch-size of baselines As we use the negative augmented view per example, the effective batch size is doubled compared to the vanilla ViT-B/16 trained with only cross-entropy loss.
Therefore, we further investigate if the robustness improvement is a result from a larger batch size.
When we increase the batch size from 4096 to 8192 in pre-training while keeping the same 300 training
[Table]
[Table]
E Visualization of Patch-based Transformations
We display more examples with patch-based transformations without cherry-picking in Figure 5,
Figure 6 and Figure 7.