Evaluating Model Robustness to Patch Perturbations
Abstract
Recent advances in Vision Transformer (ViT) have demonstrated its impressive performance in image classification, which makes it a promising alternative to Convolutional Neural Network (CNN).
Unlike CNNs, ViT represents an input image as a sequence of image patches.
The patchbased input image representation makes the following question interesting: How does ViT perform when individual input image patches are perturbed with natural corruptions or adversarial perturbations, compared to CNNs?
In this submission, we propose to evaluate model robustness to patch-wise perturbations.
Two types of patch perturbations are considered to model robustness.
One is natural corruptions, which is to test models’ robustness under distributional shifts.
The other is adversarial perturbations, which are created by an adversary to specifically fool a model to make a wrong prediction.
The experimental results on the popular CNNs and ViTs are surprising.
We find that ViTs are more robust to naturally corrupted patches than CNNs, whereas they are more vulnerable to adversarial patches.
Given the architectural traits of state-of-the-art ViTs and the interesting results above, we propose to add the robustness to natural patch corruption and adversarial patch attack into the robustness benchmark.
1. Motivation
Recently, Vision Transformer (ViT) has demonstrated impressive performance (Dosovitskiy et al., 2020; Touvron et al., 2021;
Wu et al., 2020; Xiao et al., 2021; Graham et al., 2021; Chen et al., 2021b; Han et al., 2021; Chen et al., 2021a; Liu et al., 2021), which makes it become a potential alternative to convolutional neural networks (CNNs).
Meanwhile, the robustness of ViT has also received great attention (Bhojanapalli et al., 2021;
Joshi et al., 2021;
Salman [Table] [Table] et al., 2021;
Shao et al., 2021; Shi & Han, 2021; Tang et al., 2021).
On the one hand, it is important to improve its robustness for safe deployment in the real world.
On the other hand, diagnosing the vulnerability of ViT can also give us a deeper understanding of its underlying working mechanisms.
Existing works have intensively studied the robustness of ViT and CNNs when the whole input image is perturbed with natural corruptions or adversarial perturbations (Bhojanapalli et al., 2021;
Shao et al., 2021; Mahmood et al., 2021; Bai et al., 2021; Aldahdooh et al., 2021).
Unlike CNNs, ViT processes the input image as a sequence of image patches.
In this work, instead, we propose to study the robustness of ViT to patch-wise perturbations based on its special patch-based architecture.
In this work, two typical types of perturbations are considered to compare the robustness between ViTs and CNN (e.g., ResNets (He et al., 2016)).
One is natural corruptions (Hendrycks & Dietterich, 2019), which is to test models’ robustness under distributional shift.
The other is adversarial perturbations (Szegedy et al., 2014; Goodfellow et al., 2014), which are created by an adversary to specifically fool a model to make a wrong prediction.
Surprisingly, we find
ViT does not always perform more robustly than ResNet.
When individual image patches are naturally corrupted, ViT is more robust compared to ResNet.
However, when input image patch(s) are adversarially attacked, ViT shows a higher vulnerability than ResNet.
To better understand the model robustness to patch perturbation, we revealed that ViT’s stronger robustness to natural corrupted patches and higher vulnerability against adversarial patches are both caused by the attention mechanism.
Specifically, the self-attention mechanism of ViT can effectively ignore the natural patch corruption, while it’s also easy to manipulate the self-attention mechanism to focus on an adversarial patch.
This is well supported by rollout attention visualization (Abnar & Zuidema, 2020) on ViT.
As shown in Fig.
1 (a), ViT successfully attends to the classrelevant features on the clean image, i.e. , the head of the dog.
When one or more patches are perturbed with natural corruptions, shown in Fig.
1 (b), ViT can effectively ignore the corrupted patches and still focus on the main foreground to make a correct prediction.
In Fig.
1 (b), the attention weights on the positions of naturally corrupted patches are much smaller even when the patches appear on the fore-
[Figure]
ground.
In contrast, when the patches are perturbed with adversarial perturbations by an adversary, ViT is successfully fooled to make a wrong prediction, as shown in Fig.(c).
This is because the attention of ViT is misled to focus on the adversarial patch instead.
In summary, given the architectural traits of state-of-the-art
ViTs and the interesting results we found, we propose to add the robustness to corrupted patches and adversarial patches into the robustness benchmark.
2. Data Generation
Natural Corrupted Patches.
For each image, we first select n image patch x i from the input image and perturb them with natural corruptions.
As in (Hendrycks & Dietterich, 2019), 15 types of natural corruptions are applied to the selected patches, respectively.
5 severity levels of perturbation are considered.
The final performance is averaged over different corruption types, all severity levels, and selected patches.
The positions and sizes of patches as well as the number of patches are hyper-parameters to be specified.
Adversarial Patches.
We now introduce adversarial patch attack (Karmon et al., 2018) used in our study.
The first step is to specify a patch position and replace the original pixel values of the patch with random initialized noise δ .
The second step is to update the noise to minimize the probability of ground-truth class, i.e. maximize the cross-entropy loss via multi-step gradient ascent (Madry et al., 2017).
Similarly, the positions and sizes of patches as well as the number of patches are hyper-parameters to be specified.
For both types of perturbations, the bound (255/255) is applied to the perturbed patches.
The resulting images are kept in the valid image space [0, 255]. Our code will be integrated into the ShiftHappens robustness benchmark.
3. Evaluation Metric
We use the standard metric Fooling Rate (FR) to evaluate the model robustness.
First, we collect a set of images that are correctly classified by both models that we compare.
The number of these collected images is denoted as P .
When these images are perturbed with natural patch corruption or adversarial patch attack, we use Q to denoted the number of images that are misclassified by the model.
The Fooling
Rate is then defined as FR = QP .
The lower the FR is, the more robust the model is.
4. Special Requirements on the Models to Be Evaluated
The evaluation of model robustness to patch perturbation is model-agnostic.
All the classification models can be evaluated on the proposed task, including the traditional machine learning classifiers.
There is a special setting for evaluation of robustness of
Vision Transformers.
ViT represent an input images as a list image patches.
The setting of our patch perturbation can be perfect (un)aligned with the patch representations.
Given the popularity of current ViT family models, it is indeed an interesting setting.
5. Evaluation Procedure
In the experiment, 10k test images are randomly selected from ImageNet-1k validation dataset (Deng et al., 2009) that are correctly classified by models to be evaluated.
Then for each image, we randomly sample n input image patches x i from 196 patches and perturb them with natural corruptions.
As in (Hendrycks & Dietterich, 2019), 15 types of natural corruptions with the highest level (or 5 severity levels) are
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
applied to the selected patches.
The fooling rate of the patch-based natural corruption is averaged over all the test images and corruption types.
The same image selection is also applied in case of adversarial patches.
The ℓ ∞ -norm bound of 255/255, the step size of 2/255, and the attack iterations of 10K is applied to create adversarial patches.
6. Evaluation Output and Analysis
Patch-wise Natural Corruption.
First, we investigate the robustness of DeiT and ResNet to patch-based natural corruptions.
We find that both DeiT and ResNet hardly degrade their performance when a small number of patches are corrupted ( e.g. , 4).
When we increase the number of patches, the difference between two architectures emerges: DeiT achieves a lower FR compared to its counter-part ResNet (See Tab. 1).
This indicates that DeiT is more robust against naturally corrupted patches than ResNet.
The same conclusion holds under the extreme case when the number of patches n = 196 where the whole image is perturbed with natural corruptions.
This is aligned with the observation in the existing work (Bhojanapalli et al., 2021) that ViTs are more robust to ResNet under distributional shifts.
In addition, we also increase the patch size of the perturbed patches, e.g. , if the patch size of the corrupted patch is 32 × 32 , it means that it covers 4 continuous and independent input patches as the input patch size is 16 × 16 .
As shown in Fig.
2 (Left), even when the patch size of the perturbed patches becomes larger, DeiT (marked with red lines) is still more robust than its counter-part ResNet (marked with blue lines) to natural patch corruption.
Patch-wise Adversarial Attack As shown in Tab. 1, DeiT achieves much higher fooling rate than ResNet when one of the input image patches is perturbed with adversarial perturbation.
This consistently holds even when we increase the number of adversarial patches, sufficiently supports that
DeiT is more vunerable than ResNet against patch-wise adversarial perturbation.
When more than 4 patches ( ∼ 2% area of the input image) are attacked, both DeiT and ResNet can be successfully fooled with almost 100 % FR.
When we attack a large continuous area of the input image by increasing the patch size of adversarial patches, the FR on DeiT is still much larger than counter-part ResNet until both models are fully fooled with 100 % FR.
As shown in
Fig.
2 (Right), DeiT (marked with red lines) has higher FR than ResNet under different adversarial patch sizes.
Taking all the results above together, we discover that DeiT is more robust to natural patch corruption than ResNet, whereas it is significantly more vulnerable to adversarial patch perturbation.
We also report different versions of ViT (Dosovitskiy et al., 2020; Touvron et al., 2021; Liu et al., 2021), CNN (He et al., 2016; Huang et al., 2017) as well as Hybrid architectures (Graham et al., 2021).
We train all the models in the same setting as in (Touvron et al., 2021) and report fooling rate on each model in Fig.
Four main conclusions can be drawn from the figure.
CNN variants are more robust than ViT models.
The robustness of LeViT model (Graham et al., 2021) with hybrid architecture ( i.e. , Conv Layers + Self-Attention Blocks) lives somewhere between ViT and
CNNs, as expected. 3).
Swin Transformers (Liu et al., 2021) are as robust as CNNs since attention cannot be manipulated by a single patch due to hierarchical attention and the shifted windows therein.
The self-attention in Swin Transformers is only conducted on patches within a local region.
With shifted windows, a single patch will interact with patches from different groups in different layers.
Both designs make effective adversarial patches challenging.
That’s the reason why Swin Transformer performs more robustly than popular ViTs. 4).
Mixer-MLP (Tolstikhin et al., 2021) uses the same patch-based architecture as ViTs and has no attention module.
Mixer-base with FR (31 . 36) is comparable to
ResNet and more robust than ViTs.
The results confirm that the vulnerability of ViT can be attributed to self-attention mechanism.
7. Related Works
The robustness of ViT have achieved great attention due to its great success (Bhojanapalli et al., 2021;
Naseer et al., 2021a;
Shao et al., 2021; Benz et al., 2021; Mahmood et al., 2021; Bai et al., 2021; Naseer et al., 2021b; Aldahdooh et al., 2021;
Salman et al., 2021; Yu et al., 2021; Hu et al., 2021;
Mao et al.,
2021b;a
; Naseer et al., 2021a; Tang et al., 2021).
On the one hand, (Bhojanapalli et al., 2021; Paul & Chen, 2021) show that vision transformers are more robust to natural corruptions (Hendrycks & Dietterich, 2019) compared to CNNs.
On the other hand, (Bhojanapalli et al., 2021;
Shao et al., 2021; Paul & Chen, 2021) demonstrate that ViT achieves higher adversarial robustness than CNNs under adversarial attacks.
These existing works, however, mainly focus on investigating the robustness of ViT when a whole image is naturally corrupted or adversarially perturbed.
Instead, our work focuses on patch perturbation, given the patch-based architecture trait of ViT.
The patch-based attack (Joshi et al., 2021; Fu et al., 2021) and defense (Mu &
Wagner, 2021; Shi & Han, 2021) methods have also been proposed recently.
Different from their work, we aim to understand the robustness of patch-based architectures under patch-based natural corruption and adversarial patch perturbation.
8. Conclusion
This work first shows our motivation to evaluating model robustness to patch robustness.
With experimental results.
our work shows an interesting observation on the robustness of ViT to patch perturbations.
Namely, vision transformer (e.g., DeiT) is more robust to natural patch corruption than
ResNet, whereas it is significantly more vulnerable against adversarial patches.
A deep understanding of the observation is then provided.
We reveal that the self-attention mechanism of ViT can effectively ignore natural corrupted patches but be easily misled to adversarial patches to make mistakes.
We hope this study can help the community better understand the robustness of ViT to patch perturbations.