Certiﬁed robustness against physically-realizable patch attacks via randomized cropping
Abstract
This paper proposes a certiﬁable defense against adversarial patch attacks on image classiﬁcation.
Our approach classiﬁes random crops from the original image independently and classiﬁes the original image as the majority vote over predicted classes of the crops.
Leveraging the fact that a patch attack can only inﬂuence a certain number of pixels in the image, we derive certiﬁed robustness bounds for the classiﬁer.
Our method is particularly effective when realistic transformations are applied to the adversarial patch, such as afﬁne transformations.
Such transformations occur naturally when an adversarial patch is physically introduced in a scene.
Our method improves upon the current state of the art in defending against patch attacks on CIFAR10 and ImageNet, both in terms of certiﬁed accuracy and inference time.
1. Introduction
Despite their incredible success in many computer vision tasks deep neural networks are known to be sensitive to adversarial attacks; small perturbations to an input image can lead to large changes in the output.
A wide range of defenses against adversarial attacks have been conducted in image classiﬁcation, where the goal of the attacker is simply to change the predicted label(s) of an image (Kurakin et al., 2016; Szegedy et al., 2013; Madry et al., 2017).
These works have mainly considered the so-called (cid:96) p -norm threat model, where an attacker is allowed to perturb the intensity at all pixels of the input image by a small amount.
In contrast, adversarial patch attacks are considered as physicallyrealizable alternatives, modeling scenarios where a small object is placed in the scene so as to alter or suppress clas- siﬁcation results (Brown et al., 2017).
Here, the attack is spatially compact, but can change the pixel value to any value within the allowable range.
This paper develops a practical and provably robust defense against patch attacks.
Inspired by the randomized smoothing defense (Cohen et al., 2019; Levine & Feizi, 2019) for the (cid:96) p -norm threat model, our approach classiﬁes randomly sampled sub-regions or crops of an image independently and outputs the majority vote across these crops as predicted class of the input image.
This approach is highly practical, as the crop classiﬁer can be trained using standard architectures such as VGG (Simonyan & Zisserman, 2014) or ResNet (He et al., 2016) with random cropping as its data augmentation strategy.
This is different from most existing work on certiﬁable defenses against patch attacks (Levine & Feizi, 2020; Xiang et al., 2020; Chiang et al., 2020) which need extra computation for certiﬁcation during training.
Also, the proposed approach separates the training procedure from the patch threat model, thus making the method ﬂexible against realistic settings of patch attacks – the same crop classiﬁer can be used to certify patches of the same size under different transformations such as rotation and aspect ratio changes, without having to train a different model for different transformations of the patch.
We summarize our main results on CIFAR10 and ImageNet in Table 1 in comparison with the current state of the art certiﬁable defense against patch attack (Xiang et al., 2020) with patch transformation.
We report certiﬁed accuracy, which is the percentage of test images for which classiﬁcation outcome equals to the ground truth label and is guaranteed to not change under patch attack.
Our method is better in both speed and certiﬁed accuracy compared to De-randomized smoothing (Levine & Feizi, 2020) and
PatchGuard (Xiang et al., 2020) under patch attack with possible afﬁne transformations of the patch.
In addition, our method outperforms these past approaches on ImageNet (though not on CIFAR10) in the setting where the patch aligns with coordinates of the image axes and does not undergo afﬁne transformations as in Table 2, which was the setting considered in this past work.
We have made several contributions in this paper: ﬁrst, we propose a defense against patch attack for image classiﬁca-
[Table]
[Figure]
[Table]
[Figure]
tion with certiﬁed robustness; second, the proposed method is fast in computing image certiﬁcation and robust against patch transformation; third, the proposed method can be applied to any image classiﬁcation model with only minimal changes to the training process.
2. Background and related work
Test-time adversarial attacks on ML models in general were studied in (Dalvi et al., 2004; Biggio et al., 2013), though the area gained considerable momentum when these methods were applied to deep learning systems to demonstrate that deep classiﬁers could be easily fooled by imperceptible changes to images (Szegedy et al., 2013; Goodfellow et al., 2014).
This imperceptible attack is so-called (cid:96) ∞ -norm attack, where attacks are permitted to modify any pixel in the image by (at most) some ﬁxed amount.
In this paper, we consider patch attacks (Brown et al., 2017; Eykholt et al., 2018), where a particular ‘pattern’ is designed to fool a deep learning system.
There are also two threads of research in defending against patch attacks: 1) empirical defense strategies which show stronger empirical robustness but no analytical guarantees (McCoyd et al., 2020; Naseer et al., 2019; Hayes, 2018), and 2) certiﬁed defenses which provide analytical lower bounds of classiﬁcation accuracy under patch attack.
In Chiang et al.
(2020), interval bound propagation was used to provide certiﬁcation, but the method is not applicable to commonly used image classiﬁcation networks such as ResNet50 or VGG16.
De-randomized smoothing (Levine & Feizi, 2020) uses ablation to exhaustively block all possible patch locations while (Zhang et al., 2020) uses Bagnets (Brendel & Bethge, 2019) with limited receptive ﬁeld to contain the number of affected features.
PatchGuard (Xiang et al., 2020) adds additional detection of patch location and then sets features extracted from the detected locations to be zero.
We will compare our proposed method with De-randomized smoothing (DRS) and
PatchGuard (PG) in the experiments and an overview of the two prior arts can be found in Appendix D.
3. Provable patch defense with randomized cropping
A physically-realizable adversarial patch attack can be found by solving the optimization problem:
[Equation]
where f θ : X → Y denotes some hypothesis function; θ denotes parameters of the model, x ∈ X denotes the input to the network and δ ∈ X the perturbation to the input;
∆ ⊆
X is the set of allowable perturbations;
y ∈ Y denotes the true label; (cid:96) : Y × Y
→ R denotes a loss function that measures the performance of the image classiﬁer; and T : X → X is the set of transformations that the perturbation δ might go through.
The feasible set ∆ denotes a simple allowable set of values for the patch, which typically would just be constrained to lie in valid RGB space.
Patches overwrite a
[Table] portion of the image with the patch perturbation itself, at a given location with a given set of transformations, such as scaling, rotation [Table] and other transformations in T .
We refer to this combination as the patch application function A :
X × X × T → X , where A ( x, δ, t ) denotes the application of patch δ to image x with transformation t .
Throughout this paper, we consider the patch being one connected area bounded by a rectangle with size p i × p j .
3.1. Randomized cropping defense
Although a patch attack can change the pixel value to any arbitrary value, it can only inﬂuence the pixels within the patch itself.
Therefore, networks extracting features with compact receptive ﬁelds and aggregating such local features for ﬁnal classiﬁcation are more robust against patch attacks.
Based on above observations [Table] we propose a randomized cropping approach as shown in Fig. 1: given a full-size test image x as input, we ﬁrst randomly select n crops uniformally over all possible locations, where crop size k i × k j is smaller than image size
m i × m j in both dimensions.
Each of the sampled crops ( (cid:98) x i ) then goes through the cropbased classiﬁer g θ and its predicted class ( (cid:98) y i ) is obtained.
The ﬁnal classiﬁcation of x is the majority of { (cid:98) y i } ni =1 .
The equivalent pseudo code of Fig.
1 is in
Appendix E.
We argue that the training procedure of randomized cropping classiﬁer f θ only requires training the crop classiﬁer
g
θ
(see Section
A) without any assumption on attack parameters (size, shape, and location), hence our method is robust against different patch shapes given the same patch size.
We show supportive experimental results in Section 4 for this claim.
3.2. Certiﬁable robustness bound
Here we compute the probability that an image x is certiﬁed robust, i.e., the classiﬁcation outcome of x cannot be changed by patch attack of given size.
Given a clean image, we randomly sample n crops { (cid:98) x i } ni =1 and obtain the set of n predicted classes { (cid:98) y i } ni
=1 of the crops.
Let n 1 be the number crops that are predicted as the majority in { (cid:98) y i } ni =1 , and n 2 be the number crops that are predicted as the second majority.
If there are fewer than n 2 to 1 = (cid:98) n 1 − n 2 (cid:99)+ 1 crops that overlap with the patch, then x is certiﬁed robust.
Therefore, p c can be represented as the probability that fewer than n 2 to 1 crops out of the n sampled crops overlap with the adversarial patch.
The below derivation assumes that the random selection is uniform over all crop locations with replacement.
Derivations for uniform sampling crops without replacement can be found in Appendix F.
Following the argument above, we ﬁrst compute the probability that a single sampled crop x i overlaps the adversarial patch.
This probability p a = n adv n all can be computed as the total number of crops overlapping with the patch (denoted by n adv ) divided by the number of all possible crops ( n all ).
When the patch is at the center of the image where it can inﬂuence the highest number of crops,
[Equation]
The probability of certiﬁcation p c is equal to the probability that out of the n crops { (cid:98) x i } ni =1 , at most n 2 to 1 − 1 of them overlaps with the adversarial patch:
[Equation]
where C ni is the binomial coefﬁcient ( n choose i ).
If p c is close to 1.0, the input image is certiﬁably robust under patch attack.
In the experiments we select p c ≥ 0 .
Throughout this paper, we consider certiﬁed accuracy , meaning only data with predicted class y (cid:48) being the ground truth class will be considered for certiﬁcation.
Note that because the crops are randomly sampled, n 2 to 1 and hence p c is an instance of a random distribution.
We argue that when n is large enough, there n 2 to 1 and p c will have very small variance.
Experiments to verify this point are in Appendix G.
[Table] [Table]
4. Experimental results
We conduct experiments on two benchmark datasets: CIFAR10 and ImageNet.
For both datasets we report certiﬁed accuracy as percentage of images that are classiﬁed correctly and can be certiﬁed with probability higher than 0.95.
Detailed experiment setup are in Appendix B, and effects of different parameters are discussed in Appendix C.
Clean accuracy and inference time.
Clean model accuracy and inference time per image are shown in Table 1.
Our method is faster than both prior methods under all cases.
This is because although multiple forward passes are needed, the number of crops is smaller than equivalent sub-regions in Bagnets and crops are much smaller the full images as in De-randomized smoothing (DRS).
PatchGuard with Bagnets (PG-Bagnets) is slower than our method in ImageNet for having to go through logits of all classes sequentially.
4.1. Without patch transformation at test time
We ﬁrst present the certiﬁcation accuracy in Table 2 of our method along with De-randomized smoothing (DRS) (Levine & Feizi, 2020) and PatchGuard (PG)(Xiang et al., 2020) with De-randomized smoothing (PG+DRS) and Bagnets (PG+Bagnets) as base structure on patch size ranging from 0 .
4% to 3% when there’s no patch transformation at test time, i.e., square patches in both test and train time.
Methods with the highest certiﬁed accuracy are highlighted in bold.
We can see that although our method is sub-optimal for CIFAR10, for ImageNet we have the highest certiﬁed accuracy except for patch size 0 .
From Table 1 andwe can see that on ImageNet our proposed method is the fastest and with the highest certiﬁed accuracy as well as clean accuracy the certiﬁed accuracy – this shows that the proposed randomized cropping defense is practical in the aspects of fast certiﬁcation and high certiﬁed accuracy for the dataset that is closer to real-life pictures.
4.2. With patch transformation at test time
As described in Section 3, one cannot assume such area would always align with coordinate axes of the image – even if the physical patch itself is square, when the scene is captured with different camera angles, the patch on the image will be rotated (rotation in x-y plane) or stretched (rotation in depth).
Therefore it is important that the patch defense can also provide guarantees when the patch is rotated or if the aspect ratio is varied.
In Table 3 we compare our method with DRS and PGDRS/PG-Bagnets when a square patch is rotated 45 degrees and with aspect ratio (AR) 1:1, 6:1, 2.7:1, 1:2.7, 1:6, respectively.
We highlight the highest certiﬁed accuracy under worst transformation in the last row.
Aspect ratio 1:1 without rotation is the same as Table 2 but listed here as reference.
We can clearly see that in Table 3, with the same patch size, transformation brings down the certiﬁed accuracy of all three competing methods, however, the proposed randomized cropping defense has the highest certiﬁed accuracy under worst-case patch transformation.
This is because of our random sampling strategy that we have neither ﬁxed the locations of sub-regions as in Bagnets, nor ﬁxed smoothing strategy as in De-randomized smoothing.
5. Conclusion
This paper proposes a new defense against adversarial patch attacks.
The proposed approach decomposes an image into a random assortment of crops, each of which is processed by a classiﬁer, and the majority across the crops is used as the classiﬁcation outcome for the input image.
This approach provides a signiﬁcant advance in improving certiﬁed accuracy when patches could be transformed at test time, while maintaining a high clean accuracy compared to prior art. Acknowledgements
This work is funded by DARPA HR
A. Training randomized cropping classiﬁer
The robust classiﬁer f θ has three components: (1) sampling n crops with uniform distribution, (2) classifying crops with g θ , and (3) majority voting.
Since both random sampling of the crops via a uniform distribution and majority voting have no trainable parameters, the set of trainable parameters of f θ is the same as g θ .
Therefore training the robust classiﬁer f θ only involves training the crop classiﬁer g θ , which can be trained in standard ways with only data augmentation of randomly cropping the input x .
Other variables are either pre-determined hyperparameters (crop size k i , k j ), or can be adjusted at test time (number of crops n ).
B. Experiment setup
For CIFAR10, we use ResNet9 without the ﬁnal pooling layer as the crop model, and for ImageNet we use ResNet as the crop model and resize and pad the image to 224x224.
Experiments and timing are done on single Nvidia
Ti
GPU, and 16-core Intel i7-5960X CPU.
For both models we use cyclic learning rate with 10 − 3 initial learning rate.
The models are trained 30 epochs and training time is
GPU-hours and 328 GPU-hours for CIFAR10 and ImageNet, respectively.
we chose crop size of 10x10 (10% total area) for CIFAR and 80x80 (13% total area) for ImageNet; for these experiments we assume patch shape is square with no rotation .
For each input image in CIFAR10, we randomly sample 128 crops and for each image in ImageNet we sample 256 crops.
Positional encoding.
Because both CIFAR10 and ImageNet images have a certain region of interest which is usually at the center of the image, some parts of the images contain more information than others.
Therefore, crops sampled at different position of the image contain different information.
To represent such information, we add learnable positional encoding (Vaswani et al., 2017) to the ﬁrst layer of our classiﬁer.
C. Discussions on different parameter setting
We also show the certiﬁcation accuracy with different thresholds for p c Table 4 in Appendix.
How to choose crop size?
Assuming that the number of crops n is ﬁxed, then in general larger crops leads to a better clean performance, as each crop contains more information when it covers more pixel area.
Also with larger crops, the crop classiﬁcation accuracy of g θ would be better, indicating that n 2 to 1 could be larger and increases the probability of certiﬁcation p c .
However, larger crops also means that the probability of overlapping the adversarial patch is higher (Eq. 2) which will decrease p c .
Therefore, for a given size of adversarial patch, there exists an optimal crop size which maximizes the certiﬁcation probability.
To demonstrate the inﬂuence of crop size on certiﬁcation accuracy and clean accuracy, clean and certiﬁed accuracy with or without positional encoding, with regards to different crop sizes are shown in Figure 2 for 2.4% patch on
CIFAR10 and 2.0% patch on ImageNet – we use square crops and square patches aligning with coordinate axes of the image, i.e., no patch transformation.
Comparing clean accuracy with and without positional encoding, we can see that although clean accuracy still increases as crop size increases when crop classiﬁer includes positional encoding, but not as much as without positional encoding.
Such results show positional encoding does provide extra information for crops sampled from different locations.
On the other hand, comparing clean accuracy with certiﬁed accuracy, it is clear that the certiﬁed accuracy actually gets lower when crop size crosses some threshold as discussed above.
Note that similar experiments as in Figure 2 can be conducted for different sizes of adversarial patches to ﬁnd optimal crop size.
However we used ﬁxed crop size in Table 1, 2, 3 to compare with DRS and PG to have a fair comparison, as these two approaches do not have tune-able ablation/kernel size.
How to choose number of crops?
We show certiﬁed accuracy and inference time with different number of sampled crops for 2.4% patch on CIFAR10 and 2.0% patch on ImageNet.
Certiﬁed accuracy of using all crops, i.e., selecting crops at all locations once without sampling, is plotted with the black dotted line as reference.
This line can be viewed as the upper bound of the proposed method.
With more crops sampled, the certiﬁed accuracy increases but however inference time also increases close to linearly.
Therefore we chose relatively small number of crop samples to balance between inference time and certiﬁcation accuracy.
D. Overview of De-randomized smoothing (DRS) and PatchGuard (PG)
De-randomized smoothing (Levine & Feizi, 2020) ablates all possible parts of the image and aggregates logits of the ablated images for certiﬁcation.
The method compares number of ablated images with high logit value of the majority predicted class with the number of ablated images with high logit value of the second majority predicted class.
If the difference between the two is larger than two times possible number of ”ablation blocks” affected by adversarial patch, then this image is certiﬁed robust against patch attack.
The paper proposes two modes of ablation: block smoothing and band smoothing.
Block smoothing ablates square blocks while band smoothing ablates one column of the image.
To represent ablated regions, the image classiﬁer accept p c ≥ 0 .p
c ≥ 0 .p
c ≥ 0 .p c ≥ 0 .
CIFAR10 2.4% 52. ImageNet 2.0% 17.
[Figure]
[Figure]
e r e n c e t i m e ( m s )
[Figure]
Algorithm 1 Certify if patch attack can change the predicted class of an image x
Input: Full size image x , label y , crop classiﬁer g
θ
i
= 1 to n set (cid:98) x
i as a k i × k j crop of x at random location (cid:98) y
i =
g θ ( (cid:98) x i ) //
Predicted class of crop (cid:98) x i (cid:98) y (cid:48) , (cid:99) y
(cid:48)(cid:48)
= majority and second majority of { (cid:98) y i } ni =1 , n 1 , n 2 = number of crops classiﬁed as (cid:98) y (cid:48) , (cid:99) y
(cid:48)(cid:48) , respectively
If (cid:98) y (cid:48) (cid:54) =
y return not certiﬁed
Else compute p c using Eq.
3, if p c is close to 1.0 then return certiﬁed, else return not certiﬁed three additional channels representing ablated pixels of the original RGB channels.
The main idea of PatchGuard (Xiang et al., 2020) is to detect possible patch location and mask these locations for downstream robust classiﬁcation.
To detect patch location, the method identify if there are any local regions that contribute abnormally strongly to a class.
If there does exist such a region, it is considered as the potential location of a patch and the features extracted from the region is masked/discarded.
Because such robust masking procedure can be combined with other robust classiﬁcation approaches, the paper evaluates its robust masking with two classiﬁcation models:
De-randomized smoothing (PG-DRS) and BagsNet (PGBN).
E. Detailed certiﬁcation procedure
We summarize the certiﬁcation procedure for a single image in Algorithm 1.
A more robust classiﬁer should be able to certify and correctly classify a higher percentage of clean images in the test set.
F. Uniform sampling without replacement
In this section we derive certiﬁcation probability and experimental results with uniform sampling without replacement.
Let n iall be the number of all possible location for the i th sampled crop, n adv be the number of crop locations that overlaps with the adversarial patch, and p ia is the probability that the i th crop overlaps with the adversarial patch, then n adv = min
( p i + k
i − 1 , m i − k
i + 1) × min ( p j + k j − 1 , m j − k j + 1) , n iall = ( m i − k
i + 1) × ( m j − k j + 1) − i, and p
[Equation]
[Equation]
The probability of the image being certiﬁed p c is then the probability of less than n 2 to 1 crops overlaps with the patch.
The closed-form expression of p c is complicated yet not informative and hence omitted here.
Comparing p ia with the p a in Section 3, we can see that when sampling without replacement, the probability of sampling a crop that overlaps with the adversarial patch increases with the number of crops sampled, and hence decreases the probability of certiﬁcation p c .
On the other hand, sampling without replacement enlarges the expected area that crops would cover, so the clean performance will be better than sampling with replacement.
We compare the certiﬁed and clean accuracy with and without replacement in Table 5.
As number of crops increases, we can see that the gain of clean accuracy for sampling without replacement decreases because when number of crops increases, even sampling with replacement is likely to cover most of the pixels, and the gain is slightly more signiﬁcant in ImageNet than CIFAR10.
This may be because ImageNet images are in general more complex than
CIFAR10 and having the crops covering more pixels over the image could help the overall classiﬁcation.
The certiﬁed accuracy for sampling without replacement gets worse than with replacement since the probability of sampling a crop increases more signiﬁcantly as the number of crops sampled increases.
This is particularly true for CIFAR – with image size 32x32 and patch size 10x10, the number of non-overlapping crops that does not overlap with the patch is only 484, out of 1024 all possible locations.
This means when sampling 512 crops, there are at least 28 crops overlapping with the adversarial patch, which signiﬁcantly decrease p c .
G. Experiments on p c interval
In this section we show the interval of n 2 to 1 and p c with different number of crops sampled.
We run the certiﬁcation process over test set 200 times to obtain the interval and variance of n 2 to 1 and p c for each test image.
Interval is deﬁned as difference between the highest value and the lowest value.
We ran this experiment over patch size 2.4% for CIFAR10 and 2.0% for ImageNet.
As shown in Table 6, with increasing number of crops, both interval and variance of n 2 to 1 and p c decrease, to negligible values.
[Table]
[Table]