C ERTIFIED D EFENSES FOR A DVERSARIAL P ATCHES ∗
A BSTRACT
Adversarial patch attacks are among of the most practical threat models against realworld computer vision systems.
This paper studies certiﬁed and empirical defenses against patch attacks.
We begin with a set of experiments showing that most existing defenses, which work by pre-processing input images to mitigate adversarial patches, are easily broken by simple white-box adversaries.
Motivated by this ﬁnding, we propose the ﬁrst certiﬁed defense against patch attacks and propose faster methods for its training.
Furthermore, we experiment with different patch shapes for testing, obtaining surprisingly good robustness transfer across shapes, and present preliminary results on certiﬁed defense against sparse attacks.
Our complete implementation can be found on: https://github.com/Ping-C/ certifiedpatchdefense .
I NTRODUCTION
Despite the great success of neural networks for vision problems, they are easily fooled by adversarial attacks in which the input to a machine learning model is modiﬁed with the goal of manipulating its output.
Research in this area is largely focused on norm-bounded attack (Madry et al., 2017; Tram ` er & Boneh, 2019; Shafahi et al., 2019), where the adversary is allowed to perturb all pixels in an image provided that the (cid:96) p -norm of the perturbation is within prescribed bounds.
Other adversarial models were also proposed, such as functional (Laidlaw & Feizi, 2019), rotation/translation (Engstrom et al., 2017), and Wasserstein (Wong et al., 2019), all of which allow modiﬁcation to all pixels.
Whole-image perturbations are unrealistic for modeling ”physical-world” attacks, in which a realworld object is modiﬁed to evade detection.
A physical adversary usually modiﬁes an object using stickers or paint.
Because this object may only occupy a small portion of an image, the adversary can only manipulate a limited number of pixels.
As such, the more practical patch attack model was proposed (Brown et al., 2017).
In a patch attack, the adversary may only change the pixels in a conﬁned region, but is otherwise free to choose the values yielding the strongest attack.
The threat to real-world computer vision systems is well-demonstrated in recent literature where carefully crafted patches can fool a classiﬁer with high reliability (Brown et al., 2017; Karmon et al., 2018), make objects invisible to an object detector (Wu et al., 2019; Lee & Kolter, 2019), or fool a face recognition system (Sharif et al., 2017).
In light of such effective physical-world patch attacks, very few defenses are known to date.
In this paper, we study principled defenses against patch attacks.
We begin by looking at existing defenses in the literature that claim to be effective against patch attacks, including Local Gradient
Smoothing (LGS) (Naseer et al., 2019) and Digital Watermarking (DW) (Hayes, 2018).
Similar to what has been observed for whole-image attacks by (Athalye et al., 2018), we show that these patch defenses are easily broken by stronger adversaries.
Concretely, we demonstrate successful white-box attacks, where the adversary designs an attack against a known model, including any pre-processing steps.
To cope with such potentially stronger adversaries, we train a robust model that produces a lower-bound on adversarial accuracy.
In particular, we propose the ﬁrst certiﬁable defense against patch attacks by extending interval bound propagation (IBP) defenses (Gowal et al., 2018; Mirman et al., 2018).
We also propose modiﬁcations to IBP training to make it faster in the patch setting.
Furthermore, we study the generalization of certiﬁed patch defenses to patches of different shapes, and observe that robustness transfers well across different patch types.
We also present preliminary results on certiﬁed defense against the stronger sparse attack model, where a ﬁxed number of possibly non-adjacent pixels can be arbitrarily modiﬁed (Modas et al., 2019).
P ROBLEM S ETUP
We consider a white-box adversary that is allowed to choose the location of the patch (chosen from a set L of possible locations) and can modify pixels within the particular patch (chosen from the set P ) similar to (Karmon et al., 2018).
An attack is successful if the adversary changes the classiﬁcation of the network to a wrong label.
In this paper, we are primarily interested in the patch attack robust accuracy (adversarial accuracy for short) as deﬁned by
[Equation]
1) where the operator A places the adversarial patch p on a given image x at location l, f is a neural network with parameter θ, X is a distribution of images, and X is a characteristic function that takes value 1 if its argument is true, and
0 otherwise.
In this model, the strength of the adversary can vary depending on the set of possible patches allowed, and the type of perturbation allowed within the patch.
In what follows, we assume the standard setup in which the adversary is allowed any perturbation that maintains pixel intensities in the range [0 , 1] .
Unless otherwise noted, we also assume the patch is restricted to a square of prescribed size.
We consider two different options for the set L of possible patch locations.
First, we consider a weak adversary that can only place patches at the corner of an image.
We ﬁnd that even this weak model is enough to break existing patch defenses.
Then, we consider a stronger adversary with no restrictions on patch location, and use this model to evaluate our proposed defenses.
Note that an adversary, when restricted to modify only a square patch at location l in the image, has the freedom to modify any non-square subset of these pixels.
In other words, a certiﬁed defense against square patch attacks also provably subverts any non-square patch attack that ﬁts inside a small enough square.
In general, calculating the adversarial accuracy (1) is intractable due to non-convexity.
Common approaches try to approximate it by solving the inner minimization using a gradient-based method.
However, in Section 3, we show that depending on how the minimization is solved, the upper bound could be very loose: a model may appear to be very robust, but fail when faced with a stronger attack.
To side-step the arms race between attacks and defenses, in Section 4, we extend the work of (Gowal et al., 2018) and (Mirman et al., 2018) to train a network that produces a lower bound on adversarial accuracy.
We will refer to approximations of the upper bound as empirical adversarial accuracy and the lower bound as certiﬁed accuracy .
V ULNERABILITY OF E XISTING D EFENSES
We start by examining existing defense strategies that claim to be effective against patch attacks.
Similar to what has been observed for whole-image attacks by Athalye et al.
, we show that these patch defenses can easily be broken by white-box attacks, where the adversary optimizes against a given model including any pre-processing steps.
3. E XISTING D EFENSES
Under our threat model, two defenses have been proposed that each use input transformations to detect and remove adversarial patches.
The ﬁrst defense is based on the observation that the gradient of the loss with respect to the input image often exhibits large values near the perturbed pixels.
In (Hayes, 2018), the proposed digital watermarking (DW) approach exploits this behavior to detect unusually dense regions of large
Smoothing and Digital Watermarking.
We consider two types of adversaries, one that takes the defense into account during backpropagation and one that does not
[Table]
gradient entries using saliency maps, before masking them out in the image.
Despite a 12% drop in accuracy on clean (non-adversarial) images, this defense method supposedly achieves an empirical adversarial accuracy of 63% for non-targeted patch attacks of size 42 × 42 ( 2% of the image pixels), using 400 randomly picked images from ImageNet (Deng et al., 2009) on VGG19 (Simonyan &
Zisserman, 2014).
The second defense, Local Gradient Smoothing (LGS) by Naseer et al.
(2019) is based on the empirical observation that pixel values tend to change sharply within these adversarial patches.
In other words, the image gradients tend to be large within these adversarial patches.
Note that the image gradient here differs from the gradient in Hayes (2018), the former is with respect the changes of adjacent pixel values and the later is with respect to the classiﬁcation loss.
Naseer et al.
(2019) propose suppressing this adversarial noise by multiplying each pixel with one minus its image gradient as in (2).
To make their methods more effective, Naseer et al.
(2019) also pre-process the image gradient with a normalization and a thresholding step.
[Equation]
The λ here is a smoothing hyper-parameter.
Naseer et al.
(2019) claim the best adversarial accuracy on ImageNet with respect to patch attacks among all of the defenses we studied.
They also claim that their defense is resilient to Backward Pass Differential Approximation (BPDA) from Athalye et al.
(2018), one of the most effective methods to attack models that include a non-differentiable operator as a pre-processing step.
3. B REAKING E XISTING D EFENSES
Using a similar setup as in (Hayes, 2018; Naseer et al., 2019), we are able to mostly replicate the reported empirical adversarial accuracy for Iterative Fast Gradient Sign Method (IFGSM), a common gradient based attack, but we show that when the pre-processing step is taken into account, the empirical adversarial accuracy on ImageNet quickly drops from ∼ 70%( ∼ 50%) for LGS(DW) to levels around ∼ 10% as shown in Table 1.
Speciﬁcally, we break DW (Hayes, 2018) by applying BPDA, in which the non-differentiable operator is approximated with an identity mapping during the backward pass.
We break LGS (Naseer et al., 2019) by directly incorporating the smoothing step during backpropagation.
Even though the windowing and thresholding steps are non-differentiable, the smoothing operator provides enough gradient information for the attack to be effective.
To make sure that our evaluation is fair, we used the exact same models as Hayes (2018)
(VGG19) and Szegedy et al.
(Inception V3).
We also consider a weaker set of attackers that can only attack the corners, the same as their setting.
Further, we ensure that we were able to replicate their reported result under similar setting.
C ERTIFIED D EFENSES
Given the ease with which these supposedly strong defenses are broken, it is natural to seek methods that can rigorously guarantee robustness of a given model to patch attacks.
With such certiﬁable guarantees in hand, we need not worry about an adversary with a stronger optimizer, or a more clever algorithm for choosing patch locations.
4. B ACKGROUND ON CERTIFIED DEFENSES
Certiﬁed defenses have been intensely studied with respect to norm-bounded attacks (Cohen et al., 2019; Wong & Kolter, 2017; Gowal et al., 2018; Mirman et al., 2018; Zhang et al., 2019b).
In all of these methods, in addition to the prediction model, there is also a veriﬁer.
Given a model and an input, the veriﬁer outputs a certiﬁcate if it is guaranteed that the image can not be adversarially perturbed.
This is done by checking whether there exists any nearby image (within a prescribed (cid:96) p distance) with a different label than the image being classiﬁed.
While theoretical bounds exist on the size of this distance that hold for any classiﬁer (Shafahi et al., 2018), exactly computing bounds for a speciﬁc classiﬁer and test image is hard.
Alternatively, the veriﬁer may output a lower bound on the distance to the nearest image of a different label.
This latter distance is referred to as the certiﬁable radius.
Most of these veriﬁers provide a rather loose bound on the certiﬁable radius.
However, if the veriﬁer is differentiable, then the network can be trained with a loss that promotes tightness of this bound.
We use the term certiﬁcate training to refer to the process of training with a loss that promotes strong certiﬁcates.
Interval bound propagation (IBP) (Mirman et al., 2018; Gowal et al., 2018) is a very simple veriﬁer that uses layer-wise interval arithmetic to produce a certiﬁcate.
Even though the
IBP certiﬁcate is generally loose, after certiﬁcate training, it yields state-of-the-art certiﬁably-robust models for l ∞ -norm bounded attacks (Gowal et al., 2018; Zhang et al., 2019b).
In this paper, we extend IBP to train certiﬁably-robust networks resilient to patch attacks.
We ﬁrst introduce some notation and basic algorithms for IBP training.
Notation We represent a neural network with a series of transformations h ( k ) for each of its k layers.
We use z ( k ) ∈ R n k to denote the output of layer k , where n k is the number of units in the k th layer and z (0) stands for the input.
Speciﬁcally, the network computes
[Equation]
Certiﬁcation Problem To produce a certiﬁcate for an input x 0 , we want to verify that the following condition is true with respect to all possible labels y :
[Equation]
Here, e i is the i th basis vector, and m y is called the margin following Wong & Kolter (2017).
Note that m y true is always equal to 0.
The vector m contains all margins corresponding to all labels.
B ( x 0 ) is the constraint set over which the adversarial input image may range.
In a conventional setting, this is an (cid:96) ∞ ball around x 0 .
In the case of patch attack, the constraint set contains all images formed by applying a patch to x 0 ;
[Equation]
The Basics of Interval Bound Propagation (IBP)
We now describe how to produce certiﬁcates using interval bound propagation as in (Gowal et al., 2018).
Suppose that for each component in z ( k − 1) we have an interval containing all the values which this component reaches as z (0) ranges over the ball B ( x 0 ) .
If z ( k ) =
h ( k ) ( z ( k − 1) ) is a linear (or convolutional) layer of the form z ( k ) =
W ( k ) z ( k − 1) +
b ( k ) , then we can get an outer approximation of the reachable interval range of activations by the next layer
z
( k ) using the formulas below
[Equation] [Equation]
Here z ( k − 1) denotes the upper bound of each interval, z ( k − 1) the lower bound, and | W ( k ) | the element-wise absolute value.
Alternatively, if h ( k ) ( z ( k − 1) ) is an element-wise monotonic activation (e.g., a ReLU), then we can calculate the outer approximation of the reachable interval range of the next layer using the formulas below.
[Equation]
When the feasible set B ( x 0 ) represents a simple (cid:96) ∞ attack, the range of possible z (0) values is trivially characterized by an interval bound z (0) and z (0) .
Then, by iteratively applying the above rules, we can propagate intervals through the network and eventually get z ( K ) and z ( K ) .
A certiﬁcate can then be given if we can show that (3) is always true for outputs in the range z
( K ) and z ( K ) with respect to all possible labels.
More speciﬁcally, we can check that the following holds for all y
[Equation]
Training for Interval Bound Propagation To train a network to produce accurate interval bounds, we simply replace standard logits with the − m vector in (3).
Note that all elements of m need to be larger than zero to satisfy the conditions in (3), and m y true is always equal to zero.
Put simply, we would like m y true = 0 to be the least of all margins.
We can promote this condition by training with the loss function
[Equation]
Unlike regular neural network training, stochastic gradient descent for minimizing equation 10 is unstable, and a range of tricks are necessary to stabilize IBP training (Gowal et al., 2018).
The ﬁrst trick is merging the last linear weight matrix with ( e y − e y true ) before calculating − m y .
This allows a tighter characterization of the interval bound that noticeably improves results.
The second trick uses an “epsilon schedule” in which training begins with a perturbation radius of zero, and this radius is slowly increased over time until a sentinel value is reached.
Finally, a mixed loss function containing both a standard natural loss and an IBP loss is used.
In all of our experiments, we use the merging technique and the epsilon schedule, but we do not use a mixed loss function containing a natural loss as it does not increase our certiﬁcate performance.
4. C ERTIFYING AGAINST PATCH ATTACKS
We can now describe the extension of IBP to patches.
If we specify the patch location, one can represent the feasible set of images with a simple interval bound: for pixels within the patch, the upper and lower bound is equal to 1 and 0.
For pixels outside of the patch, the upper and lower bounds are both equal to the original pixel value.
By passing this bound through the network, we would be able to get m single location and verify that they satisfy the conditions in (3).
However, we have to consider not just a single location, but all possible locations L to give a certiﬁcate.
To adapt the bound to all possible location, we pass each of the possible patches through the network, and take the worst case margin.
More speciﬁcally,
[Equation]
Similar to regular IBP training, we simply use m es ( L ) to calculate the cross entropy loss for training and backpropagation,
[Equation]
Unfortunately, the cost of producing this na¨ıve certiﬁcate increases quadratically with image size.
Consider that a CIFAR-10 image is of size 32 × 32, requiring over a thousand interval bounds, one for each possible patch location.
To alleviate this problem, we propose two certiﬁcate training methods:
Random Patch and Guided Patch , so that the number of forward passes does not scale with the dimension of the inputs.
Random Patch Certiﬁcate Training In this method, we simply select a random set of patches out of the possible patches and pass them forward.
A level of robustness is achieved even though a very small number of random patches are selected compared to the total number of possible patches
[Equation]
random
[Equation] [Equation]
where S is a random subset of L .
Similarly, the random patch certiﬁcate loss is calculated as below.
[Equation]
Guided Patch Certiﬁcate Training In this method, we propose using a U-net (Ronneberger et al., 2015) to predict m single patch , and then randomly select a couple of locations based on the predicted m single patch so that fewer patches need to be passed forward.
Note that very few patches contribute to the worst case bound m es in (11).
In fact, the number of patches that yield the worst case margins will be no more than the number of labels.
If we know the worst-case patches beforehand, then we can simply select the few worst-case patches during training.
We propose to use U-net as the number of locations and margins is very large.
For a square patch of size n × n and an image of size m × m , the total number of possible locations is ( m − n + 1) 2 , and for each location the number of margins is equal to the number of possible labels.
[Equation]
Given the U-net prediction of m pred , we then randomly select a single patch for each label based on the softmax of the predicted m pred .
The number of selected patches is equal to the number of labels.
After these patches are passed forward, the U-net is then updated with a mean-squared-error loss between the predicted margins m pred and the actual margins m actual .
Note that only a few patches are selected at a time, so that the mean-squared-error only passes through the selected patches.
[Equation]
The network is trained with the following loss:
[Equation]
Certiﬁcation Process
In all our experiments, we check that equation (3) is satisﬁed by iterating over all possible patches and forward-passing the interval bounds generated for each patch; this overhead is tolerable at evaluation time.
4. C ERTIFYING AGAINST SPARSE ATTACKS
IBP can also be adapted to defend against sparse attack where the attacker is allowed to modify a ﬁxed number ( k ) of pixels that may not be adjacent to each other (Modas et al., 2019).
The only modiﬁcation is that we have to change the bound calculated from the ﬁrst layer to
[Equation]
and apply equation (5) and (6) for the subsequent layers.
Here, ( . ) top k is the sum of the largest k elements in the vector.
E XPERIMENTS
In this section, we compare our certiﬁed defenses with exiting algorithms on two datasets and three model architectures of varying complexity.
We consider a strong attack setting in which adversarial patches can appear anywhere in the image.
Different training strategies for the certiﬁed defense are also compared, which shows a trade-off between performance and training efﬁciency.
Furthermore, we evaluate the transferability of a model trained using square patches to other adversarial shapes, including shapes that do not ﬁt in any certiﬁed square.
The training and architectural details can be found in Appendix A.1.
We also present preliminary results on certiﬁed defense against sparse attacks.
5. C OMPARISON AGAINST EXISTING DEFENSES
In this section, we study the effectiveness of our proposed IBP certiﬁed models against an adversary that is allowed to place patches anywhere in the image, even on top of the salient object.
If the patch is sufﬁciently small, and does not cover a large portion of the salient object, then the model should still classify correctly, and defense against the perturbation should be possible.
In the best case, our IBP certiﬁed model is able to achieve 91.6% certiﬁed (Table 2) with respect to a 2 × 2 patch ( ∼ .5% of image pixels) adversary on MNIST.
For more challenging cases, such as a
× 5 ( ∼ 2.5% of image pixels) patch adversary on CIFAR-10, the certiﬁed adversarial accuracy is only 24.9% (Table 2).
Even though these existing defenses appear to achieve better or comparable adversarial accuracy as our IBP certiﬁed model when faced with a weak adversary, when faced with a stronger adversary their adversarial accuracy dropped to levels below our certiﬁed accuracy for all cases that we analyzed.
When evaluating existing defenses, we only report cases where non-trivial adversarial accuracy is achieved against a weaker adversary.
We do not explore cases where LGS and DW perform so poorly that no meaningful comparison can be done.
LGS and DW are highly dependent on hyperparameters to work effectively against naive attacks, and yet neither Naseer et al.
nor Hayes (2018) proposed a way to learn these hyperparameters.
By trial and error, we were able to increase the adversarial accuracy against a weaker adversary for some settings, but not all.
In addition, we also notice a peculiar feature of DW: when we increase the adversarial accuracy, the clean accuracy degrades, sometimes so much that it is even lower than the empirical adversarial accuracy.
This happens because DW always removes a patch from the prediction.
When an adversarial patch is detected, it is likely to be removed, enabling correct prediction.
On the other hand, when there are no adversarial patches, DW removes actual salient information, resulting in lower clean accuracy.
Here we did not compare our results with adversarial training, because even though it produces some of the most adversarially robust models, it does not offer any guarantees on the empirical robust accuracy, and could still be decreased further with stronger attacks.
For example, Wang et al.
(2019) proposed a stronger attack that could ﬁnd 47% more adversarial examples compared to gradient based method.
Further, adversarial training on all possible patches would be even more expensive compared to certiﬁcate training, and is slightly beyond our computational budget.
Compared to state-of-the-art certiﬁed models for CIFAR with L ∞ -perturbation, where Zhang et al.
(2019a) proposed a deterministic algorithm that achieves clean accuracy of 34 . 0% , our clean accuracy for our most robust CIFAR 5 × 5 model is 47 .
8% when using a large model (Table 2).
[Table]
C OMPARISON OF TRAINING STRATEGIES
We ﬁnd that given a ﬁxed architecture all-patch certiﬁcate training achieves the best certiﬁed accuracy.
However, given a ﬁxed computational budget, random and guided training signiﬁcantly outperform all-patch training.
Finally, guided-patch certiﬁcate training consistently outperforms random-patch certiﬁcate training by a slim margin, indicating that the U-net is learning how to predict the minimum margin m .
In Table 3, we see that given a ﬁxed architecture all-patch certiﬁcate training signiﬁcantly outperforms both random-patch certiﬁcate training and guided-patch certiﬁcate training in terms of certiﬁed accuracy, outperforming the second best certiﬁed defenses in each task by 2.6% (MNIST, 2 × 2), 7.3% (MNIST, 5 × 5), 3.9% (CIFAR-10, 2 × 2), and 3.4% (CIFAR-10, 5 × 5).
However, all-patch certiﬁcate training is very expensive, taking on average 4 to 15 times longer than guided-patch certiﬁcate training and over 30 to 70 times longer than random-patch certiﬁcate training.
On the other hand, given a limited computational budget, random-patch and guided-patch training signiﬁcantly outperforms all-patch training.
Due to the efﬁciency of random-patch and guided-patch training, they scale much better to large architectures.
By switching to a large architecture (5 layer wide convolutional network), we are able to boost the certiﬁed accuracy by over 10% compared to the best performing all-patch small model (Table 2).
Note that we are unable to all-patch train the same architecture as it will take almost 15 days to complete, and is out of our computational budget.
Guided-patch certiﬁcate training is slightly more expensive compared to random patch, due to overhead from the U-net architecture.
However, given the 10 patches picked, guided-patch certiﬁcate training consistently outperforms random-patch certiﬁcate training, indicating that the U-net is learning how to predict the minimum margin m .
GPUs.
For that speciﬁc case, the training time is multiplied byfor fair comparison.
See Appendix A.6 for more detailed statistics.
*indicates the performance of the best performing large model trained with either random or guided patch.
Detailed performance of the large models can be found in Appendix A.
[Table]
5. E FFECTIVENESS AGAINST S PARSE A TTACK
The IBP based method can also be used to defend against sparse attack, see Section 4.3.
Its performance is reasonable compared to patch defense (e.g. 91.5% certiﬁed accuracy for 2 × 2 patch vs 90.8% for k=4), even though the sparse attack model is much stronger.
For convolutional networks, we increase the size of the ﬁrst convolutional layer (i.e. from 3 × 3 to 11 × 11 ) so the interval bounds calculated are tighter.
However, despite the change, fully-connected network still performs much better.
For example, the certiﬁed accuracy drops from 25.6% to 13.8% when we switch from fully-connected to convolutional network for CIFAR10 and drops from 90.8% to 75.9% for MNIST respectively.
Detailed results are shown in the Appendix A.4 Table 7.
Table 4 compares our approach with the state-of-the-art certiﬁed sparse defense (Random Ablation) Levine & Feizi (2019).
We use their best model with the largest medium radii to certify against various levels of sparsity.
As shown in the table, our method achieves higher certiﬁed accuracy on the MNIST dataset over all the sparse radii, but lower on CIFAR-10.
It is worth noting that we are using a much smaller and simpler model (a fully-connected network) compared to Random Ablation, which uses ResNet-18.
[Table]
5. T RANSFERABILITY TO PATCHES OF DIFFERENT SHAPES
Since real-world adversarial patches may not always be square, the robust transferability of the model to shapes other than the square is important.
Therefore
[Table]
we evaluate the robustness of the square-patch-trained model to adversarial patches of different shapes while ﬁxing the number of pixels.
In all these experiments, we evaluate the certiﬁed accuracy for our largest model, on both MNIST and CIFAR datasets.
We evaluate the transferability to various shapes including rectangle, line, parallelogram
[Table]
and diamond.
With the exception of rectangles, all the shapes have the exact same pixel count as the patches used for training.
For rectangles, we use multiple choices of width and length, obtaining some combinations with slightly more pixels
[Table]
and the worst accuracy is reported in Table 5.
The exact shapes used can be found in
Appendix A.2.
The certiﬁed accuracy of our models generalize surprisingly well to other shapes, losing no more than than 5% in most cases for MNIST and no more than 6% for CIFAR-10 (Table 5).
The largest degradation of accuracy happens for rectangles and lines, and it is mostly because the rectangle considered has more pixels compared to the square, and the line has less overlaps.
However, it is still interesting that the certiﬁcate even generalizes to a straight line, even though the model was never trained to be robust to lines.
In the case of MNIST with small patch size, the certiﬁed accuracy even improves when transferred to lines.
[Table]
C ONCLUSION AND F UTURE W ORK
After establishing the weakness of known defenses to patch attacks
[Table]
we proposed the ﬁrst certiﬁed defense against this model.
We demonstrated the effectiveness of our defense on two datasets
[Table]
and proposed strategies to speed up robust training.
Finally, we established the robust transferability of trained certiﬁed models to different shapes.
In its current form, the proposed certiﬁed defense is unlikely to scale to ImageNet
[Table]
and we hope the presented experiments will encourage further work along this direction.
A A PPENDIX A. E XPERIMENTAL S ETTINGS AND N ETWORK S TRUCTURE
We evaluate the proposed certiﬁed patch defense on three neural networks: a multilayer perceptron (MLP) with one 255-neuron hidden layer, and two convolutional neural networks (CNN) with different depths.
The small CNN has two convolutional layers (kernel size 4, stride 2) of 4 and 8 output channels each and a fully connected layer with 256 neurons.
The large CNN has four convolutional layers with kernel size (3, 4, 3, 4), stride (1, 2, 1, 2), output channels (4, 4, 8 ,8) and two fully connected layer with 256 neurons.
We run experiments on two datasets, MNIST and CIFAR10, with two different patch sizes 2 × 2 and 5 × 5.
For all experiments, we are using Adam (Kingma &
Ba
, 2014) with a learning rate of 5 e − 4 for MNIST and 1 e − 3 for CIFAR and with no weight decay.
We also adopt a warm-up schedule in all experiments like (Zhang et al., 2019b) where the input interval bounds start at zero and grow to [-1,1] after 61/121 epochs for MNIST/CIFAR respectively.
We train the models for a total of 100/200 epochs for MNIST/CIFAR10, where in the ﬁrst 61/121 epochs the learning rate is ﬁxed and in the following epochs, we reduce the learning rate by one half every 10 epochs.
In addition, following (Gowal et al., 2018), we further evaluate the CIFAR10 on a larger model which has 5 convolutional layers with kernel size 3 and a fully connected layer with 512 neurons.
This deeper and wider model achieves the clean accuracy around 89% , and has 17M parameters in total.
Table 8 in Appendix A.5 describes the full certiﬁed patch results for this large model
. A. S AMPLE SHAPES FOR GENERALIZATION EXPERIMENTS
We demonstrate generalization to other patch shapes that were not considered in training, obtaining surprisingly good transfer in robust accuracy; see the ﬁgure below and the results in Table 5.
A. B OUND POOLING
Besides random-patch certiﬁcate training and guided-patch certiﬁcate training, we also experimented with the idea of bound pooling.
All-patch training is very expensive as bounds generated by each potential patch has to be forward passed through the complete network.
Bound pooling partially relieves the problem be pooling the interval bounds in intermediate layers thus reducing the forward pass in subsequent layers.
Speciﬁcally, given a set of patches P , the interval bounds in the i th layer are ¯ Z ( i ) ( P ) =
{ ¯ z ( i ) ( p ) : p ∈
P } and Z ( i ) P
= { z ( i ) ( p ) : p ∈ P } .
We can reduce the number of interval bounds by partitioning P into n subsets { S 1 , ..., S n } and calculate a new set of bounds ¯ Z ( i ) pool ( P ) = { max p ∈ S
i ¯ z ( i ) ( p ) : i ∈ [ n ] } and Z ( i ) pool ( P ) =
{ min p ∈ S
i z ( i ) ( p ) : i ∈ [ n ] } .
Depending on how P is partitioned, the bound pooling would work differently.
In our experiments, we always select adjacent patches for each S i with the assumption that adjacent patches tend to generate similar bounds thus resulting in tighter certiﬁcate.
Bound pooling, similar to random- and guided- patch training, trades performance for efﬁciency compared to all-patch certiﬁcate training.
However, the trade off is not as favorable compared to random-patch and guided-patch training.
For example, in Table 6, Pooling 16 (4 × 4) patches in the ﬁrst layer reduces training time by 35% while loosing 0.7% in performance (on MNIST 2 × 2), but a similar level of performance can be achieved with guided-patch training with almost 90% reduction in training time.
The trade off becomes greater when the model becomes larger.
Overall, bound pooling is still quite expensive, and cannot scale to larger models like random-patch or guided-patch training.
the adjacent 4 × 4 patches (16 patches) are pooled together in the ﬁrst layer.
Pool 2-2 means that the adjacent 2 × 2 bounds are pooled together in the ﬁrst layer and then another 2 × 2 bound pooling happens at the second layer.
This is similar to 4 × 4 pooling except the pooling operation is distributed between the ﬁrst and second layer.
All experiments are performed on a 4-layer convolutional network.
[Table]
A. M ULTI - PATCH S PARSE T RAINING
Here we list the detailed certiﬁed accuracy for various sparsity levels and model architectures.
[Table]
A. T RAINING WITH LARGER MODELS
Recall that all-patch training considers all possible patches during training
[Table]
which can be too expensive for larger models and/or images.
The proposed random- and guided-patch training methods aim to reduce the training cost by considering only a subset of patches; please see Section 4.2 for more details.
[Table]
A. D ETAILED S TATISTICS ON T RAINING S TRATEGIES
Here we list the detailed statistics for each training strategies for Table
[Table] [Table]