C ERTIFIED D EFENCES A GAINST A DVERSARIAL P ATCH A TTACKS ON S EMANTIC S EGMENTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Adversarial patch attacks are an emerging security threat for real world deep learning applications.
We present D EMASKED S MOOTHING , the ﬁrst approach (up to our knowledge) to certify the robustness of semantic segmentation models against this threat model.
Previous work on certiﬁably defending against patch attacks has mostly focused on image classiﬁcation task and often required changes in the model architecture and additional training which is undesirable and computationally expensive.
In D EMASKED S MOOTHING , any segmentation model can be applied without particular training, ﬁne-tuning, or restriction of the architecture.
Using different masking strategies, D EMASKED S MOOTHING can be applied both for certiﬁed detection and certiﬁed recovery.
In extensive experiments we show that
D EMASKED S MOOTHING can on average certify 63% of the pixel predictions for a 1% patch in the detection task and 46% against a 0.5% patch for the recovery task on the ADE20K dataset.
I NTRODUCTION
Physically realizable adversarial attacks are a threat for safety-critical (semi-)autonomous systems such as self-driving cars or robots.
Adversarial patches (Brown et al., 2017; Karmon et al., 2018) are the most prominent example of such an attack.
Their realizability has been demonstrated repeatedly, for instance by Lee & Kolter (2019): an attacker places a printed version of an adversarial patch in the physical world to fool a deep learning system.
While empirical defenses (Hayes, 2018; Naseer et al., 2019; Selvaraju et al., 2019; Wu et al., 2020) may offer robustness against known attacks, it does not provide any guarantees against unknown future attacks (Chiang et al., 2020).
Thus, certiﬁed defenses for the patch threat model, which allow guaranteed robustness against all possible attacks for the given threat model, are crucial for safety-critical applications.
Research on certiﬁable defenses against adversarial patches can be broadly categorized into certiﬁed recovery and certiﬁed detection.
Certiﬁed recovery (Chiang et al., 2020; Levine & Feizi, 2020; Zhang et al., 2020; Xiang et al., 2021; Metzen & Yatsura, 2021; Lin et al., 2021;
Xiang et al., 2022a; Salman et al., 2021; Chen et al., 2022) has the objective to make a correct prediction on an input even in the presence of an adversarial patch.
In contrast, certiﬁed detection (McCoyd et al., 2020; Xiang & Mittal, 2021b; Han et al., 2021; Huang & Li, 2021) provides a weaker guarantee by only aiming at detecting inputs containing adversarial patches.
While certiﬁed recovery is more desirable in principle, it typically comes at a high cost of reduced performance on clean data.
In practice, certiﬁed detection might be preferable because it allows maintaining high clean performance.
Most existing certiﬁable defenses against patches are focused on image classiﬁcation, with the exception of DetectorGuard (Xiang & Mittal, 2021a) and ObjectSeeker
(Xiang et al., 2022b) that certiﬁably defend against patch hiding attacks on object detectors.
Moreover, existing defences are not easily applicable to arbitrary downstream models, because they assume either that the downstream model is trained explicitly for being certiﬁably robust (Levine & Feizi, 2020; Metzen & Yatsura, 2021), or that the model has a certain network architecture such as BagNet (Zhang et al., 2020;
Metzen & Yatsura, 2021; Xiang et al., 2021) or a vision transformer (Salman et al., 2021; Huang & Li, 2021).
A notable exception is
PatchCleanser (Xiang et al., 2022a), which can be combined with arbitrary downstream models but is restricted to image classiﬁcation.
Adversarial patch attacks were also proposed for the image segmentation problem (Nesti et al., 2022), mostly for attacking CNN-based models that use a localized receptive ﬁeld (Zhao et al., 2017).
[Figure]
However, recently self-attention based vision transformers (Dosovitskiy et al., 2021) have achieved new state-of-the-art in the image segmentation task (Liu et al., 2021; Bousselham et al., 2021).
Their output may become more vulnerable to adversarial patches if they manage to manipulate the global self-attention (Lovisotto et al., 2022).
We demonstrate how signiﬁcant parts of the segmentation output may be affected by a small patch for Swin tranfromer Liu et al.
(2021) in Figure 1a.
Full details on the attack are available in Appendix D. We point out that preventive certiﬁed defences are important because newly developed attacks can immediately be used to compromise safety-critical applications unless they are properly defended.
In this work, we propose the novel framework D EMASKED S MOOTHING (Figure 1c) to obtain the ﬁrst (to the best of our knowledge) certiﬁed defences against patch attacks on semantic segmentation models.
Similarly to previous work (Levine & Feizi, 2020), we mask different parts of the input (Figure 1b) and provide guarantees with respect to every possible patch that is not larger than a certain pre-deﬁned size.
While prior work required the classiﬁcation model to deal with such masked inputs, we leverage recent progress in image inpainting (Dong et al., 2022) to reconstruct the input before passing it to the downstream model.
This decoupling of image demasking from the segmentation task allows us to support arbitrary downstream models.
Moreover, we can leverage state of the art methods for image inpainting.
We also propose different masking schemes tailored for the segmentation task that provide the dense input allowing the demasking model to understand the scene but still satisfy the guarantees with respect to the adversarial patch.
We summarize our contributions as follows:
• We propose D EMASKED S MOOTHING which is the ﬁrst (to the best of our knowledge) certiﬁed recovery or certiﬁed detection based defence against adversarial patch attacks on semantic segmentation models (Section 4).
• D EMASKED S MOOTHING can do certiﬁed detection and recovery with any off-the-shelf segmentation model without requiring ﬁnetuning or any other adaptation.
• We implement D EMASKED S MOOTHING , evaluate it for different certiﬁcation objectives and masking schemes (Section 5).
We can certify 63% of all pixels in certiﬁed detection for a 1% patch and 46% in certiﬁed recovery for a 0.5% patch for the BEiT-B (Bao et al., 2022) segmentation model on the ADE20K Zhou et al.
(2017) dataset.
R ELATED W ORK
Certiﬁed recovery.
The ﬁrst certiﬁed recovery defence for classiﬁcation models against patches was proposed by Chiang et al.
(2020), who adapted interval-bound propagation (Gowal et al., 2019) to the patch threat model.
Levine & Feizi (2020) proposed De-Randomized Smoothing (DRS), which provides signiﬁcant accuracy improvement when compared to Chiang et al.
and scales to the ImageNet dataset.
In DRS, a base classiﬁer is trained on images where everything but a small local region is masked (ablated).
At inference time, a majority vote of all speciﬁed ablations is taken as the ﬁnal classiﬁcation.
If this vote has a large enough margin to the runner-up class, the prediction cannot be shifted by any patch that does not exceed a pre-deﬁned size.
A similar approach was adopted in Randomized Cropping (Lin et al., 2021).
A general drawback of these approaches is that the classiﬁer needs to be trained to process masked/cropped inputs, which (in contrast to our work) prohibits the usage of arbitrary pretrained models.
A further line of work studies network architectures that are particularly suited for certiﬁed recovery.
For instance, models with small receptive ﬁelds such as BagNets (Brendel & Bethge, 2019) have been explored, either by combining them with some ﬁxed postprocessing (Zhang et al., 2020; Xiang et al., 2021) or by training them end-to-end for certiﬁed recovery (Metzen & Yatsura, 2021).
Salman et al.
(2021) propose to apply DRS to Vision Transfomers (ViTs) .
In contrast to the aforementioned works, our
Demasked Smoothing can be applied to models with arbitrary architecture.
This is a property shared with PatchCleanser (Xiang et al., 2022a), which however is limited to image classiﬁcation and it is not clear how it can be extended to semantic segmentation where a class needs to be assigned to every pixel including the masked ones.
Certiﬁed recovery against patches has also been extended to object detection, speciﬁcally to defend against patch hiding attacks.
Two notable works in this direction are
DetectorGuard (Xiang & Mittal, 2021b), an extension of PatchGuard (Xiang et al., 2021) to object detection, and ObjectSeeker (Xiang et al., 2022b).
Randomized smoothing (Cohen et al., 2019) has been applied to certify semantic segmentation models against (cid:96) 2 -norm bounded adversarial attacks (Fischer et al., 2021).
However, to the best of our knowledge, no certiﬁed defence against patch attacks for semantic segmentation has been proposed so far.
Certiﬁed detection.
An alternative to certiﬁed recovery is certiﬁed detection .
Here, an adversarial patch is allowed to change the model prediction.
However, if it succeeds in doing so, there is a mechanism that detects this attack certiﬁably with zero false negatives.
Minority Reports (McCoyd et al., 2020) was the ﬁrst certiﬁed detection method against patches, which is based on sliding a mask over the input in a way that ensures that there will be one mask position that completely hides the patch.
PatchGuard++ (Xiang & Mittal, 2021b) is an extension of Minority Reports where the sliding mask is not applied on the input but on the feature maps of a BagNet-type feature extractor.
This reduces inference time drastically since the feature extractor needs to be executed only once per input.
ScaleCert (Han et al., 2021) tries to identify “superﬁcial important neurons”, which allows pruning the network in a way that the prediction needs to be made for fewer masked inputs.
Lastly, PatchVeto (Huang & Li, 2021) is a recently proposed method for certiﬁed detection that is tailored towards
ViT models.
It implements masking by removing certain input patches of the ViT.
In this work, we propose a novel method for certiﬁed detection in the semantic segmentation task that can be used for any pretrained model.
Image reconstruction.
The problem of learning to reconstruct the full image from inputs where parts have been masked out was pioneered by Vincent et al.
It recently attracted attention as proxy task for self-supervised pre-training, especially for the ViTs (Bao et al., 2022;
He et al., 2021).
Recent approaches to this problem are using Fourier convolutions (Suvorov et al., 2022) and visual transformers (Dong et al., 2022).
SPG-Net (Song et al., 2018) trains a subnetwork to reconstruct the full semantic segmentation directly from the masked input as a part of the image inpainting pipeline.
In this work, we use the state-of-the-art ZITS (Dong et al., 2022) inpainting method.
P ROBLEM S ETUP 3. S EMANTIC S EGMENTATION
In this work, we focus on the semantic segmentation task.
Let X be a set of rectangular images.
Let x ∈ X be an image with height H , width W and the number of channels C .
We denote Y to be a ﬁnite label set.
The goal is to ﬁnd the segmentation map s ∈ Y H
× W for x .
For each pixel x i,j , the corresponding label s i,j denotes the class of the object to which x i,j belongs.
We denote S to be a set of segmentation maps and f : X → S to be a segmentation model.
3. T HREAT MODEL
Let us consider an untargeted adversarial patch attack on a segmentation model.
Consider an image x ∈ [0 , 1] H × W × C and its ground truth segmentation map s .
Assume that the attacker can modify an arbitrary rectangular region of the image x which has a size of H (cid:48)
× W (cid:48) .
We refer to this modiﬁcation as a patch .
Let l ∈ { 0 , 1 } H × W be a binary mask that deﬁnes the patch location in the image in which ones denote the pixels belonging to the patch.
Let L be a set of all possible patch locations for a given image x .
Let p ∈
[0 , 1] H × W × C be the modiﬁcation itself.
Then we deﬁne an operator A as A ( x, p, l ) =
(1 − l ) (cid:12) x + l (cid:12) p , where (cid:12) is element-wise product.
The operator A applies the H (cid:48)
× W (cid:48) subregion of p deﬁned by a binary mask l to the image x while keeping the rest of the image unchanged.
We denote P :=
[0 , 1] H × W × C × L to be a set of all possible patch conﬁgurations ( p, l ) that deﬁne an H (cid:48)
× W (cid:48) patch.
Let s ∈ S be the ground truth segmentation for x . Let Q ( f ( x ) , s ) be some quality metric such as global pixel accuracy or mean intersection over union (mIoU). The goal of an attacker is to
[Equation]
D [Equation] OBJECTIVE
In this paper, we propose certiﬁed defences against patch attacks.
It means that we certify against any possible attack from P including ( p (cid:63) , l (cid:63) ) .
We consider two robustness objectives.
Certiﬁed recovery For a pixel x i,j our goal is to verify that the following statement is true
[Equation]
Certiﬁed detection We consider a veriﬁcation function v deﬁned on X such that v ( x ) ∈ { 0 , 1 } H × W .
If v ( x ) i,j = 1 , then the adversarial patch attack on x i,j can be detected by applying the function v to the attacked image x (cid:48)
=
[Equation] x, p, [Equation]
[Equation]
v ( x (cid:48) )
i,j = 0 means an alert on pixel x (cid:48)
i,j . However, if x (cid:48) is not an adversarial example, then this is a false alert. In that case the fraction of pixels for which we return false alert is called false alert ratio (FAR). The secondary objective is to keep FAR as small as possible.
Depending on the objective our goal is to certify one of the conditions 1, 2 for each pixel x i,j .
This provides us an upper bound on an attacker’s effectiveness under any adversarial patch attack from P .
D EMASKED S MOOTHING
D EMASKED S MOOTHING (Figure 1c) consists of several steps.
First, we apply a predeﬁned set of masks with speciﬁc properties to the input image to obtain a set of masked images.
Then we reconstruct the masked regions of each image based on the available information with an inpainting model g .
After that we apply a segmentation model f to the demasked results.
Finally, we aggregate the segmentation outcomes and make a conclusion for the original image with respect to the statements (1) or (2).
4. I NPUT MASKING
Motivation.
Like in previous work (Section 2) we apply masking patterns to the input image and use predictions on masked images to aggregate the robust result.
If an adversarial patch is completely masked, it has no effect on further processing.
However, in semantic segmentation, we predict not a single whole-image label like in the classiﬁcation task, but a separate label for each pixel.
Thus, making prediction on a masked image must allow us to predict the labels also for the masked pixels.
Preliminaries.
Consider an image x ∈ [0 , 1] H × W × C .
We deﬁne " ∗ " to be a special masking symbol that does not correspond to any pixel value and has the property ∀ z ∈ R : z ×∗ = ∗ .
Please note that ∗ needs to be different from 0 since 0 is a valid pixel value in unmasked inputs.
Let m ∈ {∗ , 1 } H × W be a mask .
We call the element-wise product x (cid:12) m a masking of x . In a masking, a subset of [Figure] pixels becomes ∗ and the rest remains unchanged.
We consider the threat model P with patches of size H (cid:48) × W (cid:48) (Section 3.2).
To deﬁne the structure of our masks, we break m into an array B of non-intersecting blocks, each having the same size H (cid:48)
× W (cid:48) as the adversarial patch.
We index the blocks as B [ q, r ] , 1 ≤ q ≤ (cid:100) HH (cid:48) (cid:101) , 1 ≤ r ≤ (cid:100) WW (cid:48) (cid:101) .
We say that the block B [ q, r ] is visible in a mask m if ∀ ( i, j ) ∈ B [ q, r ] : m i,j = 1 Consider an array M of K masks.
We deﬁne each mask M [ k ] by a set of blocks that are visible in it.
For certiﬁed recovery, each block is visible in exactly one mask and masked in the others. We say that a mask m is affected by a patch ( p, l ) if A ( x, p, l ) (cid:12) m (cid:54) =
x (cid:12) m .
We deﬁne T ( M ) = [Equation]
p, [Equation] ∈ [Equation]
( x, p, l ) (cid:12) m (cid:54) =
x (cid:12) m }| .
That is: T ( M ) is the largest number of masks affected by some patch.
If M is deﬁned, we refer to the value T ( M ) as T for simplicity.
Certiﬁed recovery.
We deﬁne column masking M for which T = 2 .
We assign every k -th block column to be visible in the mask M [ k ] (Figure 2c).
Any ( p, l ) ∈ P can intersect at most two adjacent columns since ( p, l ) has the same width as a column.
Thus, it can affect at most two masks (Figure 2b).
A similar scheme can be proposed for the rows.
Due to the block size in B , the patch ( p, l ) cannot intersect more than four blocks at once.
We deﬁne a mask set that we call 3-mask s. t. for any four adjacent blocks two are visible in the same mask (Figures 2d).
Hence, a patch for 3-mask can affect no more than 3 masks, T = 3 . To achieve T =
any assignment of visible blocks to the masks works.
We consider 4-mask that allows uniform coverage of the visible blocks in the image (Figure 2f).
See details on masking schemes in Appendix B.
Certiﬁed detection.
We deﬁne M d to be a set of masks for certiﬁed detection (we use subscript d for distinction). M d should have the property: ∀ ( p, l ) ∈ P ∃ m ∈ M d : A ( x, p, l ) (cid:12) m =
x (cid:12) m i. e. for every patch exists at least one mask not affected by this patch.
For a patch of size H (cid:48) × W (cid:48)
we consider K = W − W (cid:48)
+ 1 masks such that the mask M d [ k ] masks a column of width W (cid:48) starting at the horizontal position k in the image (Figure 2h).
To obtain the guarantee for the same P with a smaller K , we consider a set of strided columns of width W (cid:48)(cid:48) ≥ W (cid:48) and stride W (cid:48)(cid:48) − W (cid:48) + 1 that also satisfy the condition (see the proof adapted from Xiang et al.
(2022a) in Appendix A).
A similar scheme can be proposed for the rows (Figure 2i).
Alternatively, we could use a set of block masks of size H (cid:48) × W (cid:48) .
Then the number of masks grows quadratically with the image resolution.
Hence, in the experiments we focus on the column and the row masking schemes.
Let g be a demasking model, g ( x (cid:12) m ) ∈ [0 , 1] H × W × C .
The goal of g is to make the reconstruction g ( x (cid:12) m ) as close as possible (in some metric) to the original image x .
For a segmentation model f we deﬁne a segmentation array S ( M, x, g, f ) , S [ k ] :
= f ( g ( x (cid:12) M [ k ])) , 1 ≤ k ≤ K .
4. C ERTIFICATION
Certiﬁed recovery.
For the threat model P consider a set M of K masks.
We deﬁne a function h : X → S that assigns a class to the pixel x i,j via majority voting over class predictions of each reconstructed segmentation in S .
A class for the pixel that is predicted by the largest number of segmentations is assigned.
We break the ties by assigning a class with a smaller index.
Theorem 1.
If the number of masks K satisﬁes K ≥ 2 T ( M ) + 1 and for a pixel x i,j we have
[Equation]
(i.e. all the votes agree), then ∀ ( p, l
) ∈ P : h ( A ( x, p, l ))
i,j = h ( x ) i,j .
See the proof in Appendix A.
Certiﬁed detection.
Consider M d = { M d [ k ] } Kk =1 .
For a set of demasked segmentations S we deﬁne the veriﬁcation map v ( x ) i,j :=
[ f ( x ) i,j = S [1] i,j = . . .
= S
[ K ] i,j ] i.e. the original segmentation is equal to all the other segmentations on masked-demasked inputs, including the one in which the potential patch was completely masked.
Theorem 2.
Assume that v ( x ) i,j = 1 .
Then
[Equation]
See the proof in Appendix A. For a given image x the veriﬁcation map v ( x ) is complementary to the model segmentation output f ( x ) that stays unchanged.
Thus, there is no drop in clean performance however we may have some false positive alerts in the veriﬁcation map v in the clean setting. We present the Demasked Smoothing procedure in Algorithm 1.
E XPERIMENTS
In this section, we evaluate D EMASKED S MOOTHING with the masking schemes proposed in Section 4, compare our approach with the direct application of Derandomized Smoothing Levine & Feizi (2020) to the segmentation task and evaluate the performance on different datasets and models.
Certiﬁed recovery and certiﬁed detection provide certiﬁcates of different strength (Section 4) which are not comparable.
We evaluate them separately for different patch sizes.
5. E XPERIMENTAL S ETUP
We evaluate D EMASKED S MOOTHING on two challenging semantic segmentation datasets: ADE20K (Zhou et al., 2017) (150 classes, 2000 validation images) and COCO-Stuff-10K (Caesar et al., 2018) (171 classes, 1000 validation images).
For demasking we use the ZITS Dong et al.
(2022) inpainting model with the checkpoint provided in the ofﬁcial paper repository 1 .
The model was trained on {+Places
+} (Zhou et al., 2016) dataset with images resized to 256 × 256.
As a segmentation model f we use BEiT-B Bao et al.
, Swin Liu et al.
PSPNet Zhao et al.
(2017) and DeepLab v (Chen et al., 2018).
We note that the ﬁrst two models are based on transformers and obtain near stateof-the-art results.
PSPNet and DeepLab v3 are CNN-based segmentation methods that we consider to demonstrate that D EMASKED S MOOTHING is not speciﬁc to transformer-based architectures.
We https [Figure] use the model implementations provided in the mmsegmentation framework Contributors (2020).
An illustration of the image reconstruction and respective segmentation can be found in Figure 3.
We run the evaluation in parallel on 5 Nvidia Tesla V100-32GB GPUs.
The certiﬁcation for the whole ADE20K validation set with ZITS and BEiT-B takes around 1.2 hours for certiﬁed recovery andhours for certiﬁed detection (due to a larger number of masks).
5. E VALUATION METRICS
For both certiﬁed recovery and certiﬁed detection, we generate a standard segmentation output (without any abstention) and a corresponding certiﬁcation map (Figure 4).
In case of certiﬁed detection, the segmentation output remains the same as for the original segmentation model, however, there may be false alerts in the certiﬁcaton map.
For the certiﬁed recovery, the output is obtained by a majority vote over the segmentations of demasked images (Section 4.2).
We evaluate the mean intersection over union (mIoU) for these outputs.
The certiﬁcation map is obtained by assigning to each certiﬁed pixel the corresponding class from the segmentation output and assigning a special uncertiﬁed label to all non-certiﬁed pixels.
For each image we evaluate the fraction of pixels which are certiﬁed and correct (coincide with the ground truth).
%C is a mean of these fractions over all the images in the dataset.
In semantic segmentation task, the class frequencies are usually skewed, therefore global pixel-wise accuracy alone is an insufﬁcient metric.
Matching the certiﬁcation map separately for each class y ∈ Y with the ground truth segmentation for y in the image x allows us to compute the guaranteed lower bound ( cTP y ( x ) ) on the number of true positive pixel predictions ( TP y ( x ) )
i.e. those that were correctly classiﬁed into y .
If a pixel was certiﬁed with a correct class, then this prediction cannot be changed by a patch (or, alternatively, the change will be detected by the veriﬁcation function v in certiﬁed detection).
We consider recall R y ( x ) = TP y ( x ) TP y ( x )+ FN y ( x ) where FN y ( x ) is the number of false negative predictions for y in x .
P y ( x ) = TP y ( x ) + FN y ( x ) is the total area of y in the ground truth and does not depend on our prediction.
We can evaluate certiﬁed recall cR y ( x ) = cTP y ( x ) P y ( x ) , a lower bound on the recall R y ( x ) .
Total recall and certiﬁed total recall of class y in a dataset D are TR y ( D ) = (cid:80) x ∈ D TP y ( x ) (cid:80) x ∈ D P y ( x ) and cTR y ( D ) = (cid:80) x
[Equation]
D cTP y ( x ) (cid:80) x ∈ D P y
x ) respectively.
Then, we obtain mean recall mR
( D ) = 1 |Y| (cid:80) y ∈Y TR y ( D ) and certiﬁed mean recall cmR ( D ) = 1 |Y| (cid:80) y ∈Y cTR y ( D ) . Evaluating lower bounds on other popular metrics such as mean precision or mIoU this way results in vacuous upper bound since they depend on the upper bound on false positive ( FP ) predictions.
For the pixels that are not certiﬁed we cannot guarantee that they will not be assigned to a certain class, therefore, a non-trivial upper bound on FP is not straightforward.
We leave this direction for future work.
In certiﬁed detection, we additionally consider false alert ratio (FAR) which is the fraction of correctly classiﬁed pixels for which we return an alert on a clean image.
Smaller FAR is preferable.
[Figure] [Table] Due to our threat model, certifying small objects in the scene can be difﬁcult because they can be partially or completely covered by an adversarial patch in a way that there is not chance to recover the prediction.
To provide an additional perspective on our methods, we also evaluate mR and cmR speciﬁcally for the “big” classes, which occupy on average more than 20% of the images in which they appear.
These are, for example, road, building, train, and sky, which are important for understanding the scene.
The full list of such classes for each dataset is provided in the Appendix H.
5. D ISCUSSION
In Table 1, we compare different masking schemes proposed in Section 4.1. Evaluation of all the models with all the masking schemes is consistent with these results and can be found in Appendix E. We see that column masking achieves better results in both certiﬁcation modes.
Effectiveness of column masking for classiﬁcation task was also empirically observed by Levine & Feizi (2020).
We attribute the effectiveness of column masking to the fact most of the images in the datasets have a clear horizont line, therefore having a visible column provides a slice of the image that intersects most of the scene background objects. We use Swin Liu et al. (2021) on 200 ADE20K images. See details and illustrations in Appendix G.
[Table] [Table] [Figure] In Table 2, we extend Derandomized Smoothing (DRS) proposed by Levine & Feizi (2020) for certiﬁed recovery in classiﬁcation to the segmentation task and compare it to our method Direct adaptation requires training a model that is able to predict the full image segmentation from a small visible region.
Since it is not clear what architectural design and training procedure would be needed for that, we consider two alternative baselines.
DRS-S predicts the segmentation directly from the masked image and DRS-E uses our inpainting method to ﬁrst reconstruct the image and then obtain the segmentation.
See the implementation details in Appendix G. DRS with column smoothing performs poorly on the segmentation task, which emphasizes the need for speciﬁc masking schemes.
In Table 3, we evaluate our method with column masking on different models. For certiﬁed detection we can certify more than 60% of the pixels with all models on ADE20K and more than 46 % on COCO10K. False alert ratio on correctly classiﬁed pixels is around 20%.
In certiﬁed recovery, we certify more than 44% pixels on ADE20K and more than 28% pixels on COCO10K. Figure 5 shows how the performance of D EMASKED S MOOTHING depends on the patch size for the BEiT-B model.
We see that certiﬁed detection metrics remain high even for a patch as big as 5% of the image surface and for the recovery they slowly deteriorate as we increase the patch size to 2%.
Ablations with respect to inpainting can be found in Appendix F. D
EMASKED S MOOTHING illustrations procedure are provided in Appendix K.
C ONCLUSION
In this work, we propose D EMASKED S MOOTHING , the ﬁrst (up to our knowledge) certiﬁed defence framework against patch attacks on segmentation models.
Due to its novel design based on masking schemes and image demasking, D EMASKED S MOOTHING is compatible with any segmentation model and can on average certify 63% of the pixel predictions for a 1% patch in the detection task and 46% against a 0.5% patch for the recovery task on the ADE20K dataset.  E THICS S TATEMENT
This work contributes to the ﬁeld of certiﬁed defences against physically-realizable adversarial attacks.
The proposed approach allows to certify robustness of safety-critical applications such as medical imaging or autonomous driving.
The defence might be used to improve robustness of systems used for malicious purposes such as (semi-)autonomous weaponry or unauthorized surveillance.
This danger may be mitigated e.g. by using a system of sparsely distributed patches which makes certifying the image more challenging. All activities in our organization are carbon neutral, so the experiments performed on our GPUs do not leave any carbon dioxide footprint.  R EPRODUCIBILITY S TATEMENT We provide the details of our experimental setup in Section 5.1.
We discuss the evaluation metrics and their computation in Section 5.2.
A P ROOFS (S ECTION 4)
In this section, we provide the proofs for the theorems stated in Section 4.
Lemma 1.
(Section 4.1) Consider an image of the size H × W .
Let H (cid:48) × W (cid:48) be a ﬁxed adversarial patch size.
Let M d ( K, L ) be a set of masks where each mask is masking an H × W (cid:48)(cid:48) vertical column,
W (cid:48)(cid:48) ≥ W (cid:48) .
Let the stride between the columns in two adjacent masks be W (cid:48)(cid:48)
− W (cid:48)
+ 1 .
Then for any location l ∈ L of the patch, there exists a mask that covers it completely.
Proof.
(Adapted from the proof of Lemma 4 in PatchCleanser Xiang et al.
(2022a)).
Without loss of generality, we consider the ﬁrst two adjacent column masks.
The ﬁrst one covers the columns from 1 to W (cid:48)(cid:48) .
The second mask covers the columns from 1 + ( W (cid:48)(cid:48) − W (cid:48)
+ 1)
=
W (cid:48)(cid:48) − W (cid:48) +to ( W (cid:48)(cid:48) − W (cid:48) + 2) + ( W (cid:48)(cid:48) − 1) =
2 W (cid:48)(cid:48) − W (cid:48)
+ 1 (See Figure 6).
Now consider an adversarial patch of size H (cid:48)
× W (cid:48) .
Let us ﬁnd the smallest possible start index of this patch so that it does not get covered by the ﬁrst mask.
For that it should be visible at the column W (cid:48)(cid:48) + 1 and, therefore, start at the column with index not smaller than ( W (cid:48)(cid:48)
+ 1) − W (cid:48)
+ 1 = W (cid:48)(cid:48) − W (cid:48)
+ 2 .
However, it is the same column in which second mask starts.
Therefore, given that W (cid:48)(cid:48) ≥ W (cid:48) we have that the patch is completely masked by the second mask.
Then for a patch which is only partially masked by the second mask from the left we use an analogous argument to show that it is completely masked by the third mask and so on.
Certiﬁed recovery.
For the threat model P (Section 3) consider a set M of K masks.
We deﬁne a function h : X → S that assigns a class to the pixel x i,j via majority voting over class predictions of each reconstructed segmentation in S .
A class for the pixel that is predicted by the largest number of segmentations is assigned.
We break the ties by assigning a class with a smaller index.
Theorem 1 .
(Section 4.2)
If the number of masks K satisﬁes K ≥ 2 T ( M ) + 1 and for a pixel x i,j we have
[Equation]
i.e. all the votes agree, then ∀ ( p, l
) ∈ P : h ( A ( x, p, l ))
i,j = h ( x ) i,j .
Proof.
Assume that
[Equation]
Let us denote x (cid:48) :
= A ( x, p, l ) and S (cid:48) to be the segmentation array for x (cid:48) .
Then the class h ( x ) i,j did not get the majority vote for S (cid:48) .
However, by deﬁnition of T ( M ) we know that ( p, l ) could affect at most T ( M ) segmentations.
Since all K segmentations of S have voted for h ( x ) i,j , then at least
K − T > K 2 of them are still voting for h ( x ) i,j in S (cid:48)
meaning that h ( x ) i,j still has the majority vote in S (cid:48) .
Therefore h ( x (cid:48) )
i,j = h ( x ) i,j
Certiﬁed detection.
Consider M d = { M d [ k ] } Kk =1 .
For a set of demasked segmentations S we deﬁne the veriﬁcation map v ( x ) i,j :=
[ f ( x ) i,j = S [1] i,j = . . .
= S
[ K ] i,j ] i.e. the original segmentation coincides with all the other segmentations including the one in which the potential patch was completely masked.
Theorem 2 .
(Section 4.2) Assume that v ( x ) i,j = 1 .
Then ∀ ( p, l ) ∈ P : v ( A ( x, p, l ))
i,j = 1 ⇒ f
( A ( x, p, l ))
i,j = f ( x ) i,j
Proof.
Assume that ∃ ( p, l ) ∈ P s. t. v ( A ( x, p, l )) i,j = 1 and f ( A ( x, p, l ))
i,j (cid:54) =
f ( x ) i,j .Let us denote x (cid:48) :
= A ( x, p, l ) and S (cid:48) to be the segmentation set for x (cid:48) .
By deﬁnition of M d , ∃ M d [ k ] ∈ M d s. t. M d [ k ] masks the patch ( p, l ) Hence,
[Equation]
M
[Equation]
k ]
[Equation]
= S
[Equation]
[ k ] ,
Since v ( x ) i,j = 1 , we have f ( x ) i,j = S [ k ] i,j .
Since v ( x (cid:48) )
i,j = 1 , we have f ( x (cid:48) )
i,j = S (cid:48)
[ k ] i,j .
Thus, f ( x (cid:48) ) i,j = f ( x ) i,j .
B D ETAILED DESCRIPTION OF MASKING STRATEGIES
In this section, we provide additional details for constructing certiﬁed recovery masks proposed in
Section 4.1.
We deﬁne mask sets M that satisfy different values of T .
We divide the image x into a set of non-intersecting blocks B of the same size as an adversarial patch, H (cid:48)
× W (cid:48)
(see Figure 7), 1 ≤ q ≤ (cid:100) H/H (cid:48) (cid:101) , 1 ≤ r ≤ (cid:100) W/W (cid:48) (cid:101) .
In each mask, each of these blocks will be either masked or not masked (i. e. visible ).
Moreover, for each block there exists only one mask in which it is visible.
For a set M of K masks we deﬁne the mapping µ M : B → { 1 , . . .
, K } .
If µ ( B [ q, r ]) =
k , then B [ q, r ] is not masked in M [ k ] .
Therefore, each mask M [ k ] is deﬁned by a B k ⊂ B s. t. for b ∈ B k µ ( b ) = k .
We deﬁne a set M that we call 3-mask for which T ( M )
= 3 .
We assign the blocks in each row to the masks as follows: µ ( B [1 , 1]) = 1 ; µ ( B [1 , 2]) = µ ( B [1 , 3]) = 2 ; µ ( B [1 , 4]) =
µ ( B [1 , 5]) =
and so on until we reach the end of the row.
If we ﬁnish the ﬁrst row with the value k , then we start the second row as follows µ ( B [2 , 1])
=
µ ( B [2 , 2]) =
k + 1 ; µ ( B [2 , 3]) = µ ( B [2 , 4]) =
k
+ 2 : . . .
If we ﬁnish the second row on n , we start the third row similarly to the ﬁrst: µ ( B [3 , 1]) = n + 1 ; µ ( B [3 , 2]) = µ ( B [3 , 3]) = n + 2 ; . .
When we reach the number K , we start from 1 again (Figure 7d).
Due to the block size, the patch cannot intersect more than four blocks at once.
Our parity-alternating block sequence ensures that in any such intersection of four blocks either the top ones or the bottom ones will belong to the same masking, so at most three different maskings can be affected.
We deﬁne a set M that we call 4-mask for which T ( M )
= 4 .
Due to our block size any assignment of masks will work because the patch cannot intersect more than four blocks.
We consider the one that allows uniform distribution of the unmasked blocks (Figure 7g).
We point out that for the described methods each masking keeps approximately 1 /K of the pixels visible and the unmasked regions are uniformly distributed in the image.
This means that for any masked pixel there exists an unmasked region located close enough to this pixel.
It is the core difference between our masks and the ones proposed for certiﬁed classiﬁcation such as block or column smoothing Levine & Feizi (2020).
It was observed that the image demasking is facilitated when the visible regions are uniformly spread in the masked image He et al.
C T EST - TIME INPUT CERTIFICATION
In this section, we discuss how certiﬁed recovery (Theorem 1) can be applied to guaranteed veriﬁcation of the robustness on a test image.
We also discuss how robustness guarantees for the test-time images can be evaluated by using a dataset of clean images such as ADE20K (Zhou et al., 2017) or
COCOStuff-10K (Caesar et al., 2018).
C. T EST - TIME CERTIFIED RECOVERY
Let x (cid:48) be a test-time input which can be either a clean image or an image attacked with an adversarial patch.
We know that there exists a clean image x corresponding to x (cid:48) which removes the patch if it is present.
We have either x (cid:48)
= x or x (cid:48) ∈
A ( x ) , where A ( x ) :=
{ A ( x, p, l ) | ( p, l ) ∈ P} .
However, at test time we do not have access to the clean image x .
[Figure] Our goal is to certify that for our segmentation model h and a pixel x i,j we have h ( x (cid:48) )
i,j = h ( x ) i,j .
We can achieve this result by applying the recovery certiﬁcation (Theorem 1) to the test-time image.
It allows us to verify whether ∀ ( p, l ) ∈ P : h ( A ( x (cid:48) , p, l ))
i,j = h ( x (cid:48) )
i,j .
We also know that if x (cid:48) ∈
A ( x ) , then x ∈ A ( x (cid:48) )
(Figure 8a).
Indeed, if x (cid:48) is only different from x by one patch, then x can be be obtained from x (cid:48) by removing this patch.
Therefore, by obtaining the guarantee for A ( x (cid:48) ) , we implicitly obtain the guarantee also for the image x even though we do not have direct access to it.
We note that this test-time guarantee is only possible for certiﬁed recovery.
In certiﬁed detection, we would need to evaluate the veriﬁcation function v (Theorem 2) for both the clean image x and the attacked image x (cid:48) to obtain the result.
This cannot be done if x is implicit.
C. R OBUSTNESS GUARANTEES EVALUATION
The typical certiﬁed robust error for a given test data set (and pixel ( i, j ) in the segmentation case) is an estimate for
[Equation]
where D is the data generating probability measure and we assume that our test set to be an i.i.d.
sample of it.
This is the expected robust error (worst case over our threat model P for clean inputs) for a given pixel ( i, j ) .
Using the test sample to get an estimate of this quantity, we get a probabilistic guarantee that the corresponding pixel ( i, j ) of a new clean test sample x (cid:48) drawn i.i.d.
from P will have its whole “patch”-neighborhood certiﬁed.
However, more important for a practical security analysis is that we can certify a given instance, which can be even potentially adversarially perturbed.
Formally, this means that for an input z ∈ A ( x ) , where x ∼ P is an unknown sample from P , that we guarantee
[Equation]
and as x ∈ A ( z, p, l ) this implies that we certify that the pixel ( i, j ) of the potentially manipulated image is classiﬁed the same as pixel ( i, j ) of the unperturbed image x .
However, it is now tricky to get even a probabilistic estimate of the quantity
[Equation]
as the outer maximization process cannot be simply simulated by doing adversarial patch attacks on a clean test dataset.
We propose a way to evaluate a guaranteed lower bound on the fraction of certiﬁed test-time inputs by using a dataset of clean images.
Instead of considering a standard one-patch neighbourhood
[Figure] A ( x ) deﬁned by our threat model (Section 3.2), we propose to consider a neighbourhood
A 2 ( x ) of two independent patches (Figure 8b).
A 2 ( x ) contains all the images x
(cid:48) ∈
A ( x ) as well as their respective patch neighbourhoods A ( x (cid:48) ) .
Therefore, by verifying that ∀ ( p 1 , l 1 ) , ( p 2 , l 2 ) ∈
P : h ( A ( A ( x, p 1 , l 1 ) , p 2 , l 2 ))
i,j = h ( x ) i,j , we guarantee that ∀ x (cid:48) ∈
A ( x ) ∀ ( p, l
) ∈ P : h ( A ( x (cid:48) , p, l ))
i,j = h ( x (cid:48) )
i,j .
We note that corresponding reasoning could be applied to certiﬁcation in (cid:96) p models.
Then A 2 ( x ) would correspond to doubling the radius of the (cid:15) -ball instead of adding a second patch.
Note that Theorem 1 can be directly extended to a threat model of N patches.
In the worst case each of the N patches can affect T different maskings.
Therefore, we need to change the condition of
Theorem 1 to K ≥ 2 NT + 1 .
We apply the described method to evaluating the test-time certiﬁcation guarantees for a toy example of a 0 .
1% patch in Table 4.
We also illustrate how a column mask looks in this case in Figure 8. [Table] [Table]
[Table]
D A DVERSARIAL PATCH EXAMPLE
In this section, we demonstrate an example of a real adversarial patch for a semantic segmentation model similar to the one illustrated in the Figure 1a and show how it is handled by our certiﬁed defences.
We illustrate it for the Swin (Liu et al., 2021) model on one of the images from the ADE20K (Zhou et al., 2017) dataset.
D. P ATCH OPTIMIZATION
We set the patch size to 1% of the image surface.
We select a ﬁxed position for a patch on the rear window of a car (Figure 9a).
For each pixel we extract a list of predicted logits corresponding to each class and apply multi-margin loss with respect to the ground truth label of the respective pixel.
We use random patch initialization without restarts.
As an optimizer we use projected gradient descent (PGD) with 1000 steps and initial step size of 0.01.
We use cosine step size schedule and momentum for the gradient with the rate of 0.9.
The optimization plot and the patch efﬁciency at different iterations of the PGD are illustrated in the Figure 9.
D. C ERTIFIED RECOVERY
We denote the original image as x and the patched image as x (cid:48) .
The voting-based segmentation function h (Section 4.2) provides the majority-vote prediction h ( x ) and the corresponding certiﬁcation map which shows the pixels where all the votes agree.
In Figure we see that a part of the building [Figure] [Figure] and the road is certiﬁed which means that this prediction cannot be affected by an adversarial patch.
Figure demonstrates h which correctly segments those regions in presence of an adversarial patch that fools the original model.
D. C ERTIFIED DETECTION
We perform our analysis by evaluating the veriﬁcation map v (Section 4.2) for the original image x and for the patched image x (cid:48) .
We see that in v ( x ) a major part of the building is certiﬁed i. e. for a part of pixels x i,j that belong to the building and the road we have v ( x ) i,j = 1 .
However, v ( x (cid:48) )
i,j = 0 for those pixels.
It means that we have detected that the prediction on this input is potentially affected by an adversarial patch.
E A DDITIONAL EXPERIMENTS
In Tables 5 and we provide additional experimental results for evaluating different masking schemes proposed in Section 4.1 on different models.
[Table]
[Table]
row 13.
3-mask 17.
4-mask 14.
COCO10K
PSPNet col 21.
row 18.
3-mask 18. 4-mask 17.
DeepLab v col 23.
row 20.
3-mask 20.
4-mask 19.
[Figure] [Figure]
F I NPAINTING ABLATION STUDIES
We perform ablation studies with respect to the demasking step.
The results are in Table 7.
Figureprovides additional illustrations.
As can be seen from the results, our method heavily beneﬁts from having available stronger inpainting models that allow achieving better clean and certiﬁed accuracy.
We consider this property actually as a strength of our method since it will automatically beneﬁt from future research and developments of stronger inpainting methods.
For certiﬁed recovery, we also compare it to GIN Li et al.
based on a generative model that we trained on ADE20K (without using style losses based on ImageNet trained VGG).
The results are in Table 8.
Illustrations can be found in Figure 13.
G C OMPARISON TO SIMPLIFIED D ERANDOMIZED S MOOTHING
Derandomized Smoothing (DRS) Levine & Feizi (2020) was proposed for certiﬁed recovery, therefore in this section we focus on this task.
Direct adaptation of derandomized smoothing to semantic segmentation task requires training a model that is able to predict the full image segmentation from a small visible region.
Since it is not immediately clear to us what architectural design and training [Table] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] procedure would be needed to train such a model, we consider a simpliﬁed version of DRS that we call DRS-S. In this version, we consider an off-the-shelf semantic segmentation model and evaluate how it performs with column masking from DRS.
Therefore, we do not encode the masked regions with the special ’NULL’ value like in DRS but use black color instead.
That is because an off-the-shelf model cannot work with ’NULL’ values.
We run our experiments on ADE20K dataset.
We consider the DRS parameters from the recent SOTA version of Derandomized Smoothing by Salman et al.
Salman et al.
They use column width b = 19 and stride s = 10 for certiﬁed classiﬁcation of 224x224 ImageNet images.
To account for the fact that ADE20K images have larger resolution than ImageNet, we scale the parameters to column width b = 42 and stride s = 22 .
To make the comparison consistent with the rest of our results, we use the patch occupying 0 .
5% of the image.
From Table 9 we can see that DRS-S performs poorly on semantic segmentation task.
The reason for that is illustrated in Figure 14.
Processing the column region in 14c would probably be sufﬁcient for a classiﬁcation model to classify the image into the class "house".
But it is clearly not sufﬁcient to reconstruct the whole segmentation map 14e as can be seen in the Figure 14g.
Whether doing this would be possible with a model speciﬁcally trained to reconstruct the segmentation map from a very small visible region is an open research question (up to our knowledge).
We point out that the value %C of certiﬁed and correctly classiﬁed pixels in the Table 9 is still surprisingly high for DRS-S compared to other metircs.
We attribute this to the fact that the solid black regions are usually treated as a wall by the segmentation model, therefore the images are usually segmented as a wall by the DRS majority voting.
And the wall is a common part of both indoor and outdoor scenes in ADE20K as can be implied from the Table 10 of "big" ADE20K classes.
Therefore, always classifying the output as a wall provides a decent fraction of correctly classiﬁed pixels because of the skewed classes.
However, to provide a better comparison with DRS, we emulate the model which is able to reconstruct the whole segmentation map from the column masking proposed in DRS.
We do this by applying the [Figure] [Table] demasking approach proposed in this work.
We ﬁrst try to reconstruct the whole image from one column and then segment it with an off-the-shelf model as we did with the masks proposed in this paper.
We call this approach DRS-E and the results can be found in Table 9.
H A LIST OF BIG CLASSES
In Section 5.2 we suggest another perspective on the evaluation of our D EMASKED S MOOTHING by speciﬁcally considering its performance on “big” semantic classes.
The object of these classes occupy on average more than 20% of the images in which they appear.
Correctly segmenting these classes is important for understanding the scene.
In Tables 10 and 11 we provide the full list of such classes in ADE20K
(Zhou et al., 2017) and COCO-Stuff-10K
(Caesar et al., 2018) respectively together with the average fraction of pixels that they occupy in the images in which they are present.
We point out that for COCO-Stuff-10K some typically smaller classes such as “sandwich” or “fruit” get included in the list of big classes because of the macro-scale images in which they occupy a big part of the scene.
I C OMPLEXITY ANALYSIS AND PARALLELIZATION
In D EMASKED S MOOTHING , we propose a set of K masks that are applied to the original image (denote the cost of applying a single masking by M ).
As illustrated in Figure 1c, the masked images are demasked (denote the cost of demasking an image by D ) and segmented (denote the cost of segmenting an image by S ); thereupon per-mask segmentations are aggregated into a ﬁnal segmentation and certiﬁcation (cost of aggregation proportional to K ). Asymptotically, compute grows thus with O ( K ( M + D + S ) + K ) while the cost of a standard segmentation is O ( S ) . Thus, for large K or M + D (cid:29) S , real-time applicability would actually be impractical.
However, we note that:
[Table]
[Table]
sandwich 23.
river 40.
broccoli 20.
sand 26.
pizza 25. sea 36. bed 36. sky-other 22.
dining table 21. snow 51. clouds 24. vegetable 20.
water-other 21. M + For certiﬁed recovery, we operate in a setting where K is small ( K ∈ 5 , 7 , 9 ) and does not grow with the image resolution. This is unlike Derandomized Smoothing and its derivatives, where the number of masks in the recovery task grows with the image resolution (or randomized smoothing with thousands of samples per input).
This small value of K beneﬁts our the method in time-sensitive applications. For certiﬁed detection, we can adjust the number of masks for the computational speed by using strided masking as suggested in Section 4.1. Morover, masking, demasking, and segmenting for different masks do
not use any shared data and can thus be fully parallelized if sufﬁciently powerful hardware is available.
Only the aggregation step requires the results of all the previous stages.
However, aggregation time is small compared to the other stages.
In terms of latency, a fully parallelized version of our procedure would thus have a latency proportional to O
( M + D + S + K ) .
For small K and M + D ≈ S , application to real-time video can be facilitated by means of parallelization.
J U SED DATA
In this work, we only use the datasets published under formal licenses: ADE20K
(Zhou et al., 2017) and COCO-Stuff-10K
(Caesar et al., 2018).
To the best of our knowledge, data used in this project do not contain any personally identiﬁable information or offensive content.
The models ZITS (Dong et al., 2022) and Swin (Liu et al., 2021) are published under Apache-2.0 license.
The text of the license for PSPNet (Zhao et al., 2017) can be found here: https://github.com/hszhao/
PSPNet/blob/master/LICENSE
In this section, we provide additional illustrations of our method (Figures 15, 16, 17, 18).
Similarly to the Table 1 we certify against a 1% patch for the detection task and against 0.5% patch for the recovery task. For each mask type we illustrate all the stages summarized in the Figure 1c. We also provide examples of certiﬁcation maps for certiﬁed recovery and certiﬁed detection with different images (Figure 19, 20).
[Figure] [Figure]
[Figure]
[Figure] [Figure]
[Figure]