R ELEVANCE A TTACK ON D ETECTORS
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper focuses on high-transferable adversarial attacks on detectors, which are hard to attack in a black-box manner, because of their multiple-output characteristics and the diversity across architectures.
To pursue a high attack transferability, one plausible way is to ﬁnd a common property across detectors, which facilitates the discovery of common weaknesses.
We are the ﬁrst to suggest that the relevance map for detectors is such a property.
Based on it, we design a Relevance Attack on Detectors (RAD), which achieves a state-of-the-art transferability, exceeding existing results by above 20%.
On MS COCO, the detection mAPs for allblack-box architectures are more than halved and the segmentation mAPs are also signiﬁcantly inﬂuenced.
Given the great transferability of RAD, we generate the ﬁrst adversarial dataset for object detection, i e., Adversarial Objects in COntext (AOCO), which helps to quickly evaluate and improve the robustness of detectors
I NTRODUCTION
Adversarial attacks (Szegedy et al.
; Goodfellow et al.
; Carlini & Wagner (2017); M ˛adry et al.
(2017); Baluja & Fischer (2017)
; Su et al.
(2019)) have revealed the fragility of Deep Neural
Networks (DNNs) by fooling them with elaborately-crafted imperceptible perturbations.
Among them, the black-box attack, i.e., attacking without knowledge of their inner structure and weights, is much harder and more aggressive because it is closer to real-world scenarios.
For classiﬁers, there exist some promising black-box attacks (Papernot et al.
; Brendel et al.
Dong et al.
; Xie et al.
; Lin et al.
; Chen et al.
It is also severe to attack object detection (Zhang & Wang (2019)) in a black-box manner, e.g., hiding certain objects from unknown detectors (Thys et al.
By that, life-concerning systems based on detection such as autonomous driving and security surveillance would be greatly inﬂuence.
To the best of our knowledge, no existing attack is speciﬁcally designed for black-box transferability in detectors, because they possess multiple-output property and a high diversity across architectures.
Adversarial samples do not transfer well in such situations (Su et al.
(2018)), and most attacks only decrease mAP of black-box detectors by 5 to 10% (Xie et al.
; Li et al.
(2018c;b)).
To overcome that, one plausible way is to ﬁnd common properties across detectors, which facilitates the discovery of common weaknesses.
Based on them, the designed attack can threaten variable victims.
In this paper, we adopt the relevance map as a common property, on which different detectors have similar interpretable results, as shown in Fig. 1.
Based on relevance maps, we design a Relevance
Attack on Detectors (RAD).
RAD focuses on suppressing the relevance map rather than directly attacking the prediction as in existing works (Xie et al.
; Li et al.
(2018c;a)).
Because the relevance maps are quite similar across models, those of black-box models are inﬂuenced and misled as well in attack, leading to the great transferability.
Although some works have adopted the relevance map as an indicator or reference of success attacks (Dong et al.
(2019); Zhang & Zhu (2019)
; Chen et al.
; Wu et al.
, there is no work to directly attack the relevance maps of detectors to the best of our knowledge.
In our comprehensive evaluation, RAD achieves the state-of-the-art transferability on 7 black-box models for COCO) dataset (Lin et al.
(2014)), nearly halving the detection mAP.
Interestingly, the adversarial samples of RAD also greatly inﬂuence the performance of instance segmentation, even only detectors are attacked.
Given the high transferability of RAD, we create Adversarial Objects in
COntext (AOCO), the ﬁrst adversarial dataset for object detection.
AOCO contains 10K samples that signiﬁcantly decrease the performance of black-box models for detection and segmentation.
AOCO may serve as a benchmark to test the robustness of a DNN or improve it by adversarial training.
C ONTRIBUTIONS
• We propose a novel attack framework on relevance maps for detectors.
We extend network visualization methods to detectors, ﬁnd out the most suitable nodes to attack by relevance maps, and explore on the best update techniques to increase the transferability.
• We evaluate RAD comprehensively and ﬁnd its state-of-the-art transferability, which exceeds existing results by above 20%.
Detection mAPs are more than halved, invalidating the stateof-the-art detectors to a large extent.
• By RAD, we create the ﬁrst adversarial dataset for object detection,
i e., AOCO. As a potential benchmark, AOCO is generated from COCO and contains 10K high-transferable samples.
AOCO helps to quickly evaluate and improve the robustness of detectors.
R ELATED W ORK
Since (Szegedy et al.
, there have been lots of promising adversarial attacks (Goodfellow et al.
; Carlini & Wagner (2017); M ˛adry et al.
(2017); Baluja & Fischer (2017)
; Su et al.
Chen et al.
Generally, they ﬁx the network weights and change the input slightly to optimize the attack loss.
The network then predicts incorrectly on adversarial samples with a high conﬁdence.
(Papernot et al.
ﬁnd that adversarial samples crafted by attacking a white-box surrogate model may transfer to other black-box models as well.
Input modiﬁcation (Xie et al.
Dong et al. ; Lin et al.
or other optimization ways (Dong et al.
Lin et al.
(2020)) are validated to be effective in enhancing the transferability.
(Xie et al.
(2017)) extends adversarial attacks to detectors.
It proposes to attack on densely generated bounding boxes.
After that, losses about localization and classiﬁcation are designed (Li et al.
(2018c)) for attacking detectors.
(Lu et al. and (Li et al.
(2018a))
propose to attack detectors in a restricted area.
Existing works achieve good results in white-box scenarios, but are not speciﬁcally designed for transferability.
The adversarial impact on black-box models is quite limited, i.e., ato 10% decrease from the original mAP, even when two models only differ in backbone (Xie et al.
(2017); Li et al.
(2018c;b)).
(Wang et al.
(2020)) discusses black-box attacks towards detectors based on queries rather than the transferability as we do.
The performance is satisfactory, but it requires over 30K queries, which is easy to be discovered by the model owner.
For the great transferability, we propose to attack on relevance maps, which are calculated by network visualization methods (Zeiler & Fergus (2014); Selvaraju et al.
; Shrikumar et al.
They are originally developed to interpret how DNNs predict and help users gain trust on them.
Speciﬁcally, they display how the input contributes to a certain node output in a pixel-wise manner.
Typical works include Layer-wise Relevance Propagation (LRP) (Bach et al.
(2015)), Contrastive LRP (Gu et al.
and Softmax Gradient LRP (SGLRP) (Iwana et al.
These methods encourage the reference of relevance maps in attack (Dong et al.
(2019); Zhang & Zhu (2019)
; Chen et al.
Wu et al.
and also inspire us.
However, none of them attack on relevance maps for detectors.
R ELEVANCE A TTACK ON D ETECTORS
We propose an attack speciﬁcally designed for black-box transferability, named Relevance Attack on
Detectors (RAD).
RAD suppresses multi-node relevance maps for several bounding boxes.
Since the relevance map is commonly shared by different detectors as shown in Fig.
1, attacking on it in the white-box surrogate model achieves a high transferability towards black-box models.
In this section, we ﬁrst provide a high-level overview of RAD, and analyze the potential reasons of its transferability.
Then we thoroughly discuss three crucial concrete issues in RAD.
• In Section 3.3, we specify the calculation of relevance maps for detectors, where current visualization methods are not applicable.
• In Section 3.4, we introduce the proper nodes to attack by RAD.
• In Section 3.5, we explore on the suitable techniques to update samples in RAD.
3. W HAT IS RAD
We present the framework of RAD in Fig.
Initialized by the original sample x 0 , the adversarial sample x k in the k th iteration is forward propagated in the surrogate model, getting the prediction f ( x k ) .
Current attacks generally suppress the prediction values of all attacked output nodes in T .
In contrast, RAD suppresses the corresponding relevance map h ( x k , T ) .
To restrain that, gradients of h ( x k , T ) back propagate to x k , which is then modiﬁed to x k +1 .
It is notable that RAD is a complete framework to attack detectors, and its each component requires a special design.
Besides the calculation of relevance maps of detectors, other components in RAD, e.g., the attacked nodes or the update techniques, also need a customized analysis. The reason is that no existing work directly attacks the relevance of detectors, and the experience in attacking predictions is not totally applicable here.
For example, (Zhang & Wang (2019)) emphasizes classiﬁcation loss and localization loss equally, but the former is validated to be signiﬁcantly better in attacking the relevance in Section 3.4.
3. W HY RAD T RANSFERS
RAD’s transferability comes from the attack goal: changing the common properties, i.e., the relevance maps.
As shown in Fig.
3, the relevance maps are clear and structured for the original sample in both detectors.
After RAD, the relevance maps are induced to be meaningless without a correct focus, leading to wrong predictions, i.e., no or false detection.
Because relevance maps transfer well across models, those for black-box detectors are also signiﬁcantly inﬂuenced, causing a great performance drop, which is illustrated visually in Section 4.2.
RAD also attacks quite “precisely”, i.e., the perturbation pattern is signiﬁcantly focused on distinct areas and has a clear structure as shown in Fig. 4.
That is to say, RAD accurately locates the most discriminating parts of a sample and concentrates the perturbation on them, leading to a great transferability when the perturbations is equally bounded.
3. W HAT IS THE R ELEVANCE M APS FOR D ETECTORS
We analyze the potential of RAD above, below we make it feasible by addressing three crucial issues.
To conduct the relevance attack, we ﬁrst need to know the relevance maps for detectors.
Currently, there have been lots of methods to calculate the relevance maps for classiﬁers as described in Section 2, but none of them are suitable for detectors.
We take SGLRP (Iwana et al.
as an example to explain this and then to modify, because it excels in discriminating ability against irrelevant regions of a certain target node.
SGLRP visualizes how the input contributes to one output node in a pixel-wise way by backpropagating the relevance from the output to the input.
R is the initial relevance in the output layer and its n th component is calculated as
[Equation]
where y n is the predicted probability of class n , and y t is that for the single-node target t .
The pixel-wise relevance map h ( x, t ) for the single-node target t is calculated by back propagating the relevance R from the ﬁnal layer to the input following rules speciﬁed in (Iwana et al.
In detectors, we need the pixel-wise contributions from the input to m bounding boxes.
This multi-node relevance map could not be directly calculated by (1), so we naturally modify SGLRP as
[Equation]
where y t i is the predicted probabilities for one target node t i .
T is the set containing all target nodes { t 1 , t 2 , ..., t m } .
We illustrate the difference between SGLRP and our Multi-Node SGLRP in Fig.
SGLRP only displays the relevance map for one bounding box, e.g., “TV”, “chair” and “bottle”.
Multi-
Node SGLRP, in contrast, visualizes the overall relevance.
3. W HERE TO A TTACK
Besides the calculation of relevance maps, it is also important to choose a proper node set T to attack.
Speciﬁcally, we need to select certain bounding boxes and the corresponding output nodes for RAD.
Heuristically, the most “obvious” bounding boxes are desired to be eliminated, so we select the bounding boxes with the highest conﬁdence, following (Xie et al.
Concretely, it is feasible to statically choose m bounding boxes to attack in each iteration, or dynamically attack all bounding boxes whose conﬁdence exceeds a threshold.
In our evaluation, the two strategies differ a little in performance and are not sensitive to hyper-parameter as demonstrated in Appendix B.
This shows that
RAD does not require a sophisticated tuning of parameters, which is user-friendly.
In our following experiments, we statically attack m = 20 nodes.
After selecting bounding boxes, we could attack their size, leading them to shrink; or their localization, leading them to shift; or their conﬁdence, leading them to be misclassiﬁed.
To adopt the best strategy, we conduct a toy experiment by attacking YOLOv3 (Redmon & Farhadi (2018)), denoted as M and other models are speciﬁed in Appendix A.
Given the results in Table 1, the classiﬁcation loss induces a better black-box transferability. This may because detectors generally include a pre-trained classiﬁcation as the feature extractor, and relevance maps are believed to be an indicator of success attacks (Dong et al. ; Zhang & Zhu (2019)).
[Table]
H OW TO U PDATE
By the relevance map h ( x, T ) for certain attacked nodes T , we are able to attack, i.e., update the original sample to become adversarial with the guidance of the attack gradients g ( x ) as
[Equation]
Some update techniques are validated to be effective for enhancing the transferability in classiﬁcation.
For example, Scale-Invariant (SI) (Lin et al.
(2020)) proposes to average the attack gradients by scale copies of the samples as
[Equation]
Besides SI, Diverse Input (DI) (Xie et al.
, Translation-Invariant (TI) (Dong et al.
(2019)) are also promising in classiﬁcation.
We are curious about whether they also work well in object detection.
To explore on this, we adopt these techniques in RAD as the setting suggested by their designers (see
Appendix C).
From the results in Table 2, we discover that SI is quite effective, further decreasing the mAP from the baseline to a large extent.
Accordingly, we adopt (4) to update the sample in RAD.
[Table]
With the calculated gradient, we update the sample as
[Equation]
where α stands for the step length.
x is (cid:96) ∞ -norm bounded by ε from the original sample in each iteration as in (Xie et al.
Dong et al.
; Lin et al.
Gradient g ( x ) is normalized by its average (cid:96) 1 -norm,i.e., || g ( x ) || 1 /N to prevent numerical errors and control the degree of perturbations.
N is the dimension of the image, i.e., N = height × width × channel .
Division by N is necessary because (cid:96) 1 -norm sums all components of the tensor x , which is too large as a normalization factor.
We do not adopt the mainstream sign method because it is not suitable to generate small perturbations as shown in other attacks in detectors (Xie et al.
E XPERIMENTS
In this section, we evaluate the performance of RAD, especially its transferability.
The results are presented numerically and visually.
In comprehensive evaluation, RAD achieves a great transferability in across models and even across tasks.
4. S ETUP
Our experiments are based on Keras (Chollet et al.
, Tensorﬂow (Abadi et al. (2015)) and
PyTorch (Paszke et al.
in 4 NVIDIA GeForce RTX 2080Ti GPUs.
Library iNNvestigate (Alber et al.
(2019)) is used to implement Multi-Node SGLRP.
We conduct experiments on MS COCO 2017 dataset (Lin et al.
(2014)), which is a large-scale benchmark for object detection, instance segmentation and image captioning.
For a fair evaluation, we generate adversarial samples from all 5K samples in its validation set and test several black-box models on their mAP, a standard criteria in many works (He et al.
; Chen et al. (2019a)).
All attacks are conducted with the step length α = 2 for 10 iterations and the perturbation is (cid:96) ∞ - bounded in ε = 16 to guarantee the imperceptibility as in (Dong et al. To validate that the mAP drop comes from the attack instead of resizing or perturbation, we add large Gaussian noises ( σ = 9 ) to the resized images, and report it as “Ablation”.
We choose 8 typical detectors ranging from the ﬁrst end-to-end detector to recent ones for attack and test.
The variety of model guarantees the validity of results.
We specify their information in
Appendix A and the corresponding pre-processing in Appendix C.
4. V ISUAL R ESULTS OF RAD
We visualize several predictions on the same adversarial sample by black-box models in Fig.to intuitively illustrate the transferability of RAD.
The objects in the image, e.g., the laptop and keyboard, are quite large and obvious to detect.
However, with a small perturbation from RAD,black-box models all fail to detect the laptop, keyboard and mouse.
Surprisingly, 4 of them even detect a non-existent “bed”, which is neither relevant nor similar in the image.
The attack process of
RAD is analyzed in Appendix D.
4. RAD’ S T RANSFERABILITY IN O BJECT D ETECTION
To evaluate the in-domain transferability of detection attacks and cross-domain transferability of classiﬁcation attacks, we test the detection mAP of 8 models in COCO adversarial samples generated in the same setting.
For detection attacks, adversarial samples are crafted by attacking surrogate model M2 (YOLOv
Redmon & Farhadi (2018)).
For classiﬁcation attacks, we use the model output on the clean sample as the label.
By several state-of-the-art attacks on surrogate classiﬁers (InceptionV3 Szegedy et al.
(2016) here as in Xie et al.
Dong et al.
, the adversarial samples are generated and tested the mAP as the transferability towards detectors.
Details of implementation are described in Appendix C.
We present the results in Table 3.
Among the classiﬁcation attacks and detection ones, cross-domain attack (Naseer et al.
(2019)) is the most effective, but RAD is more aggressive.
RAD enjoys a state-of-the-art transferability towards most black-box models, outperforming other methods for above 20%.
The detection mAPs are more than halved, making state-of-the-art detectors worse than the early single-shot detector (SSD Liu et al.
(2016), M1).
[Table]
SI-PGD 27.
MI-DI-PGD 22.
MI-TI-PGD 20.
CD-painting 16.
CD-comics 16.
DetectionAttack
Dfool 23.
Loc 21.
DAG 20.
Ours
RAD 14.
4. RAD’ S T RANSFERABILITY TO I NSTANCE S EGMENTATION
Detection and segmentation are similar in some aspects, so they could be implemented in one network (He et al.
; Cai & Vasconcelos (2018); Chen et al. (2019a)).
Also, adversarial samples for object detection tend to transfer to instance segmentation (Xie et al.
Accordingly, we evaluate this cross-task transferability by RAD on surrogate detectors YOLOv3 (Redmon & Farhadi (2018),
M2), RetinaNet (Lin et al.
(2017), M3) and
Mask R-CNN
(He et al.
(2017), M5).
From the results in Table [Table] we ﬁnd that RAD also greatly hurts the performance of instance segmentation, leading to a drop on mAP of over 70%.
This inspire the segmentation attackers to indirectly attack detectors
[Table]
[Table]
A DVERSARIAL O BJECTS IN C ONTEXT
Given the great transferability of RAD, we create Adversarial Objects in COntext (AOCO), the ﬁrst adversarial dataset for object detection.
AOCO is generated from the full COCO 2017 validation set (Lin et al.
with 5k samples.
It contains 5K adversarial samples for evaluating object detection (AOCO detection) and 5K for instance segmentation (AOCO segmentation).
All 10K samples in AOCO are crafted by RAD.
The surrogate model we attack is YOLOv3 for
AOCO detection and Mask R-CNN for AOCO segmentation given the results in Table 3 and Table 4.
We measure the perturbation ∆ x in AOCO by Root Mean Squared Error (RMSE) as in (Xie et al.
Liu et al.
It is calculated as (cid:112)(cid:80) i (∆ x i ) 2 /N in a pixel-wise way, and N is the size of the image.
Performance of AOCO is reported in Table 5.
The RMSE in AOCO is lower than that in (Wu et al.
, and the perturbation is quite imperceptible.
Details and samples of AOCO are presented in Appendix E.
[Table]
[Table]
C ONCLUSION
To pursue a high transferability, this paper proposes Relevance Attack on Detectors (RAD), which works by suppressing the multi-node relevance, a common property across detectors calculated by our Multi-Node SGLRP.
We also thoroughly discuss where to attack and the how to update in attacking relevance maps.
RAD achieves a state-of-the-art transferability towards 7 diverse black-box models, exceeding existing results by above 20%, and also signiﬁcantly hurts the instance segmentation.
Given the great transferability of RAD, we generate the ﬁrst adversarial dataset for object detection, i.e., Adversarial Objects in COntext (AOCO), which helps to quickly evaluate and improve the robustness of detectors.
Also, attacking other common properties is promising for a good transferability.
A M ODEL I NFORMATION
Table 6 presents the models’ information from our evaluation and MMdetection (Chen et al.
(2019b)).
[Table]
B I NFLUENCE OF H YPER -P ARAMETERS IN N ODE S ELECTION
Performance of RAD is not sensitive to hyper-parameter, no matter the strategy to select bounding boxes is dynamic or static.
Attackers are not bothered to tune them carefully.
The parameter for dynamic strategy refers to the pre-softmax conﬁdence threshold to select a bounding box. The parameter for static strategy refers to the ﬁxed number of selected bounding boxes in each iteration.
[Table]
C I MPLEMENTATION D ETAILS C. P RE - PROCESSING
To pre-process, we resize the image with its long side as 416 for YOLOv3 or RetinaNet and 448 for
Mask R-CNN, and then zero-pad it to a square.
The resolution is kept relatively the same for a fair evaluation.
Images are normalized to [0,1] in YOLOv3 or subtracted by the mean of COCO training set in RetinaNet and Mask R-CNN.
Accordingly, samples in AOCO detection have the long sideand that for AOCO segmentation is 448.
C. T RANSFER -E NHANCING U PDATE T ECHNIQUES
DI (Xie et al.
(2019)) transforms the image for 4 times with probability p ( p = 1 for better transferability as suggested) and averaging the gradients.
The transformation is to resize the image to 0 . 9 × its size and randomly padding the outer areas with white pixels.
SI (Lin et al.
(2020)) divides the sample numerically by the power 2 for 4 times and averages the 4 obtained gradients.
TI (Dong et al.
(2019)) translates the image to calculate the augmented gradients.
To implement it efﬁciently, it adopts a kernel to simulate the averaging of gradients.
We choose the kernel size 15 as suggested.
MI (Dong et al.
(2018)) uses momentum optimization (parameter µ = 1 as suggested) for a better transferability and a faster attack.
Cross-domain attack (Naseer et al.
(2019)) uses extra datasets
(paintings, denoted as CD-paintings, and comics, denoted as CD-comics) to train a perturbation generator with the relative loss.
The adopted surrogate model is also InceptionV3 for consistency.
All perturbations are resized to ﬁt the sample size.
C. D ETECTION A TTACKS
For DAG (Xie et al.
, we follow the setting of generating dense proposals.
The classiﬁcation probabilities of 3000 bounding boxes with highest conﬁdence are attacked.
But we alter its optimization to (5) because its original update produces quite small perturbation, leading to a poor transferability, which is unfair for comparison.
Dfool (Lu et al.
suppresses the classiﬁcation conﬁdence for the original bounding boxes, which is the same in our experiment.
Localization loss is shown to be useful in (Zhang & Wang (2019)), and here we suppress the width and height of the original bounding boxes.
D V ISUAL R ESULTS OF RAD P ROCESS
By RAD, the relevance map is attacked to be meaningless and loss its focus.
In Fig.
7, the initial prediction is correct and the relevance map is clear.
RAD constantly misleads the relevance map to be unstructured without outline of objects.
Finally, all bounding boxes vanish.
E M ORE ABOUT AOCO
We report the mAP50 and mAP75 of AOCO in Table 8 and Table 9.
We show in Fig.
8 the visual comparative results.
M
M M
[Table]
COCO detection 49.
AOCO detection 26.
COCO segmentation / / / / 60.
AOCO segmentation / / / / 2.
M
M M
[Table]
COCO detection 30.
AOCO detection 14.
COCO segmentation / / / / 40.
AOCO segmentation / / / / 1.