Adversarial Attack on Attackers: Post-Process to Mitigate Black-Box Score-Based Query Attacks
Anonymous Author(s)
AffiliationAddress
Abstract
The score-based query attacks (SQAs) pose practical threats to deep neural networks by crafting adversarial perturbations within dozens of queries, only usingthe model’s output scores.
Nonetheless, we note that if the loss trend of the outputsis slightly perturbed, SQAs could be easily misled and thereby become much lesseffective.
Following this idea, we propose a novel defense, namely Adversarial Attack on Attackers (AAA), to confound SQAs towards incorrect attack directions byslightly modifying the output logits.
In this way, (1) SQAs are prevented regardlessof the model’s worst-case robustness; (2) the original model predictions are hardlychanged, i.e. , no degradation on clean accuracy; (3) the calibration of confidencescores can be improved simultaneously.
Extensive experiments are provided toverify the above advantages.
For example, by setting ℓ ∞ = 8 / 255 on CIFAR-10,our proposed AAA helps WideResNet-28 secure 80 .
59% accuracy under Squareattack ( 2500 queries), while the best prior defense ( i.e. , adversarial training) onlyattains 67 .
Since AAA attacks SQA’s general greedy strategy, such advantagesof AAA over 8 defenses can be consistently observed on 8 CIFAR-10/
ImageNetmodels under 6 SQAs, using different attack targets, bounds, norms, losses, andstrategies.
Moreover, AAA calibrates better without hurting the accuracy.
Introduction
Deep Neural Networks (DNNs) are vulnerable to adversarial examples (AEs), where humanimperceptible perturbations added to clean samples can fool DNNs to give wrong predictions [1, 2].
Recently, such a threat is made practically feasible by the black-box score-based query attacks (SQAs)[3, 4, 5], as they only require the same information as users to craft efficient AEs.
Users, for betterjudgments, need the model’s prediction confidence indicated by DNNs’ output scores, which is theonly knowledge needed by SQAs to perform attacks.
In contrast, white-box attacks [6, 7, 8] ortransfer-based attacks [9, 10] require the gradients or training data of DNNs.
Moreover, it has beenshown that SQAs can achieve a non-trivial attack success rate by a reasonable number of queries,e.g. , dozens, compared to thousands of queries for decision-based query attacks [11, 12, 13].
Thus,such feasibility and effectiveness of SQAs are attracting increasing attention from defenders [14].
Defending against SQAs is a different goal compared to improving the worst-case robustness ascommonly studied [15, 16, 17].
Because in real-world scenarios which SQAs are designed for,
DNNs are treated as black boxes, in which the only interaction between models and users/attackers isthe model’s output scores.
Thus, altering the scores is all defenders could do here, either in direct orindirect ways.
Most existing defenses indirectly change outputs by optimizing the model [8, 18, 19]or pre-processing the inputs [14, 20], which, however, severely affect models’ normal performancefor different reasons.
Training a robust model, e.g. , by adversarial training [8, 21], diverts the model’sattention to learning AEs, yielding the so-called accuracy-robustness trade-off [22].
Randomizing
[Figure]
𝑦 [Figure] in
[Figure] model [Figure] hurdling accurate decision.
Dynamic inference [19, 30] is time-consuming due to the optimization on model.
Thus, it is imperative to develop a user-friendly defense against SQAs
In this paper, we hereby consider a post-processing defense as demonstrated in Fig.
1 (a), which naturally enjoys
How to serve users while avoiding SQA attackers when they access the same output information?
Since SQAs are black-box attacks that find the adversarial direction to update AEs by observing theloss change indicated by DNN’s output scores, we can perturb such scores directly to fool attackersinto incorrect attack tracks.
Following this idea, we propose the adversarial attack on attackers (AAA),which manipulates the loss trend so that attackers, trying to greedily update AEs following the originaltrend, are led to an incorrect path.
Specifically, AAA directly optimizes DNN’s logits to control theoutput loss to approximate certain designed curves so that attackers, seeing our confounding outputs,would be attacked to lose their direction. In Fig.
1 (b), we give two representative designs of theloss curve to achieve the above goal. One is to increase loss mostly and decrease it dramatically andshortly along the correct attack direction, and another is to oscillate the curve around the original blueline.
Now that a post-processing module is adopted here, we could simultaneously use it to calibratethe model as commonly proposed [31, 33].
Thus, a simple plug-in post-module is all you need toboth fool attackers and offer users accurate confidence.
By post-processing, AAA not only lowers the calibration error without hurting accuracy in all cases,but also efficiently prevents SQAs, e.g. , helping a WideResNet-28 [34] on CIFAR-10 [35] secure80 .
59% accuracy under Square attack [5] ( 2500 queries), while the best prior defense ( i.e. , adversarialtraining [36]) only attains 67 .
Even if attackers try to guess defender’s strategy (which requiresheavy queries) and develop adaptive strategies, AAA could also confound both non-adaptive andadaptive attacks by the sine design.
Because AAA attacks the general greedy update of SQAs, suchadvantages of AAA over 8 baseline defenses can be consistently observed on 8 CIFAR-10/
ImageNetmodels under 6 SQAs, using different attack targets, bounds, norms, losses, and strategies, verifying AAA a user-friendly, effective, and generalizable defense.
Our contributions are three-fold.
• We analyze current defenses from the view of SQA defenders and point out that a post-
processing module forms not only effective but also user-friendly and plug-in defenses.
• We design a novel adversarial attack on attackers (AAA) defense that fools SQA attackersto incorrect attack directions by slightly perturbing the DNN output scores.
• We conduct comprehensive experiments showing that AAA outperforms the other 8 defensesin the accuracy, calibration, and protection from all tested SQAs under various settings
Related work
observing the loss change indicated by DNN’s output scores, i.e. , logits or probabilities.
Early SQAstry to estimate DNN’s gradients by additional queries around the sample [38, 42].
Recently, it isvalidated query-efficient to perform fast pure random-search SQAs [4, 5].
It has also been proposedto use pre-trained surrogate models in SQAs [39, 43], which, however, demands unfeasible accessto DNN’s training sample as in transfer-based attacks [9, 10].
Thus, we do not focus on defendingsuch SQAs as also in [14].
Besides SQAs, DQAs rely only on DNN’s decisions, e.g. , the top-1predictions, to generate AEs.
Since DQAs could not perform the greedy update, they start crafting the
AE from a different sample and keep DNN’s prediction wrong during the attack.
Currently, DQAsneed thousands of queries to reach a non-trivial attack success rate [44] compared to dozens of timesfor SQAs [5] as compared in Appendix A, limiting their threats.
Additionally, SQAs are mostlyapplicable and attackers do not have to resort to DQAs because, in the real world, DNNs must notonly be accurate but also indicate when they tend to be incorrect [31] by outputting confidence scores.
Adversarial defense mainly has two different goals.
The first goal is to improve DNN’s worst-caserobustness [17], which demands DNNs to have no AE around clean samples bounded by a normball.
Such robustness is originally evaluated by gradient-based white-box attacks [6, 7], but later,gradient obfuscation phenomenon is discovered [15], motivating evaluations to incorporate randomnoise and black-box attacks [16] in adaptive methods [16, 45, 46].
In this assessment, adversarialtraining (AT) [8, 21] is validated as the most effective defense, and other methods, e.g. , using moreor augmented data [47, 48], designing special architecture [18, 49, 50], and inducing randomnessin training [51, 52], all need collaborations with AT to achieve good performance.
Another defensegoal is to mitigate real-case adversarial threats, and the defense performance is evaluated by thefeasible and query-efficient SQAs.
A more robust model in the worst cases certainly protects itselfin real cases.
Besides, it is also possible to defend by dynamic inference [19, 30], and randomizing[14, 23, 24, 25], denoising [20, 27] or quantifying [28, 29] inputs.
However, all aforementioneddefenses on inputs or models exert a non-negligible impact on accuracy, calibration, or inferencespeed due to the focus on learning AEs in AT, the reduction of signal-noise ratio in pre-processing, orthe costly test-time optimization in dynamic inference as compared meticulously in Section 3.
Model calibration performance is a higher demand for a good model besides high accuracy.
Besides correct predictions, model calibration additionally requires DNNs to produce accurateconfidence in their predictions [53].
For instance, exactly 80% of the samples predicted with 80%confidence should be correctly predicted.
In this point of view, the expected calibration error (ECE)[32] is widely used to quantify the error between accuracy and confidence.
Various methods have beenproposed to improve DNN’s calibration in training [33, 54, 55, 56] or by a post-processing moduleafter training [31, 32, 57].
Among them, temperature scaling [32] is a simple but effective method[31], which divides all output logits by a single scalar tuned by a validation set so that the ECE intesting samples would be significantly reduced.
Since division is a simple post-processing operation,calibration could be simultaneously achieved by AAA, the first post-processing defense.
There havebeen methods to avoid attacks by calibration [56] or calibrate by attack [33], but the simultaneousimprovement of defense and calibration has not been reported to the best of our knowledge.
Preliminaries and motivation
Before presenting the proposed method, we first introduce the key ideas of SQAs and analyze existingdefenses.
For a sample x labelled by y , an SQA on a DNN f generates an AE x ′ by minimizing themargin between logits [4, 5, 40, 58] as
[Equation]
namely the margin loss [7].
For defenders without the label y , it is possible to calculate the unsupervised margin loss based only on the logits vector z
= f ( x ) as
[Equation]
by assuming the model prediction ˆ y to be correct, since handling originally misclassified samplesfalls beyond the scope of defense [17].
For attackers with label y , it can be known that the attacksucceeds if L ( f ( x ) , y ) < 0 .
To quickly realize this, SQAs only update AEs if a query x q has a lowermargin loss compared to the current best query x k , i.e. ,
[Equation]
Existing SQAs [4, 5, 40] are quite effective, capable of halving the accuracy within 100 queries on
CIFAR-10 when ℓ ∞ = 8 / 255 , posing significant and practical threats.
Aware of this, the first defensespecifically against SQAs has been recently proposed [14] to pre-process inputs by random noise.
Besides, other existing defenses are also useful to avoid SQAs, among which the most representativeones are adversarial training (AT) [8, 21], dynamic inference [19, 30], and pre-processing [14, 28].
These defenses work in different mechanisms as illustrated in Fig.
1 (a) and Section 2.
Here in Table1, we present a comparison of their key characteristics related to the defense performance.
[Table]
AT [8, 21] pre-pro [14, 28] dyn-inf [19, 30]
AAA (ours) accuracy = ↓↓ ↓ =
= calibration ↑ / ↓ ↓ ↑ testing cost =
=
= ↑↑↑
= training cost =
↑↑↑ =
= =
acc under SQA ↑ ↑↑ ↑ ↑ ↑↑↑
Real-case defenders are expected to serve users and meanwhile avoid SQA attackers.
The formerdemands good accuracy, calibration, and inference speed.
While the latter requires good protection,and preferably, no additional training (denoted as " = ").
As a training-time defense on model, ATexerts a significant impact on accuracy [22], e.g. , > 10% accuracy drop on ImageNet [49], with severalfold training computation and an indefinite impact (denoted as "/") on calibration.
Pre-processingis a test-time defense on input, which avoids SQAs by reducing the input’s signal-noise ratio butinevitably hurts normal performance.
Recently, the dynamic inference is proposed as a test-timedefense by optimizing the model, which shows no accuracy drop but dramatically increases thecomputation, e.g. , DENT [19] consumes > 700% test-time calculation.
Given the above discussion, itis imperative to develop a user-friendly and efficient method to effectively defend against SQAs.
Adversarial attack on attackers
Defending by post-processing naturally carries the advantages of good accuracy and calibration[31], no additional training and negligible test-time computation.
Although it does not improve thecommonly-studied worst-case robustness [16], its potential to defeat SQAs in real cases has not beenexplored yet to the best of our knowledge.
By investigating current defenses from the perspective of
SQA defenders in Sec.
3, it is interesting to find that in the real world, no matter whether defendersalter inputs, models, or outputs, what users/attackers get are just the changes in outputs, i.e. , theblack-box setting.
Thus, it is reasonable and preferable for SQA defenders to directly manipulate
DNN’s output scores, which has already become a standard practice in model calibration [31].
But how?
Now that attackers conduct the adversarial attack on the model, why cannot we defendersactively perform an adversarial attack on attackers?
If attackers search for the adversarial directionby scores, we can manipulate such scores to cheat them into an incorrect attack path.
Following thisidea, we develop the adversarial attack on attackers (AAA) to control what the SQA attackers basetheir actions on, i.e. , the loss (1) indicated by the DNN’s output scores.
Under such direct misleading,
SQA attackers, following their original policy, would be guided to wherever we want them to godue to their greediness, seeing (3).
But such change on outputs should be slight to ensure it is justattackers (observing loss trend) rather than users (observing outputs) that are misled. Therefore, theloss trend can not be reversed completely, but only be reversed locally in periodically-set intervals.
In each local interval, there are lots of designs to manipulate the loss trend.
For example, we canlet the loss along the adversarial direction (almost) always increase and only dramatically decreasebetween intervals, seeing the orange line of Fig.
1 (b).
It is also possible to smooth the cross-intervalloss change by making it oscillate like the green line in Fig.
1 (b), so that attackers could not easilyguess defender’s strategy by observing the loss drop. Either design cheats attackers by slight outputmodifications so that users get accurate scores.
This is because we manually divide loss values intosmall intervals and handle them separately, which is realized by designing the misleading loss curvebased on periodic loss attractors as
[Equation]
where l org =
L u ( z org ) is the original loss for the unmodified logits z org .
τ is the period of attractorintervals, and "floor" denotes rounding down decimals to integers. Eq.
(4) sets loss values from[ l atr − τ/ 2 , l atr + τ/ 2] as an interval.
For example, if τ = 6 , then according to (4), the closest lossattractor of logits with l org =
→ 6 is l atr = 3 .
Thus, the 1 / 2 term is necessary to avoid settingl atr = 0 , i.e. , the decision boundary, which may flip model’s decision.
Periodically-set loss attractors divide logits with different l org into intervals, so that we couldmanipulate the loss trend in each interval to form our designed loss curve that cheats attackers.
Thetarget loss values l trg in the two misleading curves l trg _ lnr , l trg _ sin in Fig. 1 (b) can be expressed as
[Equation]
In the former linear design, when l org decreases from l atr + τ/ 2 to l atr − τ/ 2 as the sampleapproximates the decision boundary, we output loss that increases from l atr − α × τ/ 2 to l atr + α × τ/ 2to fool attackers that this is an incorrect attack direction. In contrast, AAA-sine outputs an increasingloss when l org decreases from around l atr + τ/ 4 to l atr − τ/ 4 . Although we do not always reversethe loss trend here, it is sufficient to resist SQAs.
Plus, AAA-sine enjoys an additional advantage tofool attackers that try to guess our defense strategy due to the smooth loss transition across intervals.
Although the periodic design has already altered output confidence very slightly, we could also stepfurther to improve the precision of confidence because post-processing logits is a standard practice inmodel calibration [32, 57].
Thus, simultaneous defense and calibration is obtainable in a single AAAmodule by controlling the loss while encouraging the output confidence σ ( z ) = max( softmax ( z )) ,i.e. , the maximum probability after softmax, to approach the calibrated one p trg as
[Equation]
which is a straightforward design to fulfill two purposes.
The first term encourages the perturbedlogits z to have a loss L u ( z ) close to the target value l trg , forming the misleading loss curve for SQAs. The second item motivates the output confidence σ ( z ) close to the calibrated one p trg so thatusers get accurate confidence scores. β balances between the above two goals.
Despite its simplicity,
Eq. 6 has to be solved by optimization because the exponential operation in softmax makes Eq.6 a transcendental equation without closed-form solutions.
Luckily, optimizing low-dimensionallogits is not costly [32, 57].
By optimizing Eq. 6 for κ iterations, DNN’s output is both accurate andmisleading, achieving two seemingly contradictory goals simultaneously.
The calibrated confidence p trg is obtainable by various model calibration methods [32, 57].
Amongthem, temperature scaling [32] is simple but effective [31], which divides all logits by a scalar T asp trg =
σ ( z org /T ) .
The temperature T is tuned by a validation set to minimize the calibration error,and here, such tuning is conducted when the optimization (6) is also performed for κ iterations sothat we can find the temperature that suits AAA best.
After that, T is fixed for inference in AAA.
Input: the logits z org , T , τ, α, β, κ .
Output: post-processed logits z 1: get original loss l org =
L u ( z org ) by (2) 2: set target loss l trg by (5) 3: set confidence p trg = σ ( z org / T )
We summarize our algorithm in Alg. 1.
AAAfirst calculates the original loss l org , according towhich sets the target loss l trg that forms the misleading loss curve.
Then AAA optimizes the logitsto reach l trg , and also the target confidence p trg ,which is obtained by dividing the original logitsusing a pre-tuned temperature T .
The overall procedure is in test-time with only optimization onthe logits, making AAA a computation-efficient,plug-in, and model-agnostic method with good accuracy, calibration, and defense against SQAs.
Experiments
5.1 Setup
We evaluate AAA along with 8 defense baselines, including random noise defense (RND [33]),adversarial training (AT [36, 59, 49]), dynamic inference (DENT [19]), training randomness (PNI[51]), and ensemble (TRS [60]).
Results of AT with extra data [22]) are in Appendix A. For AAA,we mostly experiment with the linear design, because the two designs of AAA share most good characteristics. AAA-sine could fool adaptive attacks, which is costly to design in real world asdiscussed in Sec 5.6. AAA-linear uses α = 1 , τ = 6 , β = 5 , κ = 100 , and an Adam optimizer[61] with learning rate 0 .
1 , β 1 = 0 .
9 , β 2 = 0 .
AAA-sine uses α = 0 .
7 and inherits otherhyper-parameters in study of adaptive attacks.
To calibrate simultaneously, we perform temperaturescaling using 1K/5K samples for CIFAR-10 testing set / ImageNet validation set to and make themdisjoint with the attack samples as much as possible.
Other hyper-parameters are in Appendix E.
The defenses are assessed by 6 state-of-the-art SQAs, which are random search methods including
Square [5], SignHunter [40], and SimBA [4], and gradient estimation methods including NES[58], and Bandits [3].
Square attack [5] selects a decreasing size of random square perturbations.
SignHunter [40] estimates the sign of gradient and flips corresponding perturbations.
SimBA[4] randomly samples perturbations given an orthonormal basis.
NES [58] adopts the Natural
Evolutionary Strategies for gradient estimation.
Bandits [3] jointly uses the time and data-dependentgradient prior.
Some SQAs use pre-trained models [39, 43], which, however, demands unfeasibleaccess to DNN’s training sample.
To evaluate real-case defenses against training-based attacks, wealternatively consider a practical SQA QueryNet [62] based on model stealing.
Results of DQAs and
SQAs using other than margin loss are put in Appendix A. We mostly perform untargeted ℓ ∞ attacks,but we also test the targeted and ℓ 2 attacks under different bounds to observe AAA’s generalization.
We use 8 DNNs, which are mostly WideResNets [34] as in [17].
The pre-trained models of PNI [51] /
TRS [60] are ResNet-20 [63], and we also test ResNeXt-101 [64] in ImageNet.
Other studied DNNscome from RobustBench [17] and torchvision [65] as specified in Appendix E. AT models are testedusing the same bound as in AT, if not otherwise stated.
We use all 10K CIFAR-10 testing samples.
For ImageNet, we randomly-select 1K validation samples from all 1K classes respectively (1 imagein 1 class) to eliminate the class bias as in [62].
All images are rescaled to [0 , 1] , and the ImageNetones are resized to 224 × 224 .
Before feeding them to DNNs, we quantify images to 8-bit, imitatingthe real-case 8-bit image setting as in [62].
Experiments are performed on an NVIDIA Tesla A
GPU but could be run on any GPU with over 4GB memory.
As defenders, we are concerned about DNN’s remaining accuracy after it being attacked by SQAsfor a certain query times.
Thus, we report such SQA adversarial accuracy after 100 and 2500queries, reflecting DNN’s performance under mild and extreme SQAs.
We include the average querytimes, another metric commonly used by attackers, in Appendix C. To measure the calibration, theexpected calibration error (ECE) [32] is commonly used.
ECE divides all N testing samples into Mbins, and each bin contains samples with confidence ranging from the m th quantile to the ( m + 1) thquantile of [0 , 1] .
Then ECE is calculated by the difference between accuracy and confidence as
[Equation]
, where ˆ y i is the predicted label of the i th sample inthe m th bin B m , and ˆ p i ˆ
y i represents the probability confidence of this prediction.
RobustBench [17] by Square attack [5] ( ℓ ∞ = 8 / 255 ) using the 9953 th CIFAR-10 test sample (other samples have similar trends as in Appendix B).
AAA fools attackers precisely as shown by the tiny symmetric oscillations of two lines.
5.2 Visual illustration of AAA
We first visually illustrate the mechanism and effects of AAA.
AAA, besides fooling attackers, alsocalibrates the model, forming the loss curve as in Fig.
Slightly different from the ideal defensein Fig.
1 (b) is that the orange curve is lowered for calibration.
However, calibration does nothurdle fooling attackers, i.e. , the orange line is mostly going opposite to the blue line in a preciselysymmetric manner.
Also, two lines cross the decision boundary at the same time, meaning that AAAdoes not change the decision (the negligible increase of AAA accuracy comes from randomness).
Besides the loss, it is also necessary to observe the output probabilities of AAA, which is displayedin Fig.
3 (left).
AAA, indicated by the orange lines, changes the scores slightly without hurtingaccuracy (dotted lines) compared to RND [33] and AT [36].
By such small modifications, AAA notonly improves the calibration but also prevents SQAs by misleading them into incorrect directions.
As displayed by the right part of Fig.
3, AAA is super effective in avoiding SQAs, even if it is aquery-efficient attack working at a large bound.
Specifically, AAA preserves a standard trained DNNto have > 60% accuracy after 2500 queries, doubling the performance of AT and tripling that for RND.
[Table]
ResNetSimBA [4] 53.04 / 03.
NES [58] 83.42 / 12.
Bandit [3] 69.86 / 41.
ImageNet
ECE (%) 5.
ℓ ∞ =
Acc (%) 77.
Square [5] 52.27 / 09.
WideSignHunter [40] 53.05 / 13.
ResNetSimBA [4] 71.79 / 20.
NES [58] 77.11 / 64.
Bandit [3] 71.33 / 65.
ImageNet
ECE (%) 8.
ℓ ∞ =
Acc (%) 78.
Square [5] 54.51 / 11.
ResSignHunter [40] 54.12 / 13.
NeXtSimBA [4] 70.86 / 24.
NES [58] 78.21 / 66.
Bandit [3] 72.19 / 67.
5.3 Numerical results of AAA
We report the main numerical results in Table 2, where RND and AAA-linear are directly implementedin the undefended model denoted as "None".
The AT method for the three models are PSSiLU [36],vanilla AT [59], and feature denoising [49], respectively.
According to Table 2, AAA-linear notonly consistently reduces ECE by > 12% , but also does not hurt clean accuracy.
In contrast, the AT models lose > 7% accuracy and RND also endures a drop in accuracy and calibration.
Under
SQAs, AAA-linear preserves a significantly higher accuracy than the undefended model, which istotally destroyed.
The most threatening SQAs in two datasets more than halve the CIFAR-10 model’saccuracy within 100 queries and degrade ImageNet ones to < 15% by 2500 times.
However, the AAAmodel remains > 75% and > 55% accuracy in extreme cases.
AT and RND are useful in mitigating
SQAs, but it is AAA-linear that tops the defense performance in almost all cases.
Besides AT and RND, diverse defenses have also been proposed, and it would be interesting to seetheir results, which we report in
Appendix F.
The conclusion is that they are not comparable to AAA-linear in real cases regarding the accuracy, calibration, and defense performance.
5.4 Generalization of AAA
Aside from the untargeted ℓ ∞ attacks, we also conduct targeted attacks, ℓ 2 attacks under differentbounds to study the generalization of AAA.
A targeted attack is successful only if an AE is mispredicted as a pre-set class, which, is randomly chosen from incorrect classes for each sample here.
Andℓ 2 attacks bound the perturbations by ℓ 2 norm, which is reported to fool DNNs better [10].
Here weadditionally validate AAA’s plug-in advantage by combining it with AT.
Note that what differs from
AAA and most existing defenses [49, 50, 51] combined with AT is that AAA has already achievedexcellent defense performance, so such a combination is feasible but not necessary.
We choose
Square attack [5] to perform the above evaluations because it is the most effective SQA as testedin Table 2.
The results are presented in Table 3, where all models are exactly the same as in Table2 (CIFAR-10) without tuning hyper-parameters of defense to fairly evaluate the generalization ofdifferent methods.
The results of ℓ 2 AT model and other ℓ ∞ AT models are put in Appendix A.
In the difficult targeted attack setting, the undefended model remains only 2 .
84% accuracy after 2500queries, which is approximately just the accuracy drop of the AAA model.
For ℓ 2 attacks, AAA-linearis still capable of mitigating threats without hurting users, and its superiority is more outstandingas the attack bound becomes larger.
AT models, although robust, suffer from attacks under a largeor different norm ball [56].
Thus, its defense effects decrease as SQA alters the setting, seeing thebottom line.
Defended by AAA, however, this drawback would be greatly avoided.
An ℓ ∞ AT model,even under ℓ 2 = 2 . 5 attack after 2500 queries, is hardly influenced, i.e. , increasing query times after100 queries hardly better the attack performance, discouraging SQA attackers.
[Table]
5.5 Time consumption of AAA
We study the influence of optimization times κ in AAA to see the balance between the amount ofcomputation and defense performance.
AAA’s runtime is estimated by inferring 10K CIFAR-10samples in a WideResNet28 model on an NVIDIA GeForce RTX 2080Ti GPU.
The results areshown in Table 4.
As we see, optimizing low-dimensional logits is not costly, and good defense andcalibration results are also obtainable by 60-80 iterations, which costs less time.
Since optimizinglogits is independent of model size, model owners could determine AAA runtime precisely.
[Table]
A study of other hyper-parameters has been put in
Appendix G.
An intuitive conclusion is that AAAis insensitive to even exponentially tuned hyper-parameters.
And a larger α , a larger τ , or a smaller βthat highlights defense more v.s. calibration thereby increases the adversarial accuracy and ECE.
5.6 Adaptive attacks of AAA
Designing adaptive attacks, though possible, is costly and easy to bypass in real-world scenarios.
Because here, attackers and defenders are in a double-blind relationship, i.e., attackers do not knowthe model, including the defense strategy.
Thus, the discovery process of defense strategies fordeveloping adaptive attacks would require additional queries.
After that, attackers also have todevote considerable manual efforts to creatively deduce what defenders actually do.
In this regard, AAA-linear has already imposed a great hurdle to adaptive attackers.
But still, it is interesting to see that after unveiling defender’s strategy by great effort, how adaptiveattackers could break the AAA model.
There are two straightforward adaptive attacks against AAA:going bidirectional or opposite to the original attack direction, which are both based on the exceptionalloss change that attackers observe in AAA-linear to decide the update direction.
However, adaptiveattackers are also easy to fool because the dramatic drop of loss across intervals could be smoothed,e.g. by using a sine function to design the target loss as the second equation in (5).
In this way, neitherdirection of search is likely to figure out the defense strategy, seeing results below.
[Table]
Square 39.
bi-Square 57.
op-Square 94.
The bidirectional / opposite search weakens
AAA-linear from 81.36% to 62.91% / 57.31%,but AAA-sine that ascends and descends theloss along the attack direction misleads bothnon-adaptive and adaptive SQAs.
By a sine l trg ,the defended adversarial accuracy for the nonadaptive attack is kept at 78.34% while that under adaptive attacks is improved to 76.69% / 76.41%.
Thus, defenders could easily mitigate even adaptive SQAs following our idea to fool attackers.
Conclusion, impacts and limitations
We develop a novel defense against score-based query attacks (SQAs).
Our main idea is to activelyattack the attackers (AAA), misleading them into incorrect attack directions.
AAA achieves that bypost-processing DNN’s logits while enforcing the new output confidence to be calibrated, making
AAA a deterministic plug-in test-time defense with improvements in calibration and accuracy bycosting negligible computation overhead.
Compared to alternative defenses, AAA is effective inmitigating SQAs according to our study on 8 defenses, 6 SQAs, and 8 DNNs under various settings.
As a defense in real-world applications, AAA greatly mitigates the adversarial threat without requiringhuge computational burden.
For example, in autonomous driving or supervision systems, the postprocessing defense module could be directly implemented in pre-trained models.
Despite the lowcost, the benefits of adopting AAA are profound.
In most cases, users would be provided with a moreaccurate confidence score so that they know better when the model tends to fail.
In adversarial cases,
SQAs, the most threatening attack in real cases, would be effectively prevented.
AAA is developed to especially prevent SQAs.
Thus, defending our types of attacks is beyond ourscope.
For example, AAA does not improve the worst-case robustness evaluated in white-box settings[16, 46] where attackers have complete knowledge of the model (the AutoAttack [16] robust accuracywould be increased in an undesirable manner by AAA).
Also, AAA is not applicable to avoidingtransfer-based attacks and decision-based query attacks, which are either unfeasible or inefficient inthe real world, because AAA induces only a negligible impact on the decision boundary.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See the third paragraph in Sec. 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
the second paragraph in Sec. 6 for potential impacts.
Negative ones have not been
identified since we aim at defense.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
Our contribu-
[Table]
tions are mostly empirical.
(b) Did you include complete proofs of all theoretical results?
[N/A]
Our contributions are
mostly empirical.
If you ran experiments...
[Table]
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] All the codewould be released with detailed instructions and annotations.
Core code of AAA is
shown in Appendix D.
The data are described in experimental setup in Sec. 5.1.
(b) Did you specify all the training details ( e.g. , data splits, hyperparameters,
how they
were chosen)?
[Yes] Details are presented in Sec.
5.1 and Appendix E
(c) Did you report error bars ( e.g. , with respect to the random seed after running exper- iments multiple times)?
[Yes] Error bars and computation amount are reported in
Appendix C.
(d) Did you include the total amount of compute and the type of resources used ( e.g. , typeof GPUs, internal cluster, or cloud provider)?
[Yes] Our devices are described in Sec.5.1, and the amount of computation is quantified in Appendix C.
If you are using existing assets ( e.g. , code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We state the source
of DNNs
in experimental setup in Sec. 5.1.
(b) Did you mention the license of the assets?
[Yes] Most used code assets are under MIT
License.
Square attack is under BSD 3-Clause License.
ImageNet is under a customresearch and non-commercial license.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We would release our code as an asset.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] Our empirical studies are based on public datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
Our used public datasets generally do not
contain personally identifiable information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A] Our experiments involve no human subjects and crowdsourcing.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A] Our experiments involve no human
subjects or crowdsourcing.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A] Our experiments involve no human subjects
or crowdsourcing.