Adversarial Attack on Attackers: Post-Process to Mitigate Black-Box Score-Based Query Attacks
Anonymous Author(s)
AffiliationAddress email
Abstract
The score-based query attacks (SQAs) pose practical threats to deep neural networks by crafting adversarial perturbations within dozens of queries, only usingthe model’s output scores.
Nonetheless, we note that if the loss trend of the outputsis slightly perturbed, SQAs could be easily misled and thereby become much lesseffective.
Following this idea, we propose a novel defense, namely Adversarial Attack on Attackers (AAA), to confound SQAs towards incorrect attack directions byslightly modifying the output logits.
In this way, (1) SQAs are prevented regardlessof the model’s worst-case robustness; (2) the original model predictions are hardlychanged, i.e. , no degradation on clean accuracy; (3) the calibration of confidencescores can be improved simultaneously.
Extensive experiments are provided toverify the above advantages.
For example, by setting ℓ ∞ = 8 / 255 on CIFAR-10,our proposed AAA helps WideResNet-28 secure 80 .
59% accuracy under Squareattack ( 2500 queries), while the best prior defense ( i.e. , adversarial training) onlyattains 67 .
Since AAA attacks SQA’s general greedy strategy, such advantagesof AAA over 8 defenses can be consistently observed on 8 CIFAR-10/
ImageNetmodels under 6 SQAs, using different attack targets and bounds.
Moreover, AAAcalibrates better without hurting the accuracy.
Our code would be released.
Introduction
Deep Neural Networks (DNNs) are vulnerable to adversarial examples (AEs), where humanimperceptible perturbations added to clean samples can fool DNNs to give wrong predictions [1, 2].
Recently, such a threat is made practically feasible by the black-box score-based query attacks (SQAs)[3, 4, 5], as they only require the same information as users to craft efficient AEs.
Users, for betterjudgments, need the model’s prediction confidence indicated by DNNs’ output scores, which is theonly knowledge needed by SQAs to perform attacks.
In contrast, white-box attacks [6, 7, 8] ortransfer-based attacks [9, 10] require the gradients or training data of DNNs.
Moreover, it has beenshown that SQAs can achieve a non-trivial attack success rate by a reasonable number of queries,e.g. , dozens, compared to thousands of queries for decision-based query attacks [11, 12, 13].
Thus,such feasibility and effectiveness of SQAs are attracting increasing attention from defenders [14].
Defending against SQAs is a different goal compared to improving the worst-case robustness ascommonly studied [15, 16, 17].
Because in real-world scenarios which SQAs are designed for,
DNNs are treated as black boxes, in which the only interaction between models and users/attackers isthe model’s output scores.
Thus, altering the scores is all defenders could do here, either in direct orindirect ways.
Most existing defenses indirectly change outputs by optimizing the model [8, 18, 19]or pre-processing the inputs [14, 20], which, however, severely affect models’ normal performancefor different reasons.
Training a robust model, e.g. , by adversarial training [8, 21], diverts the model’sattention to learning AEs, yielding the so-called accuracy-robustness trade-off [22].
Randomizing
[Figure]
Specifically, if we perturb the original loss curve to the orange one, then attackers that trying to decrease the loss would be mostly cheated away from their true destiny, i.e. , the adversarial direction.
[14, 23, 24, 25, 26] and blurring [20, 27, 28, 29] inputs reduce the signal-noise ratio, inevitablyhurdling accurate decision.
Dynamic inference [19, 30] is time-consuming due to the test-timeoptimization on model.
Thus, it is imperative to develop a user-friendly defense against SQAs.
In thispaper, we hereby consider a post-processing defense as demonstrated in Fig.
1 (a), which naturallyenjoys the benefits as follows: (1) model’s decision is hardly affected since good predictions havealready been obtained; (2) model calibration can be simultaneously improved via post-processing[31, 32] to output accurate prediction confidence; (3) it can be flexibly used as a plug-in module forpre-trained models with negligible test-time computation overhead.
Despite these merits, it remainsunexplored that by post-processing,
How to serve users while avoiding SQA attackers when they access the same output information?
Since SQAs are black-box attacks that find the adversarial direction to update AEs by observing theloss change indicated by DNN’s output scores, we can perturb such scores directly to fool attackersinto incorrect attack tracks.
Following this idea, we propose the adversarial attack on attackers (AAA),which reverses the loss trend so that attackers, trying to greedily update AEs following the originaltrend, are led to the non-adversarial path farther away from the decision boundary as shown in Fig. 1(b).
Specifically, AAA directly optimizes DNN’s logits to control the output loss to approximate thereversed curve, which is orange in Fig.
1 (b), where one could see that attackers that are threateningto the undefended model (going adversarial by reducing loss) would be attacked to help the defended
DNN predict (going non-adversarial by reducing loss).
Now that a post-processing module is adoptedhere, we could simultaneously use it to calibrate the model as commonly proposed [31, 33].
Thus, asimple plug-in post-module is all you need to both fool attackers and offer users accurate confidence.
By post-processing, AAA not only lowers the calibration error without hurting accuracy in all cases,but also efficiently prevents SQAs, e.g. , helping a WideResNet-28 [34] on CIFAR-10 [35] secure80 .
59% accuracy under Square attack [5] ( 2500 queries), while the best prior defense ( i.e. , adversarialtraining [36]) only attains 67 .
Because AAA attacks the general greedy update of SQAs, suchadvantages of AAA over 8 baseline defenses can be consistently observed on 8 CIFAR-10/
ImageNetmodels under 6 SQAs, using different attack targets, norms, bounds, and losses, verifying AAA auser-friendly, effective, and generalizable defense.
Contributions
• We analyze current defenses from the view of SQA defenders and point out that a post-
processing module forms not only effective but also user-friendly and plug-in defenses.•
We design a novel adversarial attack on attackers (AAA) defense that fools SQA attackersto incorrect attack directions by slightly perturbing the DNN output scores.
• We conduct comprehensive experiments showing that AAA outperforms the other 8 defensesin the accuracy, calibration, and protection from all tested 6 SQAs under various settings.
Related work
Query attacks are black-box attacks that only require the model’s output information.
Queryattacks can be divided into score-based query attacks (SQAs) [3, 4, 5, 37, 38, 39, 40] and observing the loss change indicated by DNN’s output scores, i.e. , logits or probabilities.
Early SQAstry to estimate DNN’s gradients by additional queries around the sample [38, 42].
Recently, it isvalidated query-efficient to perform fast pure random-search SQAs [4, 5].
It has also been proposedto use pre-trained surrogate models in SQAs [39, 43], which, however, demands unfeasible accessto DNN’s training sample as in transfer-based attacks [9, 10].
Thus, we do not focus on defendingsuch SQAs as also in [14].
Besides SQAs, DQAs rely only on DNN’s decisions, e.g. , the top-1predictions, to generate AEs.
Since DQAs could not perform the greedy update, they start crafting the
AE from a different sample and keep DNN’s prediction wrong during the attack.
Currently, DQAsneed thousands of queries to reach a non-trivial attack success rate [44] compared to dozens of timesfor SQAs [5] as compared in Appendix A, limiting their threats.
Additionally, SQAs are mostlyapplicable and attackers do not have to resort to DQAs because, in the real world, DNNs must notonly be accurate but also indicate when they tend to be incorrect [31] by outputting confidence scores.
Adversarial defense mainly has two different goals.
The first goal is to improve DNN’s worst-caserobustness [17], which demands DNNs to have no AE around clean samples bounded by a normball.
Such robustness is originally evaluated by gradient-based white-box attacks [6, 7], but later,gradient obfuscation phenomenon is discovered [15], motivating evaluations to incorporate randomnoise and black-box attacks [16] in adaptive methods [16, 45, 46].
In this assessment, adversarialtraining (AT) [8, 21] is validated as the most effective defense, and other methods, e.g. , using moreor augmented data [47, 48], designing special architecture [18, 49, 50], and inducing randomnessin training [51, 52], all need collaborations with AT to achieve good performance.
Another defensegoal is to mitigate real-case adversarial threats, and the defense performance is evaluated by thefeasible and query-efficient SQAs.
A more robust model in the worst cases certainly protects itselfin real cases.
Besides, it is also possible to defend by dynamic inference [19, 30], and randomizing[14, 23, 24, 25], denoising [20, 27] or quantifying [28, 29] inputs.
However, all aforementioneddefenses on inputs or models exert a non-negligible impact on accuracy, calibration, or inferencespeed due to the focus on learning AEs in AT, the reduction of signal-noise ratio in pre-processing, orthe costly test-time optimization in dynamic inference as compared meticulously in Section 3.
Model calibration performance is a higher demand for a good model besides high accuracy.
Besides correct predictions, model calibration additionally requires DNNs to produce accurateconfidence in their predictions [53].
For instance, exactly 80% of the samples predicted with 80%confidence should be correctly predicted.
In this point of view, the expected calibration error (ECE)[32] is widely used to quantify the error between accuracy and confidence.
Various methods have beenproposed to improve DNN’s calibration in training [33, 54, 55, 56] or by a post-processing moduleafter training [31, 32, 57].
Among them, temperature scaling [32] is a simple but effective method[31], which divides all output logits by a single scalar tuned by a validation set so that the ECE intesting samples would be significantly reduced.
Since division is a simple post-processing operation,calibration could be simultaneously achieved by AAA, the first post-processing defense.
There havebeen methods to avoid attacks by calibration [56] or calibrate by attack [33], but the simultaneousimprovement of defense and calibration has not been reported to the best of our knowledge.
Preliminaries and motivation
Before presenting the proposed method, we first introduce the key ideas of SQAs and analyzeexisting defenses.
For a [R3.3] sample x labelled by y , an SQA on a DNN f generates an AE x ′ byminimizing the margin between logits [4, 5, 40, 58] as
[Equation]
namely the margin loss [7].
For defenders without the label y , it is possible to calculate the unsupervised margin loss based only on the logits vector z
= f ( x ) as [R3.3]
[Equation]
by assuming the model prediction ˆ y to be correct, since handling originally misclassified samplesfalls beyond the scope of defense [17].
For attackers with label y , it can be known that the attacksucceeds if L ( f ( x ) , y ) < 0 .
To quickly realize this, SQAs only update AEs if a query x q has a lowermargin loss compared to the current best query x k , i.e. ,
[Equation]
Existing SQAs [4, 5, 40] are quite effective, capable of halving the accuracy within 100 queries on
CIFAR-10 when ℓ ∞ = 8 / 255 , posing significant and practical threats.
Aware of this, the first defensespecifically against SQAs has been recently proposed [14] to pre-process inputs by random noise.
Besides, other existing defenses are also useful to avoid SQAs, among which the most representativeones are adversarial training (AT) [8, 21], dynamic inference [19, 30], and pre-processing [14, 28].
These defenses work in different mechanisms as illustrated in Fig.
1 (a) and Section 2.
Here in Table1, we present a comparison of their key characteristics related to the defense performance.
[Table]
accuracy = ↓↓ ↓ =
= calibration ↑ / ↓ ↓ ↑ testing cost =
=
= ↑↑↑
= training cost =
↑↑↑ =
= =
acc under SQA ↑ ↑↑ ↑ ↑ ↑↑↑
Real-case defenders are expected to serve users and meanwhile avoid SQA attackers.
The formerdemands good accuracy, calibration, and inference speed.
While the latter requires good protection,and preferably, no additional training (denoted as " = ").
As a training-time defense on model, ATexerts a significant impact on accuracy [22], e.g. , > 10% accuracy drop on ImageNet [49], with severalfold training computation and an indefinite impact (denoted as "/") on calibration.
Pre-processingis a test-time defense on input, which avoids SQAs by reducing the input’s signal-noise ratio butinevitably hurts normal performance.
Recently, the dynamic inference is proposed as a test-timedefense by optimizing the model, which shows no accuracy drop but dramatically increases thecomputation, e.g. , DENT [19] consumes > 700% test-time calculation.
Given the above discussion, itis imperative to develop a user-friendly and efficient method to effectively defend against SQAs.
Adversarial attack on attackers
Defending by post-processing naturally carries the advantages of good accuracy and calibration[31], no additional training and negligible test-time computation.
Although it does not improve thecommonly-studied worst-case robustness [16], its potential to defeat SQAs in real cases has not beenexplored yet to the best of our knowledge.
By investigating current defenses from the perspective of
SQA defenders in Sec.
3, it is interesting to find that in the real world, no matter whether defendersalter inputs, models, or outputs, what users/attackers get are just the changes in outputs, i.e. , theblack-box setting.
Thus, it is reasonable and preferable for SQA defenders to directly manipulate
DNN’s output scores, which has already become a standard practice in model calibration [31].
But how?
Now that attackers conduct the adversarial attack on the model, why cannot we defendersactively perform an adversarial attack on attackers?
If attackers search for the adversarial directionby scores, we can manipulate such scores to cheat them into an incorrect attack path.
Following thisidea, we develop the adversarial attack on attackers (AAA) to control what the SQA attackers basetheir actions on, i.e. , the loss (1) indicated by the DNN’s output scores.
Under such direct misleading,
SQA attackers, following their original policy, would be guided to wherever we want them to godue to their greediness, seeing (3).
Therefore, we can adopt the extreme fooling that leads attackersto walk opposite towards their destiny, i.e. , the non-adversarial direction.
In this way, attackers areattacked to actually help DNNs become more confident in predictions.
Reversing the attacker’s path by directly altering outputs is simple.
Since SQA attackers observe themargin loss (1), what we should do is force the modified loss to increase when the original unmodifiedloss actually decreases.
This can be achieved by a one-to-one mapping between unmodified andmodified loss values, e.g. , outputting large margin loss (high confidence) when the original marginis small (model is not confident).
However, this naive method prevents users from getting accurateprediction confidence since it greatly changes scores to reverse the overall loss trend along theadversarial direction.
Thus, we alternatively reverse the loss in a small interval while keeping theoverall trend unchanged as in Fig.
1 (b).
This design cheats attackers in most cases, but only requiresslight modifications on scores so that the precision of confidence is largely preserved.
Specifically, we periodically set loss attractors and encourage the DNN’s unsupervised margin loss(2) to move towards its closest attractor [R1.2, R3.1, R3.3]
[Equation]
where l org =
L u ( z org ) is the unmodified loss, and z org means the unmodified logits.
τ is the periodof attractor intervals, a constant hyper-parameter set by defenders, and "floor" denotes roundingdown decimals to integers.
For example, if τ = 4 , then according to (4), the closest loss attractorof logits with l org =
→ 4 is l atr = 2 .
As we can see, the 1 / 2 bias is necessary to avoid settingan attractor on l atr = 0 , i.e. , the decision boundary, because approximating the logits to it may flipmodel’s decision.
Periodically-set loss attractors divide logits with different l org into intervals, so that we could reversethe loss trend in each of the interval to cheat attackers into the opposite attack direction as in Fig. 1(b).
However, reversing the trend demands more than mapping in-interval losses to a fixed value l atr .
It requires mapping a small loss with a large one.
Thus, we set the target loss value l trg for logits as
[Equation]
That is, if the unmodified loss l org is larger than l atr by l diff , we hope the modified loss l trg to besmaller than l atr by α × l diff .
In this way, as a sample is actually close to the decision boundary ( l orgdecreases), AAA outputs an increasing l trg to mislead attackers away from this adversarial direction.
Although the periodic design has already altered output confidence very slightly, we could also stepfurther to improve the precision of confidence because post-processing logits is a standard practice inmodel calibration [32, 57].
Thus, simultaneous defense and calibration is obtainable in a single AAAmodule by controlling the loss while encouraging the output confidence σ ( z ) = max( softmax ( z ))to approach the calibrated one p trg as
[Equation]
where the first item requires the output loss to form the reversed curve so that attackers would becheated to non-adversarial directions, while the second item motivates the output confidence to beaccurate, and β controls the balance between them.
Through this design, DNN’s output could be bothaccurate and misleading, achieving two seemingly contradictory goals at the same time.
The calibrated confidence p trg is obtainable by various model calibration methods [32, 57].
Amongthem, temperature scaling [32] is simple but effective [31], which divides all logits by a scalar T asp trg = σ ( z org /T ) .
The temperature T is tuned by a validation set to minimize the calibration error,and here, such tuning is conducted when the optimization (6) is also performed for κ iterations sothat we can find the temperature that suits AAA best.
After that, T is fixed for inference in AAApost-processing module.
Input: the logits z org , T , τ, α, β, κ .
Output: post-processed logits z 1: calculate original loss l org =
L u ( z org ) by (2) 2: set target loss l trg by (5) 3: set target confidence p trg = σ ( z org /T ) 4: initialize z = z org and optimize it for (6) 5: return z
We summarize our algorithm in Alg. 1.
AAAfirst calculates the original loss l org , according towhich sets the target loss l trg that forms the reversed loss curve.
Then AAA optimizes the logitsto reach l trg , and also the target confidence p trg ,which is obtained by dividing the original logitsusing a pre-tuned temperature T .
The overall procedure is in test-time with only optimization onthe logits, making AAA a computation-efficient,plug-in, and model-agnostic method with goodaccuracy, calibration, and defense against SQAs.
Experiments
5.1 Setup
We evaluate AAA along with 8 defense baselines, including random noise defense (RND [33]),adversarial training (AT [36, 59, 49]), dynamic inference (DENT [19]), training randomness (PNI[51]), and ensemble (TRS [60]).
Results of AT with extra data [22]) are in Appendix A. AAAuses hyper-parameters α = 1 , τ = 6 , β = 5 , and an Adam optimizer [61] with learning rate
1 , β 1 = 0 .
9 , β 2 = 0 .
999 , and the iteration of optimization 100 .
To calibrate simultaneously by
AAA, we perform temperature scaling using 1K/5K samples for CIFAR-10 testing set / ImageNetvalidation set respectively to and make them disjoint with the testing samples in attack as much aspossible.
Detailed hyper-parameters of other defenses and attacks are in Appendix E.
The defenses are assessed by 6 state-of-the-art SQAs, which are random search methods including
Square [5], SignHunter [40], and SimBA [4], and gradient estimation methods including NES [58],and Bandits [3].
[R3.3] Square attack [5] selects a decreasing size of random square perturbations.
SignHunter [40] estimates the sign of gradient and flips corresponding perturbations.
SimBA[4] randomly samples perturbations given an orthonormal basis.
NES [58] adopts the Natural
Evolutionary Strategies to for gradient estimation.
Bandits [3] jointly uses the time and datadependent gradient prior.
Some SQAs use pre-trained models [39, 43], which, however, demandsunfeasible access to DNN’s training sample.
To evaluate real-case defenses against training-basedattacks, we alternatively consider a practical SQA QueryNet [62] based on model stealing.
Results of
DQAs and SQAs using other than margin loss are put in Appendix A. Since AAA aims to defend inreal cases without resorting to randomness, it would be unsuitable to evaluate AAA by the commonadaptive or white-box attacks [16], e.g., EOT [15].
We mostly perform untargeted ℓ ∞ attacks, but wealso test the targeted and ℓ 2 attacks under different bounds to observe the generalization of defenses.
We use 8 DNNs, which are mostly WideResNets [34] as in [17].
The pre-trained models of PNI [51] /
TRS [60] are ResNet-20 [63], and we also test ResNeXt-101 [64] in ImageNet.
Other studied DNNscome from RobustBench [17] and torchvision [65] as specified in Appendix E. AT models are testedusing the same bound as in AT, if not otherwise stated.
We use all 10K CIFAR-10 testing samples.
For ImageNet, we randomly-select 1K validation samples from all 1K classes respectively (1 imagein 1 class) to eliminate the class bias as in [62].
All images are rescaled to [0 , 1] , and the ImageNetones are resized to 224 × 224 .
Before feeding them to DNNs, we quantify images to 8-bit, imitatingthe real-case 8-bit image setting as in [62].
Experiments are performed on an NVIDIA Tesla A
GPU but could be run on any GPU with over 4GB memory.
As defenders, we are concerned about DNN’s remaining accuracy after it being attacked by SQAsfor a certain query times.
Thus, we report such SQA adversarial accuracy after 100 and 2500queries, reflecting DNN’s performance under mild and extreme SQAs.
We include the average querytimes, another metric commonly used by attackers, in Appendix C. To measure the calibration, theexpected calibration error (ECE) [32] is commonly used.
ECE divides all N testing samples into Mbins, and each bin contains samples with confidence ranging from the m th quantile to the ( m + 1) thquantile of [0 , 1] .
Then ECE is calculated by the difference between accuracy and confidence as
[Equation]
, where ˆ y i is the predicted label of the i th sample inthe m th bin B m , and ˆ p i ˆ y i represents the probability confidence of this prediction.
[5] ( ℓ ∞ = 8 / 255 ) using the 9953 th CIFAR-10 test sample (other samples have similar trends as in
Appendix B).
AAA fools attackers precisely as shown by the tiny symmetric oscillations of two lines.
5.2 Visual illustration of AAA
We first visually illustrate the mechanism and effects of AAA.
AAA, besides fooling attackers, alsocalibrates the model, forming the loss curve as in Fig.
Slightly different from the ideal defensein Fig.
1 (b) is that the orange curve is lowered for calibration.
However, calibration does nothurdle fooling attackers, i.e. , the orange line is mostly going opposite to the blue line in a preciselysymmetric manner.
Also, two lines cross the decision boundary at the same time, meaning that AAAdoes not change the decision.
The right figure shows DNN’s accuracy under Square attack [5] when ℓ 2 = 2 . 5 , indicating that AAA outperforms alternatives in defending SQAs by a large margin with also the highest clean accuracy.
Besides the loss, it is also necessary to observe the output probabilities of AAA, which is displayedin Fig.
3 (left).
AAA, indicated by the orange lines, changes the scores slightly without hurtingaccuracy (dotted lines) compared to RND [33] and AT [36].
By such small modifications, AAA notonly improves the calibration but also prevents SQAs by misleading them into incorrect directions.
As displayed by the right part of Fig.
3, AAA is super effective in avoiding SQAs, even if it is aquery-efficient attack working at a large bound.
Specifically, AAA preserves a standard trained DNNto have > 60% accuracy after 2500 queries, doubling the performance of AT and tripling that for RND.
[Table]
ResNetSimBA [4] 53.04 / 03.
NES [58] 83.42 / 12.
Bandit [3] 69.86 / 41.
ImageNet
ECE (%) 5.
ℓ ∞ =
Acc (%) 77.
Square [5] 52.27 / 09.
WideSignHunter [40] 53.05 / 13.
ResNetSimBA [4] 71.79 / 20.
NES [58] 77.11 / 64.
Bandit [3] 71.33 / 65.
ImageNet
ECE (%) 8.
ℓ ∞ =
Acc (%) 78.
Square [5] 54.51 / 11.
ResSignHunter [40] 54.12 / 13.
NeXtSimBA [4] 70.86 / 24.
NES [58] 78.21 / 66.
Bandit [3] 72.19 / 67.
5.3 Numerical results of AAA
We report the main numerical results in Table 2, where RND and AAA are directly implementedin the undefended model denoted as "None".
The AT method for the three models are PSSiLU[36], vanilla AT [59], and feature denoising [49], respectively.
According to Table 2, AAA not onlyconsistently reduces ECE by > 12% , but also does not hurt clean accuracy.
In contrast, the AT modelslose > 7% accuracy and RND also endures a drop in accuracy and calibration.
Under SQAs, AAApreserves a significantly higher accuracy than the undefended model, which is totally destroyed.
Themost threatening SQAs in two datasets more than halve the CIFAR-10 model’s accuracy within 100queries and degrade ImageNet ones to < 15% by 2500 times.
However, the AAA model remains> 75% and > 55% accuracy in extreme cases.
AT and RND are useful in mitigating SQAs, but it is
AAA that tops the defense performance in almost all cases.
Besides AT and RND, diverse defenses have also been proposed, and it would be interesting to seetheir results, which we report in
Appendix F.
The conclusion is that they are not comparable to AAAin real cases regarding the accuracy, calibration, and defense performance.
5.4 Generalization of AAA
Aside from the untargeted ℓ ∞ attacks, we also conduct targeted attacks, ℓ 2 attacks under differentbounds to study the generalization of AAA.
A targeted attack is successful only if an AE is mispredicted as a pre-set class, which, is randomly chosen from incorrect classes for each sample here.
Andℓ 2 attacks bound the perturbations by ℓ 2 norm, which is reported to fool DNNs better [10].
Here weadditionally validate AAA’s plug-in advantage by combining it with AT.
Note that what differs from
AAA and most existing defenses [49, 50, 51] combined with AT is that AAA has already achievedexcellent defense performance, so such a combination is feasible but not necessary.
We choose
Square attack [5] to perform the above evaluations because it is the most effective SQA as testedin Table 2.
The results are presented in Table 3, where all models are exactly the same as in Table2 (CIFAR-10) without tuning hyper-parameters of defense to fairly evaluate the generalization ofdifferent methods.
The results of ℓ 2 AT model and other ℓ ∞ AT models are put in Appendix A.
In the difficult targeted attack setting, the undefended model remains only 2 .
84% accuracy after 2500queries, which is approximately just the accuracy drop of the AAA model.
For ℓ 2 attacks, AAAis still capable of mitigating threats without hurting users, and its superiority is more outstandingas the attack bound becomes larger.
AT models, although robust, suffer from attacks under a largeor different norm ball [56].
Thus, its defense effects decrease as SQA alters the setting, seeing thebottom line.
Defended by AAA, however, this drawback would be greatly avoided.
An ℓ ∞ AT model,even under ℓ 2 = 2 . 5 attack after 2500 queries, is hardly influenced, i.e. , increasing query times after100 queries hardly better the attack performance, discouraging SQA attackers.
[Table]
untargeted
[Table]
untargeted
[Table]
5.5 Time consumption of AAA
[R1.3, R2.2]
We study the influence of optimization times τ in AAA to see the balance betweenthe amount of computation and defense performance.
AAA’s runtime is estimated by inferring10K CIFAR-10 samples in a WideResNet28 model.
The results are shown in Table 4.
As we see,optimizing low-dimensional logits is not costly, and good defense and calibration results are alsoobtainable by 60-80 iterations, which costs less time.
Since optimizing logits is independent of modelsize, model owners could determine AAA runtime precisely.
[Table]
A study of other hyper-parameters has been put in
Appendix G.
An intuitive conclusion is that AAAis insensitive to even exponentially tuned hyper-parameters.
And a larger α , a larger τ , or a smaller βthat highlights defense more v.s. calibration thereby increases the adversarial accuracy and ECE.
5.6 Adaptive attacks of AAA
[R2.1, R3.2] Designing adaptive attacks, though possible, is costly and easy to bypass in real-worldscenarios.
Because here, attackers and defenders are in a double-blind relationship, i.e., attackers donot know the model, including the defense strategy.
Thus, the discovery process of defense strategiesfor developing adaptive attacks would require additional queries and creative deductions.
After that,attackers also have to devote considerable manual efforts to figure out what defenders actually do.
Inthis regard, AAA imposes a great hurdle to adaptive attackers.
But still, it is interesting to see that after unveiling defender’s strategy by great effort, how adaptiveattackers could break the AAA model.
There are two straightforward adaptive attacks against AAA:going bidirectional or opposite to the original attack direction, which are both based on the exceptionalloss change that attackers observe to decide the update direction.
However, adaptive attackers arealso foolable because the dramatic drop of loss across intervals could be smoothed, e.g. by using asine function to design the target loss as l trg
= l atr − α × sin ( π (1 − 2( l org − l atr ) /t )) .
In this way,neither direction of search is likely to figure out the defense strategy, seeing results below.
[Table]
The bidirectional / opposite search weakens
AAA from 81.36% to 62.91% / 57.31%, butthe sine-like function that ascends and descends the loss along the original attack direction fools both non-adaptive and adaptiveattacks.
By designing sine l trg , the defenseperformance for the regular attack is kept at 79.78% while that under bidirectional / opposite searchis improved to 75.36% / 75.23%.
Thus, AAA is effective in mitigating adaptive SQAs.
Conclusion, impacts and limitations
We develop a novel defense against score-based query attacks (SQAs).
Our main idea is to activelyattack the attackers (AAA), misleading them into incorrect attack directions.
AAA achieves that bypost-processing DNN’s logits while enforcing the new output confidence to be calibrated, making
AAA a deterministic plug-in test-time defense with improvements in calibration and accuracy bycosting negligible computation overhead.
Compared to alternative defenses, AAA is effective inmitigating SQAs according to our study on 8 defenses, 6 SQAs, and 8 DNNs under various settings.
As a defense in real-world applications, AAA greatly mitigates the adversarial threat without requiringhuge computational burden.
For example, in autonomous driving or supervision systems, the postprocessing defense module could be directly implemented in pre-trained models.
Despite the lowcost, the benefits of adopting AAA are profound.
In most cases, users would be provided with a moreaccurate confidence score so that they know better when the model tends to fail.
In adversarial cases,
SQAs, the most threatening attack in real cases, would be effectively prevented.
AAA is developed to especially prevent SQAs.
Thus, defending our types of attacks is beyond ourscope.
For example, AAA does not improve the worst-case robustness evaluated in white-box settings[16, 46] where attackers have complete knowledge of the model (the AutoAttack [16] robust accuracywould be increased in an undesirable manner by AAA).
Also, AAA is not applicable to avoidingtransfer-based attacks and decision-based query attacks, which are either unfeasible or inefficient inthe real world, because AAA induces only a negligible impact on the decision boundary.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See the third paragraph in Sec. 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See the second paragraph in Sec. 6 for potential impacts.
Negative ones have not been identified since we aim at defense.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
Our contribu-
tions are mostly empirical.
(b) Did you include complete proofs of all theoretical results?
[N/A]
Our contributions are
mostly empirical.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] All the codewould be released with detailed instructions and annotations.
Core code of AAA is
shown in Appendix D.
The data are described in experimental setup in Sec. 5.1.
(b) Did you specify all the training details ( e.g. , data splits, hyperparameters,
how they
were chosen)?
[Yes] Details are presented in Sec.
5.1 and Appendix E
(c) Did you report error bars ( e.g. , with respect to the random seed after running experiments multiple times)?
[Yes] Error bars and computation amount are reported in
Appendix C.
(d) Did you include the total amount of compute and the type of resources used ( e.g. , type of GPUs, internal cluster, or cloud provider)?
[Yes] Our devices are described in Sec.5.1, and the amount of computation is quantified in Appendix C.
If you are using existing assets ( e.g. , code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We state the source
of DNNs
in experimental setup in Sec. 5.1.
(b) Did you mention the license of the assets?
[Yes] Most used code assets are under MIT
License.
Square attack is under BSD 3-Clause License.
ImageNet is under a custom
research and non-commercial license.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We would release our code as an asset.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] Our empirical studies are based on public datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
Our used public datasets generally do not
contain personally identifiable information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A] Our experiments involve no human subjects and crowdsourcing.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A] Our experiments involve no human
subjects or crowdsourcing.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A] Our experiments involve no human subjects
or crowdsourcing.