W ITH F ALSE F RIENDS L IKE T
HESE , W HO C
AN H AVE
S ELF -K NOWLEDGE ?
Anonymous authors
Paper under double-blind review
A BSTRACT
Adversarial examples arise from excessive sensitivity of a model.
Commonly studied adversarial examples are malicious inputs, crafted by an adversary from correctly classiﬁed examples, to induce misclassiﬁcation.
This paper studies an intriguing, yet far overlooked consequence of the excessive sensitivity, that is, a misclassiﬁed example can be easily perturbed to help the model to produce correct output.
Such perturbed examples look harmless, but actually can be maliciously utilized by a false friend to make the model self-satisﬁed.
Thus we name them hypocritical examples .
With false friends like these, a poorly performed model could behave like a state-of-the-art one.
Once a deployer trusts the hypocritical performance and uses the “well-performed” model in real-world applications, potential security concerns appear even in benign environments.
In this paper, we formalize the hypocritical risk for the ﬁrst time and propose a defense method specialized for hypocritical examples by minimizing the tradeoff between natural risk and an upper bound of hypocritical risk.
Moreover, our theoretical analysis reveals connections between adversarial risk and hypocritical risk.
Extensive experiments verify the theoretical results and the effectiveness of our proposed methods.
I NTRODUCTION
Deep neural networks (DNNs) have achieved breakthroughs in a variety of challenging problems such as image understanding (Krizhevsky et al., 2012), speech recognition (Graves et al., 2013), and automatic game playing (Mnih et al., 2015).
Despite these remarkable successes, their pervasive failures in adversarial settings, the phenomenon of adversarial examples (Biggio et al., 2013;
Szegedy et al., 2014), have attracted signiﬁcant attention in recent years (Athalye et al., 2018;
Carlini et al., 2019; Tramer et al., 2020).
Such small perturbations on inputs crafted by adversaries are capable of causing well-trained models to make big mistakes, which indicates that there is still a large gap between machine and human perception, thus posing potential security concerns for practical machine learning (ML) applications (Kurakin et al., 2016;
Qin et al., 2019; Wu et al., 2020b).
An adversarial example is “an input to a ML model that is intentionally designed by an attacker to fool the model into producing an incorrect output” (Goodfellow & Papernot, 2017).
Following the deﬁnition of adversarial examples on classiﬁcation problems (Goodfellow et al., 2015; Papernot et al., 2016; Elsayed et al., 2018;
Carlini et al., 2019; Zhang et al., 2019; Wang et al., 2020b; Zhang et al., 2020; Tram`er et al., 2020), given a DNN classiﬁer f and a correctly classiﬁed example x with class label y
(i.e., f ( x ) = y ), an adversarial example x adv is generated by perturbing x such that f ( x adv ) (cid:54) =
y and x adv ∈ B (cid:15) ( x ) .
The neighborhood B (cid:15) ( x ) denotes the set of points within a ﬁxed distance (cid:15) > 0 of x , as measured by some metric (e.g., the l p distance), so that x adv is visually the “same” for human observers.
Then, an imperfection of the classiﬁer is highlighted by
G adv
=
Acc( D ) − Acc
( A ) , the performance gap between the accuracy (denoted by Acc( · ) ) evaluated on clean set sampled from data distribution D and adversarially perturbed set A .
An adversary could construct such a perturbed set A that looks no different from D but can severely degrade the performance of even state-of-the-art DNN models.
From direct attacks in the digital space (Goodfellow et al., 2015; Carlini & Wagner, 2017) to robust attacks in the physical world (Kurakin et al., 2016; Xu et al., 2020), from toy classiﬁcation problems (Chen et al., 2020; Dobriban et al., 2020) to complicated perception tasks (Zhang & Wang, 2019; Wang et al., 2020a), from the high dimensional nature of the input space (Goodfellow et al., 2015; Gilmer et al., 2018) to the
[Figure]
choose ResNet50 (He et al
, 2016a
(Russakovsky et al., framework of (non)-robust features (Jetley et al., 2018; Ilyas et al., 2019), many efforts have been devoted to understanding and mitigating the risk raised by adversarial examples, thus closing the gap
G adv .
Previous works mainly concern the adversarial risk on correctly classiﬁed examples.
However, they typically neglect a risk on misclassiﬁed examples themselves which will be formalized in this work.
In this paper, we ﬁrst investigate an intriguing, yet far overlooked phenomenon, where given a DNN classiﬁer f and a misclassiﬁed example x with class label y
(i.e., f ( x ) (cid:54) =
y ), we can easily perturb x to x hyp such that f ( x hyp ) =
y and x hyp ∈ B (cid:15) ( x ) .
Such an example x hyp looks harmless, but actually can be maliciously utilized by a false friend to fool a model to be self-satisﬁed.
Thus we name them hypocritical examples (see Figure 1 for a comparison with adversarial examples).
Adversarial examples and hypocritical examples are two sides of the same coin.
On the one side, a well-performed but sensitive model becomes unreliable in the existence of adversaries.
On the other side, a poorly performed but sensitive model behaves well with the help of friends.
With false friends like these, a naturally trained suboptimal model could have state-of-the-art performance, and even worse, a randomly initialized model could behave like a well-trained one (see Section 2.1). It is natural then to wonder: Why should we care about hypocritical examples? Here we give two main reasons:
This is of scientiﬁc interest. Hypocritical examples are the opposite of adversarial examples. While adversarial examples are hard test data to a model, hypocritical examples aim to make it easy to do correct classiﬁcation. Hypocritical examples warn ML researchers to
think carefully about high test accuracy: Does our model truly achieve human-like intelligence, or is it just simply because the test data prefers the model? There are practical threats. A variety of nefarious ends may be achievable if the mistakes of ML systems can be covered up by hypocritical attackers.
For instance, before allowing autonomous vehicles to drive on public roads, manufacturers must ﬁrst pass tests in speciﬁc environments (closed or open roads) to obtain a license (Administration et al., 2016; Briefs, 2015; Lei, 2018). An attacker may add imperceptible perturbations on the test examples (e g., the “stop sign” on the road) stealthily without human notice, to hypocritically help an ML-based autonomous vehicle to pass the tests that might otherwise fail.
However, the high performance can not be maintained on public roads without the help of the attacker.
Thus, the potential risk is underestimated and trafﬁc accidents might happen unexpectedly when the vehicle driving on public roads.
In such a case, if the examples used to evaluate a model are falsiﬁed by a false friend, the model will manifest like a perfect one (on hypocritical examples), but it actually may not be well performed even on clean examples, not to mention adversarial examples.
Thus a new imperfection of the classiﬁer can be found in G hyp =
Acc( F ) − Acc( D ) , the performance gap between the accuracy evaluated on clean set sampled from D and hypocritically perturbed set F .
Still, F looks no different from
D but can stealthily upgrade the performance.
Once a deployer trusts the hypocritical performance carefully designed by a false friend and uses the “well-performed” model in real-world applications, potential security concerns appear even in benign environments.
Thus we need methods to defend our models from false friends, that is, making our models have self-knowledge.
We propose a defense method by improving model robustness against hypocritical perturbations.
Speciﬁcally, we formalize the hypocritical risk and minimize it via a differentiable surrogate loss (Section 3).
Experimentally, we verify the effectiveness of our proposed attack (Section 2.1) and defense (Section 4.1).
Further, we study the transferability of hypocritical examples across models trained with various methods (Section 4.2).
Finally, we conclude our paper by discussing and summarizing our results (Section 5 and Section 6).
Our main contributions are:
• We give a formal deﬁnition of hypocritical examples.
We demonstrate the unreliability of standard evaluation process in the existence of false friends and show the potential security risk on the deployment of a model with high hypocritical performance.
• We formalize the hypocritical risk and analyze its relation with natural risk and adversarial risk.
We propose the ﬁrst defense method specialized for hypocritical examples by minimizing the tradeoff between the natural risk and an upper bound of hypocritical risk.
• Extensive experiments verify the effectiveness of our proposed methods.
We also examine the transferability of hypocritical examples.
We show that the transferability is not always desired by the attackers, which depends on their purpose.
F ALSE F RIENDS AND A DVERSARIES
Better an open enemy than a false friend!
Only by being aware of the potential risk of the false friend can we prevent it.
In this section, we expose a kind of false friends, who are capable of manipulating model performance stealthily during the evaluation process, thus making the evaluation results unreliable.
We consider a classiﬁcation task with data (
x , y ) ∈
R d × { 1 , . . .
, C } from a distribution D .
Denote by f : R d → { 1 , ..., C } the classiﬁer which predicts the class of an input example x : f ( x ) =
arg max k p k
( x ) , where p k ( x ) is the k th component of p ( x ) :
R d → ∆ C (e.g., the output after softmax activation), in which ∆ C = { u ∈ R C | 1 T u = 1 , u ≥ 0 } is the probabilistic simplex.
Adversarial examples are malicious inputs crafted by an adversary to induce misclassiﬁcation.
We ﬁrst give the commonly accepted deﬁnition of adversarial examples as follows:
Deﬁnition 1 (Adversarial Examples) .
Given a classiﬁer f and a correctly classiﬁed input ( x , y ) ∼
D (i.e., f ( x ) = y ), an (cid:15) -bounded adversarial example is an input x ∗ ∈ R d such that:
[Equation]
The assumption underlying this deﬁnition is that inputs satisfying x ∗ ∈ B (cid:15) ( x ) preserve the label y of the original input x .
The reason for the existence of adversarial examples is that a model is overly sensitive to non-semantic changes.
Next, we formalize a complementary phenomenon to adversarial examples, called hypocritical examples.
Hypocritical examples are malicious inputs crafted by a false friend to stealthily correct the prediction of a model:
Deﬁnition 2 (Hypocritical Examples) .
Given a classiﬁer f and a misclassiﬁed input ( x , y ) ∼ D (i.e., f ( x ) (cid:54) =
y ), an (cid:15) -bounded hypocritical example is an input x ∗ ∈ R d such that:
[Equation]
and x ∗ ∈ B (cid:15) ( x ) . The same as adversarial examples, hypocritical examples are bounded to preserve the label of the original input, and are another consequence that arises from excessive sensitivity of a classiﬁer.
As a false friend, a hypocritical example can be generated from a misclassiﬁed example by maximizing
[Equation]
which is equivalent to minimizing
[Equation]
) where 1 ( · ) is the indicator function.
Similar to Madry et al.
; Wang et al.
(2020b), in practice, we leverage the commonly used cross entropy (CE) loss as the surrogate loss of 1 ( f ( x (cid:48) )
(cid:54) =
y ) and minimize it by projected gradient descent (PGD).
Note that Equation 2 looks similar to but conceptually differs from the known targeted adversarial attack (Carlini & Wagner, 2017), which generates a kind of adversarial examples deﬁned on correctly classiﬁed clean inputs and targeted to wrong classes.
The hypocritical examples here are deﬁned on misclassiﬁed inputs and are targeted to their right classes.
2. A TTACK RESULTS
In this subsection [Table] we demonstrate the power of our proposed hypocritical attack on three benchmark datasets: MNIST (LeCun et al., 1998), CIFAR-10 (Krizhevsky et al., 2009) and ImageNet (Russakovsky et al., 2015)
[Table]
[Table]
[Table]
We attack models trained with standard approach using clean examples (Standard) and models that randomly initialized without training (Naive).
For MNIST, the hypocritically perturbed set F and the adversarially perturbed set A are constructed by attacking every example in the clean test set sampled from D .
Both attacks are bounded by a l ∞ ball with radius (cid:15)
= 0 .
For ImageNet, F and
A are constructed based on its validation set sampled from D .
Both attacks are bounded by a l ∞ ball with radius (cid:15)
= 16 / 255 .
For each experiment, we conduct 3 trials with different random seeds and report the averaged result to reduce the impact of random variations.
Appendix A.2 describes further experimental details about DNN architecture, training procedure and more results.
Results on MNIST and ImageNet are summarized in Table 1 and Table 2, respectively.
First, we ﬁnd that the naturally trained models are extremely sensitive to hypocritical perturbations (e.g., Standard (MLP) and Standard (LeNet) achieve 100% accuracy on hypocritically perturbed MNIST test set, and Standard (VGG16) and Standard (ResNet50) achieve 99% accuracy on hypocritically perturbed
ImageNet validation set).
Second, we ﬁnd that part of randomly initialized models is extremely sensitive (e.g., Naive (MLP) and Naive (VGG16) achieve 100% accuracy on F of MNIST and
ImageNet).
These results demonstrate the unreliability of standard evaluation process in the existence of false friends.
Once a “well-performed” model (such as Naive (MLP) or Naive (VGG16)) is permitted to be deployed in real-world applications due to that the deployer has a false sense of performance, potential security concerns appear even in benign environments.
It seems that the Naive (ResNet50) model is relatively robust to hypocritical examples on ImageNet.
But that is just a trivial defense, it simply predicts most of the points in input region as a certain class because of the poor scaling of network weights at initialization (He et al., 2016b; Elsayed et al., 2019).
More discussions are in Appendix A.2.
Therefore, it is not enough to blindly pursue robustness against hypocritical perturbations but ignore the performance on clean examples.
H YPOCRITICAL R ISK
In this section, we formalize the hypocritical risk and analyze the relation between natural risk, adversarial risk, and hypocritical risk.
We propose a defense method specialized for hypocritical examples by minimizing the tradeoff between natural risk and an upper bound of the hypocritical risk.
Moreover, by decomposing a existing method designed for adversarial defense (TRADES (Zhang et al., 2019)), we ﬁnd that, surprisingly, TRADES minimizes not only the adversarial risk on correctly classiﬁed examples, but also a looser upper bound of the hypocritical risk.
Our theoretical analysis suggests that TRADES can be another candidate defense method for hypocritical examples.
To characterize the adversarial robustness of a classiﬁer f , Madry et al.
; Uesato et al.
Cullina et al.
(2018) deﬁned the adversarial risk under the threat model of bounded (cid:15) ball:
[Equation]
The standard measure of classiﬁer performance, known as natural risk , is denoted as R nat ( f ) =
E ( x ,y ) ∼D [ 1 ( f ( x ) (cid:54) =
y )] .
Let q ( x , y ) be the probability density function of data distribution D .
We denote by S
+ f the conditional data distribution on correctly classiﬁed examples w.r.t.
f , with a conditional density function q ( x , y | E ) =
q ( x , y ) /
Z( E ) if E is true (otherwise q ( x , y | E ) = 0 ), where the event E is f ( x ) =
y and Z( E ) =
(cid:82) x ,y 1 ( f ( x ) =
y ) dq ( x , y ) is a normalizing constant.
We denote by S − f
the conditional data distribution on misclassiﬁed examples with the conditional density function q ( x , y | E ) and f ( x ) (cid:54) =
y as the event E .
Then we have the following relation between the natural risk and the adversarial risk:
Proposition 1.
Denote the adversarial risk on correctly classiﬁed examples by
[Equation]
then we have
[Equation]
Proposition 1 shows that we can view the adversarial risk R
adv ( f ) as the tradeoff between R nat ( f ) and ˆ R adv ( f ) with the scaling parameter λ = 1 −R nat ( f ) .
The adversarial risk on correctly classiﬁed examples ˆ R adv ( f ) is in sharp contrast to the hypocritical risk deﬁned on misclassiﬁed examples formalized as follows:
Deﬁnition 3 (Hypocritical Risk) .
The hypocritical risk on misclassiﬁed examples of a classiﬁer f under the threat model of bounded (cid:15) ball is deﬁned as
[Equation]
The hypocritical risk ˆ R hyp ( f ) is the proportion of perturbed examples (originally misclassiﬁed) that can be successfully correctly classiﬁed by the classiﬁer after a false friend’s attack.
When considering the existence of false friends, a good model should have not only low natural risk but also low hypocritical risk, to be robust against hypocritical perturbations.
3. T RADEOFF BETWEEN NATURAL AND HYPOCRITICAL RISKS
Motivated by the tradeoff between natural and adversarial risks Tsipras et al.
; Zhang et al.
(2019), we notice that there may also exist an inherent tension between the goal of natural risk minimization and hypocritical risk minimization. To illustrate the phenomenon, we provide a toy example here, which is modiﬁed from the example in Zhang et al. (2019), and its risk minimization solutions can be analytically found. Consider the case ( x, y ) ∈ R × {− 1 , +1 } from a distribution D , where the marginal distribution over the instance space is a uniform distribution over [0 , 1] , and for k = 0 , 1 , · · · , (cid:100)
[Equation]
See Figure 2 for visualization of η ( x ) .
In this problem, we consider two classiﬁers: a) the Bayes optimal classiﬁer sign(2 η
( x ) − 1) ; b) the all-one classiﬁer which always outputs “positive”.
Tabledisplays the trade-off between natural and hypocritical risks: the minimal natural risk 1 / 8 is achieved by the Bayes optimal classiﬁer with large hypocritical risk, while the optimal hypocritical risk 0 is achieved by the all-one classiﬁer with large natural risk.
[Table]
[Table]
[Table]
[Table]
hyp 0 (optimal)
3. U PPER BOUNDS OF HYPOCRITICAL RISK
It is natural then to optimize our models to minimize natural and hypocritical risks at the same time.
However, it’s hard to do optimization over ˆ R hyp ( f ) .
To ease the optimization obstacles in there, we derive the following upper bounds.
Theorem 1.
For any data distribution D and its corresponding conditional distribution on misclassiﬁed examples S − f w.r.t.
a classiﬁer f , we have
[Equation]
where x hyp = arg max x (cid:48) ∈B (cid:15) ( x ) 1 ( f ( x (cid:48) ) =
y ) and x rev = arg max x (cid:48) ∈B (cid:15)
[Equation]
x
[Equation]
f [Equation] x
[Equation]
= f ( x
[Equation]
Here x rev means that it pursues to reverse a clean example to a different class, from the point of view of the model.
The upper bounds found in Theorem 1 allow us to optimize the hypocritical risk using proper surrogate loss functions which are both physical meaningful and computaionally tractable.
Before moving forward to algorithmic design, we state a useful proposition below, which reveals the internal mechanism behind in TRADES.
Proposition
[Equation]
Proposition 2 shows a connection between adversarial risk and hypocritical risk: the adversarial risk on correctly classiﬁed examples
ˆ R adv ( f ) and the looser upper bound of the hypocritical risk on misclassiﬁed examples R hyp ( f ) can be seamlessly united to a new risk on all examples R rev ( f ) .
We name it reversible risk since minimizing it pursues the model whose predictions can’t be reversed by small perturbations.
3. A LGORITHMIC DESIGN
Now we are ready to design objective functions that improve model robustness against hypocritical examples while keeping model accuracy on clean examples.
Similar to Zhang et al. ; Wang et al.
(2020b); Tram`er et al. Raghunathan et al. (2020), we propose a defense objective by minimizing the tradeoff between the natural risk and the tighter upper bound of the hypocritical risk:
[Equation]
where λ > 0 is a tunable scaling parameter balancing the importance of natural risk and hypocritical risk.
We name our method THRM (Tradeoff for Hypocritical Risk Minimization).
Optimization over 0-1 loss in THRM is still intractable.
In practice, for the indicator function( f ( x ) (cid:54) =
y ) in R nat ( f ) , we adopt the commonly used CE loss as surrogate loss.
Observed that
[Equation]
( x hyp ) (cid:54) =
f ( x )) , we absorb the R nat ( f ) term into λ and use KL divergence as the surrogate loss of the indicator function 1 ( f ( x hyp ) (cid:54) =
f ( x ))
(Zheng et al., 2016;
Zhang et al., 2019; Wang et al., 2020b), since f ( x hyp ) (cid:54) =
f ( x ) implies that the perturbed examples have different output distributions to that of clean examples.
Our ﬁnal objective function for THRM becomes
[Equation]
Intuition behind the objective L THRM : the ﬁrst term in Equation 6 encourages the natural risk to be optimized, while the second regularization term encourages the output to be stable against hypocritical perturbations, that is, the classiﬁer should not be overly conﬁdent in its predictions especially when a false friend wants it to be.
To derive the objective function for TRADES, we can minimize the tradeoff between the natural risk and the reversible risk:
[Equation]
Similar to THRM, we use CE loss and KL divergence as the surrogate loss of 1 ( f ( x ) (cid:54) =
y ) and( f ( x rev ) (cid:54) = f
( x )) , respectively.
The ﬁnal objective function becomes
[Equation]
(8) which is exactly the multi-class classiﬁcation objective function ﬁrst proposed in Zhang et al.
for adversarial defense.
From the perspective of the hypocritical risk, our Proposition 2 reveals an advantage behind it, that is, TRADES is capable of minimizing the upper bound of hypocritical risk R hyp ( f ) , thus can be considered as a candidate defense method for hypocritical examples.
Proposition 2 also implies that there may be a deeper connection between adversarial robustness and hypocritical robustness.
We will discuss it more and compare our proposed THRM with TRADES in next section.
E XPERIMENTS
In this section, to verify the effectiveness of the methods (THRM and TRADES) suggested in Section 3.3, we conduct experiments on real-world datasets including MNIST and CIFAR-10.
[Figure]
[Figure] [Figure]
[Figure]
[Figure]
For the wide range of the scaling parameter λ , we conduct experiments in parallel over multiple
NVIDIA Tesla V100 GPUs.
On MNIST, perturbations are bounded by l ∞ norm with (cid:15) = 0 .
On
CIFAR-10, models are trained against 3 different hypocritical attackers bounded by l ∞ norm with (cid:15) = 1 / 255 , (cid:15) = 2 / 255 and (cid:15) = 8 / 255 , respectively. Each experiment is conducted 3 times with different random seeds.
The hypocritical risk reported here is actually an approximation of the real value, since the optimization problem in it is NP-hard and we approximately solve it using surrogate loss and PGD on test set.
Further details about model architecture and training procedure are in Appendix A.3.
Note that these experiments are extensive.
It takes over 230 GPU days to completely train the models considered in this section.
We believe that these experiments are beneﬁcial to the
ML community to further understand the tradeoffs and relative merits in THRM and TRADES.
Results on MNIST ( (cid:15) = 0 . 2 ) and CIFAR10 ( (cid:15)
= 2 / 255 ) are shown in Figure 3.
Each data point represents a model trained with different λ .
More results including comparison with Madry’s defense (Madry et al., 2018) are provided in Appendix A.3 due to the limited space.
First, we observe that, on both datasets, as the regularization parameter λ increases, the natural risk R nat increases while the hypocritical risk
ˆ R hyp decreases, which veriﬁes the effectiveness of our proposed method and the theoretical analysis in Proposition 2, where we reveal that TRADES is capable of minimizing a looser upper bound of hypocritical risk.
Second, we show that THRM achieves better tradeoff on
MNIST since it optimizes a tighter upper bound than TRADES.
However, the situation becomes nuanced on CIFAR-10.
As we can see in Figure 3(b), THRM seems to behave better in the beginning when λ is small but is surpassed by TRADES when λ increases.
Overall, optimizing only a tighter upper bound of hypocritical risk achieves better tradeoff on test set when the task is relatively simple (e.g., on MNIST with (cid:15)
= 0 . 2 )
, while simultaneously optimizing hypocritical risk and adversarial risk achieves better tradeoff on test set when the task tends to be hard (e.g., on CIFARwith (cid:15)
= 2 / 255 and (cid:15) = 8 / 255 ).
Above phenomenon shows that, when dealing with ﬁnite sample size and ﬁnite-time gradientdescent trained classiﬁers, better adversarial robustness may help the generalization of hypocritical robustness, which conforms our intuition that they are two sides of the same coin.
Interestingly, a contemporary work claims that, on CIFAR-10, TRADES achieves better adversarial robustness than
Madry’s defense in fair hyperparameter settings (Anonymous, 2021).
Thus there may be potential mutual beneﬁts between adversarial robustness and hypocritical robustness.
After all, robust training objectives force DNNs to be invariant to signals that humans are invariant to, which may lead to feature representations that are more similar to what humans use (Salman et al., 2020).
A rigorous treatment of the synergism is beyond the scope of the current paper but is an important future direction.
4. T RANSFERABILITY ANALYSIS
Transferability of adversarial examples across models is well known (Tram`er et al., 2017; Papernot et al., 2017b; Ilyas et al., 2019) and here we examine the transferability of hypocritical examples on MNIST and CIFAR-10.
We observe that hypocritical examples, i) can transfer easily between naturally trained models, ii) are hard to transfer from randomly initialized models to other models (and vise versa), iii) are hard to transfer from standard models to defended models, iv) generated from THRM models usually have high transferability.
Experimental details are in Appendix A.4.
Better transferability is beneﬁcial for black-box attacks but is not always desired by hypocritical attackers.
A hypocritical attacker only expects high transferability on the targeted model the attacker chose to help.
If there are other competing models available to the deployer, the attacker actually does not want the hypocritical examples to be successfully transferred to those competing models.
Thus ﬁne-grained attack methods are required.
We leave this to future work.
D ISCUSSION
The false friends considered in this paper are as powerful as typical adversaries.
They all know the ground truth labels of clean examples.
Such powerful friends actually can help a model to not only correctly classify a misclassiﬁed clean example but also correctly classify an adversarial example crafted by an adversary.
One may expect to rely on true friends against adversaries.
Unfortunately, an omniscient and faithful friend is unachievable in practical tasks, so far at least.
Once it is achieved, the problem of robustness disappears immediately.
What we can do at present is using a relatively more robust model as a surrogate of the true friend to improve the robustness of a weak model.
This induces a promising general method in practice, that is, high-performance models can be employed as true friends to help a weak model without exposing training data and model weights for the purpose of privacy protection and knowledge transfer (Abadi et al., 2016; Papernot et al., 2017a).
Additional discussions are in Appendix C.
C ONCLUSION
In this work, we expose a new risk arising from excessive sensitivity.
Model performance becomes hypocritical in the existence of false friends.
By formalizing the hypocritical risk and analyzing its relation with natural risk and adversarial risk, we propose to use THRM and TRADES as de-
A E XPERIMENTAL D ETAILS A. D ETAILS IN F IGURE
Attack procedure.
In adversarial attacks, we perturb clean inputs to maximize the surrogate loss using PGD.
In hypocritical attacks, we perturb clean inputs to minimize the surrogate loss using
PGD.
In both attacks, for the purpose of imperceptibility, we execute PGD attack 100 steps (step size is (cid:15)/ 50 ) with early stopping on ImageNet and the budget (cid:15) here is 2 / 255 .
More examples. More adversarial examples and hypocritical examples generated on ImageNet
using our methods are shown in Figure 5.
More hypocritical examples generated on MNIST and
CIFAR-10 are shown in Figure 4(a) and Figure 4(b).
The victim models are LeNet (Standard) and
Wide ResNet (Standard) for MNIST and CIFAR-10, respectively.
They are trained with the same procedures described in Appendix A.2.
In both attacks, for the purpose of imperceptibility, we execute 100 steps PGD attacks (step size is (cid:15)/ 50 ) with early stopping on MNIST and CIFAR-10.
The budget (cid:15) for MNIST here is 0 .
The budget for CIFAR-10 here is 8 / 255 .
(a) On MNIST dog airplane cat bird dog cat bird deer cat dog airplane frog bird horse truck ship airplane truck
[Figure]
automobile (
[Figure]
goose black swan lampshade soap dispenser screwdriver forklift washing machine stretcher bald eagle kite syringe ballpoint pen shovel wheelbarrow greenhouse library
[Figure]
Adversarial examples oystercatcher goose grey fox mongoose wardrobe washing machine photocopier washing machine remote control screwdriver grocery store restaurant paintbrush ladle tiger shark snoek
[Figure]
A. D ETAILS IN S ECTION 2.
Architecture.
For MNIST, a four-layer multilayer perception (MLP) (2 hidden layers, 768 neurons in each) with ReLU activations and a variant of LeNet model (2 convolutional layers of sizes 32 and 64, and a fully connected layer of size 1024) are adopted.
For CIFAR-10, a four-layer MLP (2 hidden layers, 3072 neurons in each) with ReLU activations, a ResNet18 (He et al., 2016a) and a
Wide ResNet (Zagoruyko & Komodakis, 2016) (with depth 28 and width factor 10) are adopted.
For
ImageNet, a VGG16 (Simonyan & Zisserman, 2014) and a ResNet50 (He et al., 2016a) are adopted.
Training procedure.
i)
Models trained with standard approach using clean examples (Standard).
For MNIST, models are trained for 80 epochs with Adam optimizer with batch size 128 and a learning rate of 0.001.
Early stopping is done with holding out 1000 examples from the MNIST training set.
For CIFAR-10, models are trained for 150 epochs with SGD optimizer with batch size 128 and the learning rate starts with 0.1, and is divided it by 10 at 90 and 125 epochs.
We apply weight decay of 2e-4 and momentum of 0.9.
Early stopping is done with holding out 1000 examples from the CIFAR-10 training set.
For ImageNet, we use the pretrained standard models available within PyTorch (torchvision.models).
ii)
Models that randomly initialized without training (Naive).
For all models, we use the default PyTorch initialization, except that we initialize the convolutional weights in Wide ResNet with He initialization (He et al., 2015).
We conduct all the experiments using a single NVIDIA Tesla V100 GPU.
Each experiment is conducted 3 times with different random seeds, except the standard models trained on ImageNet, in which we use the pretrained standard models available within PyTorch.
Attack procedure.
In adversarial attacks, we perturb clean inputs to maximize the surrogate loss using PGD.
In hypocritical attacks, we perturb clean inputs to minimize the surrogate loss using PGD.
In both attacks, we execute 50 steps PGD attacks (step size is (cid:15)/ 10 ) with 20 times of random restart on MNIST and CIFAR-10, and we use 50 steps PGD attacks (step size is (cid:15)/ 8 ) on
ImageNet.
Other hyperparameter choices didn’t offer a signiﬁcant change in accuracy.
On MNIST, the hypocritical perturbed set F and the adversarially perturbed set A are constructed by attacking every example in the clean test set sampled from D .
Both attacks are bounded by a l ∞ ball with radius (cid:15)
= 0 .
On CIFAR-10, both attacks are bounded by a l ∞ ball with radius (cid:15) = 8 / 255 .
On
ImageNet, F and A are constructed based on its validation set sampled from D .
Both attacks are bounded by a l ∞ ball with radius (cid:15)
= 16 / 255 .
Numerical results.
The attack results on CIFAR-10 are shown in Table 4.
Full results of Table 1, Table 2 and Table 4 are shown in Table 5, Table 6 and Table 7, respectively.
Moreover, we show the attack results of 9 Naive models evaluated on ImageNet in Table 6.
We ﬁnd that all the Naive models in VGG family achieve high accuracy on F and all the Naive models in ResNet family have relatively poor performance on F .
Especially, the Naive (ResNet152) model in Trial 1 is invariant to hypocritical perturbations.
Even in the existence of a strong false friend, the hypocritical performance is still as low as the clean performance (only 0 .
We carefully examined the Naive (ResNet152) model and ﬁnd that it’s actually a trivial classiﬁer, which purely classiﬁes almost all the points in input region [0 , 1] d as a certain class for some simple reasons, such as poor scaling of network weights at initialization.
Therefore, it is not enough to blindly pursue robustness against hypocritical perturbations but ignore the performance on clean examples.
Once we train a Naive model with clean examples, the model becomes vulnerable immediately (see Standard (ResNet50)), whereas the trained weights are better conditioned (Elsayed et al., 2019).
[Table]
[Table] [Table]
Naive
[Table]
Naive
[Table] [Table]
[Table]
Naive
[Table]
Naive
[Table]
Standard (
[Table]
N/A N/A
N/A N/A
N/A N/A
Standard (
[Table]
N/A N/A
N/A N/A
N/A N/A
A. D ETAILS IN S ECTION 4. Architecture.
For MNIST, a variant of LeNet model (2 convolutional layers of sizes 32 and 64, and a fully connected layer of size 1024) is adopted.
For CIFAR-10, a Wide ResNet (with depthand width factor 10) is adopted.
Training
procedure
For the wide range of the scaling parameter λ , we conduct experiments in parallel over multiple NVIDIA Tesla V100 GPUs.
Each experiment is conducted 3 times with different random seeds.
For MNIST, all models (including Standard, Madry, TRADES, THRM) are trained for 80 epochs with Adam optimizer with batch size 128 and a learning rate of 0.001.
Early stopping is done with holding out 1000 examples from the MNIST training set as suggested in Rice et al.
For CIFAR-10, all models are trained for 150 epochs with SGD optimizer with batch size 128 and the learning rate starts with 0.1, and is divided it by 10 at 90 and 125 epochs.
We apply weight decay of 2e-4 and momentum of 0.9.
Early stopping is done with holding out 1000 examples from the CIFAR-10 training set as suggested in Rice et al.
Attack procedure.
For the inner maximization in the objective function of THRM, we perturb clean inputs to minimize the CE loss as the surrogate loss.
For the inner maximization in TRADES, we maximize the KL divergence as the surrogate loss.
For the inner maximization in Madry, we maximize the CE loss as the surrogate loss.
On MNIST, the training attack is PGD with random start and 10 iterations (step size (cid:15)/ 4 ).
On CIFAR-10, the training attack is PGD with random start and 10 iterations (step size (cid:15)/ 4 ) when (cid:15) = 8 / 255 , and the training attack is PGD with random start and 7 iterations (step size (cid:15)/ 3 ) when (cid:15) = 1 / 255 and (cid:15)
= 2 / 255 .
On all experiments, the test attack is 50 steps PGD (step size is (cid:15)/ 10 ) with 20 times of random restart.
Other hyperparameter choices didn’t offer a signiﬁcant change in accuracy.
Numerical results.
The natural risk reported here is estimated on test set.
The hypocritical risk reported here is estimated on test set and is actually an approximation of the real value since we approximately solve the optimization problem by PGD on examples from test set.
Results on
MNIST ( (cid:15) = 0 . 2 ) and CIFAR-10 ( (cid:15) = 1 / 255 , (cid:15) = 2 / 255 and (cid:15) = 8 / 255 ) are shown in Figure
[Table]
Each point in Figure 6 represents one model trained with a certain λ .
Full numerical results on MNIST ( (cid:15) = 0 . 2 ) and CIFAR-10 ( (cid:15)
= 1 / 255 , (cid:15)
= 2 / 255 and (cid:15)
= 8 / 255 ) can be found in
Table 8, Table 9, Table 10 and Table 11, respectively.
On MNIST ( (cid:15) = 0 . 2 ), THRM has better tradeoff than TRADES.
However, when the task becomes hard, TRADES performs as well as or better than THRM.
On CIFAR-10, as the task becomes harder (the larger the radius (cid:15) the harder the task), the gap between TRADES and THRM becomes larger.
This phenomenon shows that better adversarial robustness may help the generalization of hypocritical robustness, especially when the task is hard.
Moreover, we compare our methods with Madry et al.
(2018)’s defense designed for adversarial robustness (denoted as Madry) 1 and standard training method (denoted as Standard).
We summarize results in Table 12.
For direct comparison, we pick a certain λ for each model trained by
TRADES and THRM in each task.
We observed that, in all tasks, Madry’s defense has nonnegligible robustness on hypocritical examples, although there is no hypocritical risk or its upper bound in the objective function.
This phenomenon indicates that optimizing only adversarial risk could bring a certain degree of robustness against hypocritical examples.
While this experimental results partly support our hypothesis (i.e., the potential mutual beneﬁts between robustness against adversarial perturbations and hypocritical perturbations), we do not take the evidence as an ultimate one and further exploration is needed.
We note that the standard deviation becomes larger when λ is bigger in TRADES and THRM, which is attributed to optimization difﬁculty and result in more signiﬁcant difference among different trials.
Reducing the initial learning rate may mitigate this phenomenon.
For completeness, we further evaluate the adversarial risk on correctly classiﬁed examples of the models trained by THRM and TRADES.
Results on MNIST ( (cid:15) = 0 . 2 ) and CIFAR-10 ( (cid:15)
= 2 / 255 ) are summarized in Table 13 and Table 14, respectively.
One interesting ﬁnding is that models trained with THRM manifest noteworthy adversarial robustness, especially on CIFAR-10, although there is no adversarial risk in the objective function of THRM.
These facts also support the hypothesis (i.e., the potential mutual beneﬁts between robustness against adversarial perturbations and hypocritical perturbations).
[Figure]
MNIST.
Perturbations are bounded by l
[Figure]
norm
[Figure]
(cid:15)
[Figure] [Figure] [Figure] CIFAR-10.
Perturbations are bounded by l
[Figure]
with (cid:15) =
[Figure] [Figure] (c) On CIFAR-10. Perturbations are bounded by l ∞ norm with (cid:15) = 2 /
[Figure] [Figure]
[Figure] [Figure] [Figure]
[Table]
[Table]
[Table] [Table]
[Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Figure]
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Table] [Figure]
[Figure] [Table] [Figure]
[Figure] [Table] [Figure]
[Table] [Figure] [Table]
[Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure]
[Table]
A. D ETAILS IN S ECTION
Hypocritical attacks here are executed by 50 steps PGD (step size is (cid:15)/ 10 ) on source models.
Note that the optimization method we used here is not to pursue state-of-the-art transferability, but to examine the transferability of hypocritical examples.
There are many methods designed to improve the transferability of adversarial examples may be extended to hypocritical examples (Liu et al., 2017; Dong et al., 2018; Wu et al., 2020a).
Figure 7 shows the transferability heatmap of hypocritical attack over 9 models trained on MNIST.
Figure 8 shows the transferability heatmap of hypocritical attack over 7 models trained on CIFAR-10.
The value in the i -th row and j -th cloumn of each heatmap matrix is the proportion of the hypocritical examples successfully transferred to target model j out of all hypocritical examples generated by source model i (including both successful and failed attacks on the source model).
N a i v e ( M
L P )
S t a n d a r d ( M L P )
N a i v e
( L e
N e t )
S t a n d a r d ( L e N e t )
N a i v e
( L e
N e t * )
S t a n d a r d ( L e N e t * )
M a d r y
( L e
N e t )
T R A D E S (
L e N e t )
T H R M ( L e
N e t )
[Figure] [Figure]
l
[Figure]
N a i v e ( M
L P )
S t a n d a r d ( M L P )
N a i v e
( W i d e R e
s N e t )
S t a n d a r d ( W i d e R e s N e t )
M a d r y
( W i d
e R e s N e t )
T R A D E S (
W i d e R e s
N e t )
T H R M ( W i d e R e s N e t )
[Figure]
B P ROOFS OF M AIN R ESULTS
In this section
[Equation]
we provide the proofs of our main results.
B. A P ROOF OF P ROPOSITION
Proposition 1.
Denote the adversarial risk on correctly classiﬁed examples by
[Equation]
then we have
[Equation]
Proof.
[Equation]
= R nat ( f )
E
[Equation]
B. A P ROOF OF T HEOREM
Theorem 1.
For any data distribution D and its corresponding conditional distribution on misclassiﬁed examples S − f w.r.t.
a classiﬁer f , we have
[Equation]
where x hyp = arg max x (cid:48) ∈B (cid:15) ( x ) 1 ( f ( x (cid:48) ) =
y ) and x rev = arg max x (cid:48) ∈B (cid:15) ( x
[Equation]
( f
[Equation]
x
[Equation]
x
[Equation]
Proof.
To prove the ﬁrst inequality
[Equation]
we have
[Equation]
where the above inequality involves two conditions:
[Equation]
To prove the second inequality, we have
[Equation]
Since ( x , y ) ∼ S − f , we have f ( x ) (cid:54) =
y .
If there exists a x hyp such that f ( x hyp ) =
y , then f ( x hyp ) (cid:54) =
f ( x ) .
Now let x rev = x hyp , then f ( x rev ) (cid:54) =
f ( x ) is true.
Otherwise, if we couldn’t ﬁnd a x hyp such that f ( x hyp ) =
y , there still exists a posibility to ﬁnd a x rev such that f ( x rev ) (cid:54) =
y but f ( x rev ) (cid:54) =
f ( x ) is true.
Therefore, the above inequalities holds.
B. A P ROOF OF P ROPOSITION
Proposition 2.
R rev ( f ) =
(1 −R nat ( f ))
ˆ R adv ( f )+ R nat ( f ) R hyp ( f ) =
[Equation]
[Equation]
f ( x rev ) (cid:54) =
f ( x )) .
Proof.
R rev
[Equation]
( f )
[Equation]
nat
[Equation]
f
[Equation]
E
[Equation]
∼S
[Equation] [Equation]
) (cid:54) =
y )]
+ R nat ( f )
E ( x ,y ) ∼S − f [ 1 ( f ( x rev ) (cid:54) =
f ( x ))]
=
E ( x ,y ) ∼D [ 1 ( f ( x ) = y ) · 1 ( f ( x adv ) (cid:54) =
y )]
+
E ( x ,y ) ∼D [ 1 ( f ( x ) (cid:54) =
y ) · 1 ( f ( x rev ) (cid:54) =
f ( x ))]
=
E ( x ,y ) ∼D (cid:20)
( f ( x ) = y )
· max x (cid:48) ∈B (cid:15) ( x ) 1 ( f ( x (cid:48) ) (cid:54) =
y ) (cid:21) +
E ( x ,y ) ∼D (cid:20)
( f ( x ) (cid:54)
[Equation]
y )
· max x (cid:48)
[Equation]
( x )
[Equation]
( f ( x
[Equation]
) (cid:54)
[Equation]
f
[Equation]
x )) (cid:21) =
E ( x ,y ) ∼D (cid:20)
( f ( x ) = y ) · max x (cid:48)
[Equation]
(cid:15) ( x ) 1 ( f ( x (cid:48) )
(cid:54) =
f ( x )) (cid:21)
+
E ( x ,y ) ∼D (cid:20)
( f ( x ) (cid:54) =
y )
· max x (cid:48) ∈B (cid:15) ( x ) 1 ( f ( x (cid:48) ) (cid:54) =
f ( x )) (cid:21) =
E ( x ,y ) ∼D (cid:20) max x (cid:48)
[Equation]
(cid:15) ( x ) 1 (
[Equation]
( x (cid:48) ) (cid:54) =
f ( x )) (cid:21) =
E ( x ,
[Equation]
∼D [ 1 ( f ( x rev ) (cid:54) =
f ( x
C A DDITIONAL D ISCUSSIONS
We showed that correctly classiﬁed examples (hypocritical examples) could be easily found in the vicinity of misclassiﬁed clean examples.
As a result, a hypocritically perturbed set could be constructed with these hypocritical examples.
The victim model’s standard accuracy evaluated on the hypocritically perturbed set becomes higher than that on the clean set.
It is natural then to wonder:
How about adversarially robust accuracy (i.e., accuracy under adversarial perturbations) of the victim model on hypocritical examples?
It’s easy to see that, if the adversary is bounded by the same (cid:15) ball as the false friend, the model’s adversarial accuracy evaluated on hypocritically perturbed set is zero, since a misclassiﬁed example exists in the (cid:15) ball of a hypocritical example (by deﬁnition).
However, if the adversary’s power is restricted by another δ ball such that δ < (cid:15) , then a robust hypocritical example may exist in the vicinity of a clean example so that a δ -bounded adversary can not change the model’s prediction on the robust hypocritical example.
In such a case, the model’s adversarial accuracy evaluated on the robustly hypocritically perturbed set could be higher than that on the clean set.
New attack and defense methods are required to further explore this phenomenon.
[Figure]
D T RADEOFF BETWEEN ADVERSARIAL AND HYPOCRITICAL RISKS
Despite the experiments in Section 4.1 and Appendix A.3 showed that, when dealing with ﬁnite sample size and ﬁnite-time gradient-descent trained classiﬁers, there may be mutual beneﬁts between adversarial robustness and hypocritical robustness in real-world datasets, we note that in general, this synergism does not necessarily exist.
We illustrate the phenomenon by providing another toy example here, which is inspired by the precision-recall tradeoff Buckland & Gey (1994);
Consider the case ( x , y ) ∈
R 2 × {− 1 , +1 } from a distribution D , where the marginal distribution over the instance space is a uniform distribution over [0 , 1] 2 .
We assume that the decision boundary of the oracle (ground truth) is a circle:
[Equation]
(9) where the centre c = (0 . 5 , 0 . 5) (cid:62) and the radius r = 0 .
The points inside the circle are labeled as belonging to the positive class, otherwise they are labeled as belonging to the negative class.
We consider the linear classiﬁer f with ﬁxed w = (0 , 1) (cid:62) and a tunable threshold b :
[Equation]
See Figure 9 for a visualization of the oracle and the linear classiﬁer over the instance space.
In this problem, we can show the tradeoffs by tuning the threshold b of the linear classiﬁer over [0 , 1] .
The precision is the number of true positives (i.e. the number of examples correctly classiﬁed as positive class) divided by the the sum of true positives and false positives (i.e. the number of examples misclassiﬁed as positive class).
The recall is the number of true positives divided by the sum of true positives and false negatives (i.e. the number of examples misclassiﬁed as negative class).
We compare the adversarial risk on correctly classiﬁed examples ˆ R adv ( f ) deﬁned in Proposition 1 and the hypocritical risk on misclassiﬁed examples
ˆ R hyp ( f ) deﬁned in Deﬁnition 3.
The computing
[Figure]
[Figure]
[Figure]
formulas of these values are visualized in Figure 10.
Here we choose the bounded l 2 ball B (cid:15) ( x ) = { x (cid:48) ∈ R 2 : (cid:107) x
(cid:48) − x (cid:107) 2 ≤ (cid:15) } with (cid:15) = 0 .
1 as the threat model.
Figure 11 plots the curve of precision and recall versus threshold b .
We can see that there is a obvious precision-recall tradeoff between the two gray dotted lines.
Similarly, Figure 12 plots the curve of ˆ
R adv ( f ) and ˆ R hyp ( f ) versus threshold b .
We can see that the tradeoff exists almost everywhere: as the adversarial risk increases, the hypocritical risk decreases, and vise versa.