AEVA: B LACK - BOX B ACKDOOR D ETECTION U SING A DVERSARIAL E XTREME V ALUE A NALYSIS
A BSTRACT
Deep neural networks (DNNs) are proved to be vulnerable against backdoor attacks.
A backdoor is often embedded in the target DNNs through injecting a backdoor trigger into training examples, which can cause the target DNNs misclassify an input attached with the backdoor trigger.
Existing backdoor detection methods often require the access to the original poisoned training data, the parameters of the target DNNs, or the predictive conﬁdence for each given input, which are impractical in many real-world applications, e.g., on-device deployed DNNs.
We address the black-box hard-label backdoor detection problem where the DNN is fully black-box and only its ﬁnal output label is accessible.
We approach this problem from the optimization perspective and show that the objective of backdoor detection is bounded by an adversarial objective.
Further theoretical and empirical studies reveal that this adversarial objective leads to a solution with highly skewed distribution; a singularity is often observed in the adversarial map of a backdoorinfected example, which we call the adversarial singularity phenomenon .
Based on this observation, we propose the adversarial extreme value analysis (AEVA) to detect backdoors in black-box neural networks.
AEVA is based on an extreme value analysis of the adversarial map, computed from the monte-carlo gradient estimation.
Evidenced by extensive experiments across multiple popular tasks and backdoor attacks, our approach is shown effective in detecting backdoor attacks under the black-box hard-label scenarios.
I NTRODUCTION
Deep Neural Networks (DNNs) have pervasively been used in a wide range of applications such as facial recognition (Masi et al., 2018), object detection (Szegedy et al., 2013; Li et al., 2022), autonomous driving (Okuyama et al., 2018), and home assistants (Singh et al., 2020; Zhai et al., 2021).
In the meanwhile, DNNs become increasingly complex.
Training state-of-the-art models requires enormous data and expensive computation.
To address this problem, vendors and developers start to provide pre-trained DNN models.
Similar to softwares shared on GitHub, pre-trained DNN models are being published and shared on online venues like the BigML model market, ONNX zoo and Caffe model zoo.
Since the dawn of software distribution, there has been an ongoing war between publishers sneaking malicious code and backdoors in their software and security personnel detecting them.
Recent studies show that DNN models can contain similar backdoors, which are induced due to contaminated training data.
Sometimes models containing backdoors can perform better than the regular models under untampered test inputs.
However, under inputs tampered with a speciﬁc pattern (called trojan trigger) models containing backdoors can suffer from signiﬁcant accuracy loss. [Figure]
There has been a signiﬁcant amount of recent work on detecting the backdoor triggers.
However, those solutions require access to the original poisoned training data (Chen et al., 2019a; Tran et al., 2018; Huang et al., 2022), the parameters of the trained model (Chen et al., 2019b; Guo et al., 2020;
Liu et al., 2019; Wang et al., 2019; 2020; Dong et al., 2021; Kolouri et al., 2020), or the predicted conﬁdence score of each class (Dong et al., 2021).
Unfortunately, it is costly and often impractical for the defender to access the original poisoned training dataset.
In situations when DNNs are deployed in safety-critical platforms (Fowers et al., 2018; Okuyama et al., 2018; Li et al., 2021b) or in cloud services (Fowers et al., 2018; Chen et al., 2019c), it is also impractical to access either their parameters or the predicted conﬁdence score of each class (Chen et al., 2019c; 2020b).
We present the black-box hard-label backdoor detection problem where the DNN is a fully blackbox and only its ﬁnal output label is accessible (Fig. 1).
Detecting backdoor-infected DNNs in such black-box setting becomes critical due to the emerging model deployment in embedded devices and remote cloud servers.
In this setting, the typical optimization objective of backdoor detection (Chen et al., 2019b; Liu et al., 2019) becomes impossible to solve due to the limited information.
However, according to a theoretical analysis, we show that the backdoor objective is bounded by an adversarial objective, which can be optimized using Monte Carlo gradient estimation in our black-box hardlabel setup.
Further theoretical and empirical studies reveal that this adversarial objective leads to a solution with highly skewed distribution; a singularity is likely to be observed in the adversarial map of a backdoor-infected example, which we call the adversarial singularity phenomenon .
Based on these ﬁndings, we propose the adversarial extreme value analysis (AEVA) algorithm to detect backdoors in black-box neural networks.
The AEVA algorithm is based on an extreme value analysis (Leadbetter, 1991) on the adversarial map.
We detect the adversarial singularity phenomenon by looking at the adversarial peak , i.e. , maximum value of the computed adversarial perturbation map.
We perform a statistical study which reveals that, there is around 60% chance that the adversarial peak of a random sample for a backdoor-infected DNN is larger than the adversarial peak of any examples for an uninfected DNN.
Inspired by the Univariate theory, we propose a global adversarial peak (GAP) value by sampling multiple examples and choosing the maximum over their adversarial peaks, to ensure a high success rate.
Following previous works (Dong et al., 2021; Wang et al., 2019), the Median Absolute Deviation (MAD) algorithm is implemented on top of the GAP values to test whether a DNN is backdoor-infected.
Through extensive experiments across three popular tasks and state-of-the-art backdoor techniques,
AEVA is proved to be effective in detecting backdoor attacks under the black-box hard-label scenarios.
The results show that AEVA can efﬁciently detect backdoor-infected DNNs, yielding an overall detection accuracy ≥ 86 .
7% across various tasks, DNN models and triggers.
Rather interestingly, when comparing to two state-of-art white-box backdoor detection methods, AEVA yields comparable performance, even though AEVA is a black-box method with limited access to information.
We summarize our contributions as below 1 :
To the best of our knowledge, we are the ﬁrst to present the black-box hard-label backdoor detection problem and provide an effective solution to this problem.
We provide a theoretical analysis which shows backdoor detection optimization is bounded by an adversarial objective.
And we further reveal the adversarial singularity phenomenon where the adversarial perturbation computed from a backdoor-infected neural network is likely to suffer from a highly skewed distribution.
We propose a generic backdoor detection framework AEVA, optimizing the adversarial objective and performing extreme value analysis on the optimized adversarial map.
AEVA is applicable to the black-box hard-label setting with a Monte Carlo gradient estimation.
We evaluate AEVA on three widely-adopted tasks with different backdoor trigger implementations and complex black-box attack variants.
All results suggest that AEVA is effective in black-box hard-label backdoor detection.
1. R ELATED W ORK
Backdoor attacks.
BadNets (Gu et al., 2019) is probably the ﬁrst work on backdoor attacks against
DNNs, which causes target misclassiﬁcation to DNNs through injecting a small trigger into some training samples and mis-labeling these samples with a speciﬁc target class.
Trojaning attack (Liu et al., 2018) generates a trigger which can cause the maximum activation value of certain selected neurons with limited training data.
Chen et al.
(Chen et al., 2017) further propose to perform backdoor attacks under a rather weak threat model where the attacker cannot access the target model and training dataset.
Most recently, a set of backdoor attacks (Ji et al., 2018; Liu et al., 2020; Saha et al., 2020; Turner et al., 2019; Yao et al., 2019) have been proposed, which are built upon existing work but focused on various speciﬁc scenarios, e.g. , physical-world, transfer learning, etc .
Backdoor detection (white-box).
A few works (Chen et al., 2019a; Tran et al., 2018) are proposed to detect the backdoor samples within the training dataset.
Chen et al. (2019a) propose a neuron activation clustering approach to identify backdoor samples through clustering the training data based on the neuron activation of the target DNN.
Tran et al.
(2018) distinguish backdoor samples from clean samples based on the spectrum of the feature covariance of the target DNN.
Few other works focus on detecting backdoor-infected DNNs (Liu et al., 2019;
Wang et al., 2019).
ABS (Liu et al., 2019) identiﬁes the infected DNNs by seeking compromised neurons representing the features of backdoor triggers.
Neural Cleanse (Wang et al., 2019) conducts the reverse engineering to restore the trigger through solving an optimization problem.
Recent works (Chen et al., 2019b; Guo et al., 2020; Wang et al., 2020) improve Neural Cleanse with better objective functions.
However, these methods are all white-box based, requiring access to model parameters or internal neuron values.
Backdoor detection (black-box).
A recent work (Chen et al., 2019b) claims to detect backdoor attacks in the “black-box” settings.
However, their method still need the DNN’s parameters to train a separate generator (Goodfellow et al., 2014).
So, strictly speaking, their method is not “black-box”, which is also revealed by (Dong et al., 2021).
To the best of our knowledge, Dong et al.
(2021) is the only existing work on detecting backdoor-infected DNNs in the black-box settings.
However, their method requires the predictive conﬁdence score for each input to perform the NES algorithm (Wierstra et al., 2014), which weakens its practicability.
Our work differs from all previous methods in that we address a purely black-box setup where only the hard output label is accessible.
Both model parameters and training examples are inaccessible in the black-box hard-label setting.
P RELIMINARIES : BLACK - BOX BACKDOOR ATTACK AND DEFENSE 2. T HREAT MODEL
Our considered threat model contains two parts: the adversary and the defender .
The threat model of the adversary follows previous works (Chen et al., 2017; Gu et al., 2019; Liu et al., 2018).
In this model, the attacker can inject an arbitrary amount of backdoor samples into the training dataset and cause target misclassiﬁcation to a speciﬁc label without affecting the model’s accuracy on normal examples.
From the perspective of the defender, we consider the threat model with the weakest assumption, in which the poisoned training dataset and the parameters of the target DNNs f ( · ) are inaccessible.
Moreover, the defender can only obtain the ﬁnal predictive label for each input from the target DNNs and a validation set (40 images for each class).
Therefore, the defender can only query the target DNNs to obtain its ﬁnal decisions.
The defender’s goal is to identify whether the target DNNs is infected and which label is infected.
2. P ROBLEM DEFINITION
Consistent with prior studies (Dong et al., 2021; Kolouri et al., 2020; Wang et al., 2019; Huang et al., 2022), we deem a DNN is backdoor infected if one can make an arbitrary input misclassiﬁed as the target label, with minor modiﬁcation to the input.
Without loss of generability, given the original input x ∈ R n , the modiﬁed input containing the backdoor trigger can be formulated as:
[Equation]
where ∆ ∈ R n represents the backdoor trigger and (cid:12) represents the element-wise product.
m ∈ { 0 , 1 } n is a binary mask that ensures the position and magnitude of the backdoor trigger.
Typically, || m ||
1 ought to be very small to ensure the visually-indistinguishability between ˆ x and x .
We deﬁne f ( · ; θ ) as the target DNN with parameters θ , which is a function that maps the input x to a label y .
φ ( x, y ; θ ) ∈ [0 , 1] is deﬁned as the probability that x is classiﬁed as y by f ( · ; θ ) .
The argument θ is sometimes omitted in favor of simplicity.
For f to mislabel ˆ x as the target label y t , there are two possible reasons: the backdoored training data either contains the same trigger but mislabeled as y t or contains similar features as the normal inputs (not belonging to y t , but visually similar to y t ) and labeled as y t .
[Equation]
where there are N o original training examples and N b backdoor examples.
(cid:96) ( · ) represents the crossentropy loss function; x i and y i represent the training sample and its corresponding ground-truth label, respectively.
In the inference phase, the target DNN will predict each ˆ x as the target label y t , which can be formulated as: f
(ˆ x ) =
y t .
Prior works (Dong et al., 2021; Wang et al., 2019; Chen et al., 2019b) seek to reveal the existence of a backdoor through investigating whether there exists a backdoor trigger with minimal
|| m ||
that can cause misclassiﬁcation to a certain label.
We mainly study the sparse mask since it is widely used in previous works (Wang et al., 2020; Dong et al., 2021;
Chen et al., 2019b; Guo et al., 2020; Wang et al., 2019; Gu et al., 2019).
In cases when the mask becomes dense, the mask m has to contain low magnitude continuous values to ensure the backdoor examples visually indistinguishable.
However, when the value of m is small, every normal example x i becomes near the decision boundary of function φ .
It is revealed by Taylor theorem, φ (ˆ x i ) =
φ ((1 − m ) x i + m ∆) =
φ ( x i + m (∆ − x i ))
≈ φ ( x i )
+ m (∆ − x i ) (cid:124) ∇ x φ .
And the function output label changes dramatically from φ ( x i ) to φ (ˆ x i ) , so the gradient ∇ x φ has to have a large magnitude, which results in a non-robust model sensitive to subtle input change.
Such models become impractical because a small random perturbation on the input can lead to signiﬁcantly different model output.
More details with empirical study can be found in the Appendix B.
2. B LACK - BOX BACKDOOR DETECTION AND ITS CHALLENGES
Most previous works (Wang et al., 2019; Dong et al., 2021; Chen et al., 2019b) focus on reverseengineering the backdoor trigger for each target label y t using the following optimization:
[Equation]
where β is the balancing parameter.
Unfortunately, solving Eq. 3 is notoriously hard in the blackbox hard-label setting since θ is unknown.
Additional difﬁculty comes with the fact that an effective zero-th order gradient estimation requires each example x i superimposed with the trigger to be close to the decision boundary of y t (Chen et al., 2020a; Cheng et al., 2020; Li et al., 2020).
However, such condition is hard to achieve in the hard-label blackbox settings.
A DVERSARIAL EXTREME VALUE ANALYSIS FOR BACKDOOR DETECTION
Our AEVA framework is introduced in this section.
We ﬁrst derive an upper bound to the backdoor detection objective in Eq.
3 and ﬁnd its connection to adversarial attack.
The adversarial singularity phenomenon is further revealed in both theoretical results and empirical studies.
Based on our ﬁndings, we propose the Global Adversarial Peak (GAP) measure, computed by extreme value analysis on the adversarial perturbation.
The GAP score will be ﬁnally used with Median Absolute
Deviation (MAD) to detect backdoors in neural networks.
Monte Carlo gradient estimation is further introduced in order to solve the adversarial objective in the black-box setting.
3. A DVERSARIAL OBJECTIVE AS AN UPPER BOUND OF THE BACKDOOR OBJECTIVE
We present in this section the connection between the backdoor objective in Eq.
3 with the objective in adversarial attacks.
To begin with, we ﬁrst transform the backdoor masking equation (1 − m ) (cid:12) x i + m (cid:12) ∆ =
x i + µ − m (cid:12) x i , where µ = m (cid:12) ∆ .
Following this, the optimization in Eq.converts to minimizing the following term:
[Equation]
We further introduce an important result from the Multivariate Taylor Theorem below.
Lemma 1.
Given a continuous differentiable function f ( · ) deﬁned on R n and vectors x, h ∈ R n , for any M ≥ | ∂f/∂h
i | , ∀ 1 ≤ i ≤ n , we have | f ( x + h ) − f ( x ) | ≤ M (cid:107) h (cid:107) 1 .
The lemma can be proved using the Lagrange Remainder.
Please refer to the Appendix C.1 for a detailed proof.
According to the lemma, we have
[Equation]
where the latter inequality holds because each example x i ≤ 1 is bounded and the last equality (cid:107) m (cid:107)
= (cid:107) m [Equation] ∆ [-(cid:107) 
-][Equation]
0 = (cid:107) µ (cid:107) 0 holds because m is a binary mask.
Then, we can see that the adversarial objective is an upper bound of the objective in backdoor detection, i.e. , F ≤ 1 N (cid:80) i (cid:96) ( φ ( x i + µ ) , y t )
+
C (cid:107) µ (cid:107) 0 .
Instead of optimizing the original Eq.
3, we here propose to minimize the (cid:96) 0 -regularized objective.
While (cid:96)
enforces the sparsity of the solution, optimizing it is NP-hard.
In practice, it is replaced with an (cid:96) 1 -norm as an envelope of the objective (Ramirez et al., 2013; Donoho, 2006):
[Equation]
This adversarial optimization can be solved using Monte Carlo gradient estimation in our black-box setting.
More details are described in Section 3.4.
In the following, we will ﬁrst discuss how the solution to this adversarial objective can be used to detect backdoors in neural networks.
3. T HE ADVERSARIAL SINGULARITY PHENOMENON
A natural and important question here is what the adversarial perturbation ˆ µ would look like when the input is infected with backdoor attacks.
A direct analysis on deep neural networks is hard.
So, we start by analyzing a linear model to shed light on the intuition of our approach.
Lemma 2.
Given a linear model parameterized by θ optimized on N o original training examples and N b backdoored training examples with an objective in Eq.
2 and a mean-squared-error loss, the adversarial solution ˆ µ optimized using Eq.
6 will be dominated by input dimensions corresponding to backdoor mask m when N b is large, i.e., lim N b →∞ (cid:107) (1 − m ) (cid:12) ˆ µ (cid:107) 1 / (cid:107) ˆ µ (cid:107) 1 = 0 .
The lemma can be proved by least square solutions and exact gradient computation.
Detailed proof can be found in the Appendix C.2.
The lemma reveals that the majority of the mass in the adversarial perturbation will be occupied in the mask area.
Since the mask m is usually sparse, it is reasonable to expect a highly skewed distribution in the adversarial map ˆ µ .
We suspect that such skewed distributions might also occur in the adversarial map of a deep neural network.
While a thorough analysis on DNN is hard, recent studies in Neural Tangent Kernel (NTK) (Jacot et al., 2018) have shown that a deep neural network with inﬁnite width can be treated as kernel least square.
So, we further extended our analysis to a k -way kernel least square classiﬁer as the model and a cross-entropy loss used in the adversarial analysis. [Figure]
Lemma 3.
Suppose the training dataset consists of N o original examples and N b backdoor examples, i.i.d.
sampled from uniform distribution and belonging to k classes.
Let φ be a multivariate kernel regression with the objective in Eq.
2, an RBF kernel.
Then the adversarial solution ˆ µ to Eq.
6 under cross-entropy loss should satisfy that lim N b →∞ E
[(1 − m ) (cid:12) ˆ µ ] = 0 .
The proof can be found in Appendix C.3.
The lemma reveals a similar result as the linear case that the majority mass of the adversarial perturbation (in expectation) stays in the mask area when sufﬁcient backdoor examples are used.
Although the lemma does not directly address a deep neural network, we hope this result could help understand the problem better in the NTK regime.
We perform an empirical study on DNNs to validate our intuition.
Following previous works (Dong et al., 2021; Wang et al., 2019; 2020), we implement BadNets (Gu et al., 2019) as the backdoor attack method.
We randomly select a label as the infected label and train a ResNet-44 on CIFARembedded with the backdoor attack, resulting in an attack success rate of 99 . 87% .
The backdoor trigger is a 4x4 square shown in Fig. 2(a), which occupies at most 1 .
6% of the whole image.
The adversarial perturbations ˆ µ (by optimizing Eq. 6) are generated for both infected and uninfected labels.
The perturbations are normalized to a heat map as | ˆ µ | /
||
ˆ µ || 1 and shown in Fig.
A high concentration of mass is clearly observed from the infected case in Figure 2(b).
We call it adversarial singularity phenomenon .
This motivates our adversarial extreme value analysis (AEVA) method.
3. A DVERSARIAL EXTREME VALUE ANALYSIS
[Figure]
A natural way to identify singularity is to look at the peak value of the distribution.
We deﬁne the adversarial peak µ max = max µ ij for an adversarial perturbation µ .
To gain a statistical understanding of the relationship between adversarial peak and backdoor attacks, We randomly sample 1000 normalized µ for the uninfected and infected labels and plot the distribution of µ max in Fig.
The experimental setup is consistent with the empirical study in Sec.
We notice from the ﬁgure that µ max of the infected labels reaches a much larger range compared to that of uninfected labels.
However, they still overlap; around 47% infected µ max stays within the uninfected range (the area with both blue and orange bars).
This result suggests that the adversarial singularity phenomenon does not always occur in backdoor-infected DNNs .
It is reasonable because some samples are inherently vulnerable to adversarial attacks and their adversarial perturbations have similar properties as backdoors, which reduces the chance of singularity.
Now, we know if we deﬁne a threshold T as the maximum value of the uninfected labels, the probability of a backdoor-infected µ max being smaller than T is 0 .
47 , i.e. , P ( µ max < T ) = 0 .
Taking one example is impossible to know whether a label is infected.
However, according to the Univariate Theory (Gomes & Guillou, 2015), if we sample k examples of µ max for the same label,
[Equation]
[Figure]
[Figure]
MAD Detector
AnomalyIndex
[Figure]
7.6219 (Malicious) 1.
[Figure]
Here we name the maximum value over all the k adversarial peaks as the Global Adversarial Peak (GAP).
We vary the choice of k in Figure 4 and we can see that the chance of GAP value being lower than T is diminishing.
For example, if we take k = 6 , then P ( GAP < T ) = 0 .
47 6 = 0 .and
the success rate of identifying a backdoor-infected label is 99% .
Please be advised that, given the long-tail distribution of the infected µ max , this threshold T is not sensitive and can be made even larger to be safe to include all uninfected labels.
However, for larger T values, we need a larger k to ensure a desired success rate.
3. B LACK - BOX OPTIMIZATION VIA GRADIENT ESTIMATION
[Figure]
After investigating the unique properties of backdoor-infected DNNs in the white-box settings, we here illustrate how to exploit such properties under a rather restricted scenario as we consider.
In the white-box settings, the free access to θ can enable us to compute the ∇ x φ ( x, y t ) accurately, resulting in µ with minimum || µ || 1 through optimizing Eq. 6.
Regarding the black-box settings, we propose to leverage the zero-order gradient estimation technique (Fu & Hu, 2012) to address the challenge for calculating ∇ x φ ( x, y t ) .
We choose Monte Carlo based method (Fu & Hu, 2012) for obtaining the estimated gradient (cid:102) ∇ φ x ( x, y t ) , which sends several inputs and utilizes the corresponding outputs to estimate the gradient.
More details about gradient estimation can be found in Appendix D.
3. F INAL ALGORITHM
Putting all the above ideas together, we present the AEVA framework (illustrated in Fig. 5).
The algorithm for computing the GAP value of a given label is described in Algorithm 1.
Our approach requires a batch of legitimate samples
X i for each class i .
For each label y t to be analyzed, the algorithm will tell whether y t is backdoor-infected.
We treat y t as the targeted label.
Then we collect a batch of legitimate samples belonging to y t , denoted as X t with the same batch size as that of X i .
Then, we leverage X t to make each sample within every { X i } ni =0 to approach the decision boundary of y t .
This is computed by optimizing ˆ µ from Eq.
6 using Monte [Figure] estimation.
The adversarial peak µ max is then computed for each example.
The global adversarial peak (GAP) is aggregated over all labels.
After calculating the GAP value R ( y i ) for each class i , following previous work (Wang et al., 2019; Dong et al., 2021), we implement Median Absolute
Deviation (MAD) to detect the outliers among {R ( y i ) } ni =1 .
Speciﬁcally, we use MAD to calculate the anomalous index for each R ( y i ) by assuming {R ( y i ) } ni =1 ﬁts Gaussian distribution.
The outlier is then detected by thresholding the anomaly index.
E XPERIMENTS 4. S ETTINGS
Datasets.
We evaluate our approach on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets.
For each task, We build 240 infected and 240 benign models with different architectures:
ResNets (He et al., 2016) and DenseNets (Huang et al., 2017).
We randomly select one label for each backdoorinfected model and inject sufﬁcient poisoned samples to ensure the attack success rate ≥ 98% .
More details can be found in the Appendix.
Attack methods.
Four different attack methods are implemented:
BadNets (Gu et al., 2019), Watermark attack (Chen et al., 2017), Label-Consistent Attack (Turner et al., 2019) and Invisible Attack (Li et al., 2021a).
For BadNets and Watermark, the triggers are 4 × 4 squares.
The transparency ratio for Watermark attack is 0.1.
For label-consistent and invisible attacks, we use the default lessvisible triggers.
Around 10% poison data is injected in training.
Each model is trained forepochs with data augmentation.
More details can be found in the Appendix G.
Baseline methods.
Since there is no existing black-box hard-label method, we compare our approach with two state-of-art white box backdoor detection methods: Neural Cleanse (NC) (Wang et al., 2019) and DL-TND (Wang et al., 2020) 2 .
For each task, we use 200 samples for the gradient estimation, and the batch size for each { X i } n i =1 is set to 40 in Algorithm 1.
Outlier detection.
To accurately identify the anomalous R ( y i ) among {R ( y i ) } ni =1 , we assume the scores ﬁt to a normal distribution and apply a constant estimator (1.4826) to normalize the computed anomaly index similar to (Wang et al., 2019).
We set the threshold value τ = 4 for our
MAD detector, which means we identify the class whose corresponding anomaly index larger than 4 as infected.
This value is chosen using a hold-out validation set of infected and benign models.
Evaluation metrics.
We follow previous work (Kolouri et al., 2020;
Wang et al., 2020; Dong et al., 2021; Guo et al., 2020) on using two common metrics: (a) The Area under Receiver Operating
Curve (AUROC) – The Receiver Operating Curve (ROC) shows the trade-off between detection success rate for infected models and detection error rate for benign models across different decision thresholds τ for anomaly index; (b) Detection Accuracy (ACC) – The proportion of models are correctly identiﬁed.
Regarding infected models, they are correctly identiﬁed
if and only if the infected labels are identiﬁed without mistagging other uninfected labels. [Figure] [Table] [Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
AEVA on CIFAR-10, CIFAR100 and TinyImageNet tasks.
AEVA is black-box while the other two methods are white-box.
4. R ESULTS
We ﬁrst investigate whether AEVA can reveal the singularity phenomenon of labels infected with different attack approaches.
We randomly select an infected model infected by each attack approach and an uninfected model for TinyImageNet task.
Notably, all these selected models are correctly identiﬁed by our approach.
We plot the corresponding normalized adversarial perturbation µ in
Fig.
6, which demonstrates that AEVA can accurately and distinguish the uninfected and infected labels and reveal the singularity phenomena of backdoor triggers.
Table 1 presents the overall results.
AEVA can accurately detect the infected label with ACC ≥ 86 .
7% across all three tasks and various trigger settings.
We compare our approach with existing white-box detection approaches including Neural Cleanse (NC) (Wang et al., 2019) and DLTND (Wang et al., 2020).
The comparison results over all infected and benign models are shown in
Fig.
More results can be found in Appendix J. The results suggest that AEVA achieves comparable performance with existing white-box detection approaches across different settings.
Such close performance also indicates the efﬁcacy of AEVA on black-box hard-label backdoor detection.
4. A BLATION S TUDY
[Figure]
The impact of trigger size.
We test AEVA on TinyImageNet with different trigger sizes for BadNets and Watermark attacks.
For each trigger size and attack method, we build 60 infected models with various architectures following the conﬁgurations in Sec.
Fig.
8 shows that AEVA remains effective when trigger size is less than 14 × 14 with ACC ≥ 71 .
For large triggers, AEVA cannot identify the infected label since the singularity property is alleviated.
This is consistent with our theoretical analysis.
Even though large trigger attacks can bypass AEVA, they are either visuallydistinguishable or leading to a non-robust model sensitive to input change, making it less stealthy and impractical.
The impact of infected label numbers.
We further investigate the impact of the number of infected labels.
We build 60 infected models for CIFAR-10 to evaluate AEVA.
We randomly select infected labels, and we inject a 2 × 2 trigger for each infected label.
Fig.
9 shows that AEVA performs effectively when the number of infected labels is less than 3 ( i.e. , 30% of the entire labels) with
ACC ≥ 78 .
It is because too many infected labels fail the MAD outlier detection.
Interestingly, existing white-box detection methods (Wang et al., 2019; Chen et al., 2019b; Guo et al., 2020; Liu et al., 2019; Wang et al., 2020) cannot perform effectively in such scenarios either.
However, too many infected labels would reduce the stealth of the attacker as well.
[Figure]
Besides these studies, we further evaluate our approach under more scenarios, including the impact of the number of available labels, multiple triggers for a single infected label, different backdoor trigger shapes and potential adaptive backdoor attacks, which are included in the Appendix.
C ONCLUSION
This paper takes a ﬁrst step addressing the black-box hard-label backdoor detection problem.
We propose the adversarial extreme value analysis (AEVA) algorithm, which is based on an extreme value analysis on the adversarial map, computed from the montecarlo gradient estimation due to the black-box hard-label constraint.
Extensive experiments demonstrate the efﬁcacy of AEVA across a set of popular tasks and state-of-the-art backdoor attacks.
E THICS S TATEMENT
Our work aims at providing neural network practitioners additional protection against backdoor attacks.
We believe our work could contribute positively to the human society and avoid potential harm since it addresses a critical safety problem.
We are unaware of any direct negative impact out of this work.
Our method has certain limitations such as the sensitivity to large backdoor trigger and multiple infected labels.
However, as we earlier in the paper, these scenarios make the attack become less stealthy and not practical in real applications.
R EPRODUCIBILITY S TATEMENT
Our work is built upon comprehensive theoretical results and clear motivations.
We believe the proposed method can be reproduced according to the content in the paper, e.g. , Algorithm 1.
In addition, we have released an anonymous link containing our implementation code as reference in
Appendix A.
A PPENDIX Table of
Contents
B Backdoor B
Feature
C Proofs
C.
The proof of Lemma 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.
The proof of Lemma 2 – a theory on the linear case . . . . . . . . . . . . . . . .
C.3 Proof of Lemma 3 – An analysis for Kernel Regression . . . . . . . . . . . . . .
D Monte Carlos gradient estimation approaches
K triggers
K.1 Evaluation dynamic and sparse but not compact triggers . . . . . . . . . . . .
L Results for impact of the number of images per labels.
N scenarios
O Potential adaptive backdoor attacks
O.1 Attacks with multiple target labels . . . . . . . . . . . . . . . . . . . . . . . . .
O.2 Attack with large triggers . .
A L INK FOR CODE FOR AEVA
The code for AEVA will be open-sourced upon the publication of this paper.
It is currently included in an anonymous github repository and can be found here: https://github.com/ aeva-backdoor-deteciton/Aeva-Blackbox-Backdoor-Detection.git
B O THER B ACKDOOR A TTACKS
B D ENSE W ATERMARK B ACKDOOR A TTACKS .
In short, we argue that low-magnitude dense backdoor attack(watermark) is not a practical attack method and is less concerned compared to the threat model in our paper.
First, the low-magnitude attack is a special case of our formulation where the mask m is a continuous value (here we use a to represent transparency).
The low-magnitude problem can be formulated as:
[Equation]
This formulation does not have discrete variables so its black-box optimization is an easier task compared to our problem (where m is discrete).
Second, we will show theoretically that, in this scenario, every normal example is near the decision boundary of the network, because due to Taylor expansion and the fact that a is a small value,
[Equation]
We know from x i to (1 − a ) x
i + a ∆ , the function output label is changed.
So every example x i is at the decision boundary of the function φ .
So the infected model is actually very sensitive to random noise and becomes ineffective in attacking purposes.
To empirically support this claim, we conduct experiments on CIFAR-10 using ResNet-44.
The results are reported over average among ten different target labels.
We select watermark triggers randomly which occupy the entire image with 0.1 transparency rate (a).
We here add noise which ﬁts uniform distribution with a low magnitude as 0.025 to each normal sample.
The proportion of normal samples misclassiﬁed by the infected label is shown below.
We ﬁnd that such dense lowmagnitude watermark triggers would make normal samples to be misclassiﬁed as the target label y t via adding small random noise.
Such observations can easily identify models infected with a dense and low-magnitude watermark trigger.
• Using a global watermark trigger, 67 . 8% of the normal samples are misclassiﬁed to other labels(Infected label)
• Using a 16x16 watermark trigger, 6 .
42% of the normal samples are misclassiﬁed to other labels(Infected label)
• Using a 4x4 watermark trigger leads to 1 .
41% of the normal samples misclassiﬁed •
Using a 2x2 watermark trigger leads to 0 .
24% of the normal samples misclassiﬁed • Training with normal examples results in 0 .
2% of the normal samples misclassiﬁed.
B. F EATURE S PACE B ACKDOOR
Feature space backdoor attack which employs a generative model to transfer the style of a normal image into a trigger.
However, the style transfer leads to signiﬁcant visual changes which becomes less stealthy.
For physical attacks like surveillance cameras, this approach is hard to implement because it requires changing the background as well.
While our work is mostly focused on the input space trigger, we believe the latent space problem is beyond our threat model as well as focused problem and could be an interesting future work (for example, consider using an off-the-shelf style transfer model to evaluate the model’s sensitivity towards different style effects.
C P ROOFS C. T HE PROOF OF L EMMA
Lemma 1.
Given a continuous differential function f deﬁned on R n and vectors x , h ∈ R n , there exists a real value M ≥ | ∂f/∂h
i | ∀ 1 ≤ i ≤ n such that
[Equation]
Proof of Lemma 1: The ﬁrst order Taylor expansion gives
[Equation]
) where R ( h ) is the Lagrange remainder, i.e. ,
[Equation]
· h i
[Equation]
for some c ∈ (0 , 1) .
[Equation]
Let M ≥ | ∂f ( x ) /∂x i | ∀ x , we have
[Equation]
So, we have
[Equation]
C.
T HE PROOF OF L EMMA 2 – A THEORY ON THE LINEAR CASE
We perform a theoretical study of linear models to shed light about our empirical ﬁnding.
Here, we restate Lemma 3 below for convenience.
Lemma 2.
Given a linear model parameterized by θ optimized on N o original training examples and N b backdoored training examples with an objective in Eq.
2 and a mean-squared-error loss, the adversarial solution ˆ µ optimized using Eq.
6 will be dominated by input dimensions corresponding to backdoor mask m when N b is large, i.e. ,
[Equation]
Proof of Lemma 2: To ease the reading of notations, we use ( · ) a × b to represent matrix with a rows and b columns, respectively.
Assuming φ ( x ; θ )
= θ (cid:124) x is a linear model whose output is a ﬂoating number, and the loss function is (cid:96) 2 , i.e., (cid:96) ( x, y ) =
( x − y ) 2 .
Then Eq.
2 becomes a linear regression problem and can be represented in the matrix form of least square, i.e. ,
[Equation]
where ˆ x i =
(1 − m ) (cid:12) x (cid:48)
i + m (cid:12) ∆ is an backdoor example converted from a normal example x (cid:48)
i and ˆ y i =
y t .
X is the matrix composed of row vectors x i and Y is the matrix composed of row vectors y i .
The solution of this least square can be found by setting gradient to zero, such that
[Equation]
It is known that a normal training without backdoors will lead to the pseudo-inverse solution θ =
( X (cid:124) X ) − 1 X (cid:124) Y .
Without loss of generality, we assume the backdoor trigger occupies the last q dimensions.
And the ﬁrst p dimensions are normal values, i.e. , q = d − p + 1 where d is the total number of dimensions.
So, we can rewrite the input X
=
[ X 1: p , X p : d ] and the mask m =
[ 0 1 × p , 1 1 × ( d − p +1) ] .
Let δ be the sub-vector of ∆ such that δ = ∆ p
+1:
d , a q dimensional row vector.
Then we have ˆ X =
[ X 1: p , 1 N b × 1 δ ] where N b is the total number of backdoor examples.
Then,
[Equation]
Similarly, we ﬁnd ˆ Y = 1 y t and assume zero mean of the input, i.e., (cid:80) x i = 0 , then we have
[Equation]
While N b becomes larger, both ˆ X (cid:124)
ˆ X
and ˆ X (cid:124)
ˆ Y increase signiﬁcantly.
When N b → ∞ , Eq.becomes
[Equation]
In another word, we know the normalized values in the non-masked area of θ approach 0 such that
[Equation]
Now, let us look at the adversarial objective in Eq. 8, which is equivalent to
[Equation]
the gradient of which is
[Equation]
Since 2( θ (cid:124) ( x + µ ) − y t ) is a scalar and each of the gradient updates is conducted by µ = µ + λ ∇ µ G , we know no matter how many gradient update steps are applied, µ , initialized at 0, always moves in the exact same direction of θ , i.e. , ˆ µ (cid:107) θ .
In this case, ˆ µ will also be dominant by the last q dimensions, or the dimensions corresponded to the mask m .
In another word, we have
[Equation]
C. P ROOF OF L EMMA 3 – A N ANALYSIS FOR K ERNEL R EGRESSION
Lemma 3.
Suppose the training dataset consists of N o original examples and N b backdoor examples, i.i.d.
sampled from uniform distribution and belonging to k classes.
Each class contains equally N o /k normal examples.
Let φ be a multivariate kernel regression with the objective in
Eq. 2, an RBF kernel K ( · , · ) Then the adversarial solution ˆ µ to Eq.
6 under cross-entropy loss (cid:96) (ˆ y, y )
= − (cid:80)
i
y i log ˆ
y i ( y ∈ { 0 , 1 } k is the one-hot label vector) should satisfy that
[Equation]
Proof of Lemma 3: The output of φ is a k dimensional vector.
Let us assume φ t ( · ) ∈ R be the output corresponding to the target class t .
We know the kernel regression solution is
[Equation]
(24) where ˆ x i =
(1 − m ) (cid:12) x i + m (cid:12) ∆ is the backdoored example and ˆ y i is the corresponding onehot label (so we always have ˆ y i,t = 1 ).
Because the examples are evenly distributed, there are
N/k examples labeled positive for class t .
Without loss of generality, we assume y j,t = 1 when j ∈ [1 , N o /k ] and y j,t = 0 otherwise.
Then the regression solution becomes
[Equation]
Please note the regression above is derived with a mean squared loss.
In the adversarial analysis Eq.
6, we can use any alternative loss function.
In this Lemma, we assume the loss in the adversarial analysis is cross-entropy, which is a common choice.
So, we have
[Equation]
where S i ( · ) =
(cid:80) y
j = i K ( · , x j ) and S = S 1 + S 2 + . . .
+ S k
and
ˆ S = (cid:80) n b i =1 K ( · , ˆ x i ) .
The derivative of loss w.r.t.
x becomes
[Equation]
[Equation]
By using gradient descent, ˆ µ moves along the negative direction of the loss gradient, i.e. ,
[Equation]
where a, b are both positive scalar values.
When N b becomes large, the ﬁrst term dominates the gradient direction, i.e. , ∆ µ ∝ ∂ ˆ S∂x .
Since we assume the kernel K ( x, x (cid:48) )
=
exp( − γ (cid:107) x − x (cid:48) (cid:107) 2 ) is RBF, then we have its derivative
[Equation]
Then we have
[Equation]
Without loss of generality, we assume the mask m =
[ 0 1 × p , 1 1 × ( d − p +1) ] (where ﬁrst p dimensions are 0s and the remaining are 1s).
Then the ﬁrst p dimensions of ˆ x i are the same as those of x i , while the remaining dimensions equivalent to those of ∆ .
Case 1: p + 1 ≤ j ≤ d .
The j -th dimension of ∂ ˆ S∂x can be written as
[Equation]
So at convergence, µ j = δ j − z j for the last d − p + 1 dimensions.
The expectation is E [ µ j ] =
δ j − E [ z j ] .
Case 2: 1 ≤ j ≤ p .
For the ﬁrst p dimensions, we have
[Equation]
Since µ s + z s = δ s =
ˆ x is = x is for p + 1 ≤ s ≤ d , then we have
[Equation]
Since N b is large and ˆ x
i is i.i.d.
, then the summation can be approximated with integration, i.e. ,
[Equation]
When γ is large, the integration is the expectation of the offset to the center of a Gaussian distribution (centered at z j + µ j ).
Due to symmetry, the derivative becomes 0 for most of the µ locations.
This would be the common cases because a smaller γ means examples with longer distance can more signiﬁcantly inﬂuence the function output (which is not desired in a classiﬁer with high accuracy).
Nonetheless, in rare cases when the γ becomes small, we can still ﬁnd the stationary point of the derivative.
Given the fact that e − γ
[Equation]
2 is always positive, we know (cid:16)
[Equation]
S∂µ
[Equation]
j = 0 = ⇒
z
j + µ j − x j = 0 .
But it is impossible to reach.
Again, utilizing symmetry of the normal distribution and the fact that x is i.i.d.
uniform, we ﬁnd when z j + µ j − E
[ x j ] < 0 , the derivative is positive; and when z j + µ j − E
[ x j ] > 0 , the derivative is negative.
Since we use gradient descent to ﬁnd the optimal ˆ µ .
We can conclude that, at the convergence, z j + µ j = E
[ x j ] .
Then E [ µ j ] =
E [ x j ] − E [ z j ] = 0 because z is sampled from the same distribution of x .
To summarize,
[Equation]
Then we know, as N b → ∞ ,
[Equation]
D M ONTE C ARLOS GRADIENT ESTIMATION
Leveraging the Monte Carlo based estimation to craft target adversarial perturbations for the targeted label y t typically requires x and x t , where x t is a legitimate sample satisﬁes f ( x t ) =
y t .
Firstly, x is forced to approach the decision boundary of f ( · ; θ ) for the target label y t ( φ ( x, y t )
→ 0 . 5 ), which stimulates the efﬁciency and accuracy of gradient estimation.
We can make φ ( x, y t )
→ 0 .
5 through projection process:
[Equation] [Equation]
α ∈ (0 , 1) is a parameters for projection.
During each gradient estimation procedure, α is set through binary search (Chen et al., 2020a).
After that, we leverage Monte Carlo sampling to estimate (cid:102) ∇ φ x ( x, y t ) , its procedure can be expressed as:
[Equation] [Equation]
where { µ (cid:48) } Ni =0 are perturbations
i.i.d sampled from uniform distribution and δ is a small positive value representing the magnitude of perturbation.
S is an indicator function such that
[Equation]
[Equation]
To further eliminate the variance induced by Monte Carlo sampling, we can improve (cid:102) ∇ φ x ( x, y t ) via:
[Equation]
(cid:48)
i (45)
Using cosine angle to measure the similarity between (cid:103) ∇ x φ ( x, y t ) and ∇ x φ ( x, y t ) , previous work (Chen et al., 2020a) have proved that:
[Equation]
thus a smaller δ (0.01) is selected for ensuring the efﬁcacy of gradient estimation.
Regarding the constraints on || µ || 1 , we further processed the estimated gradient via:
[Equation]
We refer readers to (Chen et al., 2020a) for the complete algorithm for gradient estimation.
E T HE DETAILED CONFIGURATIONS FOR EXPERIMENTAL DATASETS AND TRIGGERS
[Table]
The detailed information for each task is included in the Table.
E.
The data augmentation utilized for building each model is following https://keras.io/zh/examples/cifar10_ resnet/ .
F T HE ACCURACY AND ATTACK SUCCESS RATE (ASR) FOR EVALUATED MODELS
The accuracy and ASR for the evaluated models for each task in included in Table.
[Table]
G T
HE AMOUNTS OF EVALUATED MODELS FOR VARIOUS TASK AND ATTACK APPROACHES
For each attack and task, we build 60 infected models and 60 uninfected models, respectively.
Notably, the uninfected models for different attack approaches are randomly selected and different.
Each uninfected and infected model sets are built evenly upon ResNet-18, ResNet-44, ResNet-56,
DenseNet-33, DenseNet-58 these ﬁve models.
H D ETAILS OF VISUALIZATION PROCESS
We further investigate whether AEVA can reveal the singularity phenomenon of labels infected with different attack approaches.
We randomly select an infected model infected by each attack approach and an uninfected model for TinyImageNet task.
Notably, all these selected models are correctly identiﬁed by our approach.
We plot the corresponding normalized adversarial perturbation µ in
Fig.
10, where demonstrates that AEVA can accurately and distinguish the uninfected and infected labels and reveal the singularity phenomena of backdoor triggers.
[Figure]
D ETAILED ROC FOR VAROUS TASKS AND ATTACK APPROACHES
[Figure]
J T HE COMPARISON RESULTS BETWEEN AEVA AND NC, DL-TND
[Table] [Table]
[Table]
K E XPERIMENTS FOR DIFFERENT TRIGGERS K. E VALUATION ON DYNAMIC AND SPARSE BUT NOT COMPACT TRIGGERS
We evaluate AEVA for CIFAR-10 task under the dynamic and non-compact triggers scenario.
We randomly generate three non-compact triggers(shown in Figure. 12) and perform backdoor attacks against a randomly selected label using these dynamic triggers.
We test AEVA using 10 different
DNN models, which are evenly built upon ResNet-18, ResNet-44, ResNet-56, DenseNet-33 and
DenseNet-
[Figure]
AEVA successfully identify all models embedded with dynamic backdoors and the corresponding infected labels.
The detailed Aggregated Global Adversarial Peak(AGAP) values and Anomaly
Index given by the MAD outlier detection for infected and uninfected labels are shown in Fig.
[Figure]
We ﬁnd AEVA predicts all backdoored models as infected with Anomaly Index ≥ 4 .
Such results demonstrate that AEVA is resilient to dynamic and sparse but not compact triggers.
[Figure]
We also evaluate AEVA on three different triggers with different shapes, which are shown in Fig.
For each trigger, we build 30 models on TinyImageNet.
Each set of models are evenly built upon
ResNet-18, ResNet-44, ResNet-56, DenseNet-33, DenseNet-58 these architectures.
The results are shown in Table.
[Figure]
[Figure]
L R ESULTS FOR IMPACT OF THE NUMBER OF IMAGES PER LABELS .
We further explore how many samples for each label are required by AEVA to be effective.
We test
AEVA on TinyImageNet and the experimental conﬁgurations are consistent with Sec.4.1( i.e. ,infected and 240 benign models).
Our results are illustrated in the Fig.
15, which plots the metrics against the number of samples per class.
We ﬁnd that our approach requires more samples (32) for each class to achieve optimal performance.
[Figure]
M T HE IMPACT OF THE NUMBER OF THE LABELS FOR AVAILABLE IMAGES .
Since our approach aims to detect backdoors in the black-box setting, a realistic scenario is that while determining if a speciﬁc label is infected, the clean data from all other labels may be unavailable to the defender.
So, in this section, we investigate the impact of the number of labels for which clean data is available for our approach.
We test our approach on TinyImageNet using 60 models infected with BadNets ( 4 × 4 squares) following the conﬁgurations of Sec.
We vary the number of uninfected labels for available samples between [1, 200], randomly sampled from the 200 labels available in the TingImageNet dataset.
Figure.
16 illustrates our results by plotting the metrics against the number of available labels.
We observe that even with eight available labels, our approach can still detect backdoors in most cases, i.e., with merely 4% of the labels being available, our approach is still viable.
This study exhibits our approach’s practicality, which can work in the blackbox setting and on relatively small available data.
Notably, with fewer labels AEVA will always correctly tag the uninfected models.
This is because few labels will cause the anomaly index low thus resilient to uninfected models.
[Figure]
N M ULTIPLE TRIGGERS WITHIN THE SINGLE INFECTED LABEL SCENARIOS
We also investigate the impact caused by multiple triggers within the single infected label.
We here choose different 4 × 4 squares located at different places as the triggers implemented following
BadNets.
We randomly select a label as the infected label.
We built 60 infected models with different architectures( i.e. , ResNet-18, ResNet-44, ResNet-56, DenseNet-33, DenseNet-58).
The results for
TinyImageNet are shown in Table.
Notably, for TinyImageNet, injecting too many triggers ( ≥ 4 ) in the single label would cause the target model’s accuracy drop (i.e., ≥ 3 .
7% ), which is inconsistent with the threat model for backdoor attacks (Gu et al., 2019;
Chen et al., 2017; Liu et al., 2018).
As seen in Table.
8, our approach can still perform effective when injecting two 4 × 4 triggers in the single infected label.
However, when the number of injected triggers becomes more than 2, our approach becomes less effective.
This should be caused by that multiple-triggers would reduce the singularity properties for the adversarial perturbations.
To address this issue, we select the sum of the largest ﬁve points as the µ max instead.
The results are shown in Table.
By selecting more points to calculate the µ max , AEVA can still perform effective under the multiple triggers within the single label scenarios and have no impact on the detection accuracy for uninfected models
[Table] [Table]
O P OTENTIAL ADAPTIVE BACKDOOR ATTACKS
We here consider two potential backdoor attacks which can bypass AEVA.
O. A TTACKS WITH MULTIPLE TARGET LABELS
Since AEVA is sensitive to the number of infected labels, the attacker can infect multiple labels with different backdoor triggers.
Indeed, making multiple labels ( i.e. ≥ 30% ;) infected will make the anomaly indexes produced by the MAD detector signiﬁcantly drop.
However, such attack can only be successfully implemented for some small datasets which own a few labels(e.g., CIFAR-10, etc).
Reported by (Wang et al., 2019), the state-of-the-art model (DeepID) for Youtube Face dataset can not maintain the average attack success rate and model accuracy at the same time when more than 15 .
6% labels are infected.
In another word, when multiple labels are infected, the infected model’s accuracy is likely to get worse if the attacker wants to keep attack success rate.
So too many infected labels will reduce the stealth of the attack.
We conducted experiments on TinyImageNet, which reveals that when over 14 labels are infected with different triggers, the model accuracy (ResNetand DenseNet-58) decreases to ≤ 57 .
61% (around 3% lower than a normal model) when preserving the attack effectiveness (ASR ≥ 97 .
12% ) for each infected label.
O. A TTACK WITH LARGE TRIGGERS
Another potential adaptive backdoor attack is that the attacker would implement a dense backdoor trigger which would alleviate the singularity phenomenon.
As we claim in Section 4.3, a dense backdoor trigger would appear visually-distinguishable to the human beings which perform less stealthy, that is also reported by Neural Cleanse (Wang et al., 2019).
Indeed, there exists an interesting exceptional watermark attacks.
However, we empirically prove that such dense watermark attack (Chen et al., 2017) would make infected DNNs non-robust which would make the inputs sensitive to the random noise.
The details are included in Appendix B.