A DVERSARIAL A TTACK D ETECTION T HROUGH N ETWORK T RANSPORT D YNAMICS
Anonymous authors
Paper under double-blind review
A BSTRACT
Adversarial attacks are perturbations to the input that don’t change its class for a human observer, but fool a neural network into changing its prediction.
In this paper, we propose a detector of such attacks that is based on the view of residual networks as discrete dynamical systems.
The detector tells clean inputs from abnormal ones by comparing the discrete vector fields they follow throughout the network’s layers before the final classification layer.
We compare this detector favorably to other detectors on seen and unseen attacks.
We also show that regularizing this vector field during training makes the network more regular on the data distribution’s support, thus making the network’s activations on clean samples more distinguishable from those of abnormal samples.
This regularization of the network’s dynamics improves the performance of any detection method that uses the internal embeddings as inputs, while also improving the network’s test accuracy.
I NTRODUCTION
Neural networks have improved performances on many learning tasks, including image classification.
They are however vulnerable to adversarial attacks (Szegedy et al.
These attacks modify an image in a way that is imperceptible to the human eye but that fools the network into wrongly classifying the modified image.
These attacks can fool other networks (Moosavi-Dezfooli et al. (2017)), can be carried out in the physical word (e.g. causing autonomous cars to misclassify road signs (Eykholt et al.
(2018))), and can be generated without access to the network (Liu et al.
Developing networks that are robust to such attacks or accompanied by detectors that can detect them is indispensable to deploying them safely in the real word (Amodei et al.
In this paper, we focus on attack detection.
Networks trained with a softmax classifier produce overconfident predictions even for out-of-distribution inputs (Nguyen et al.
This makes it difficult to detect such inputs via the softmax outputs.
A detector is a system capable of predicting whether an input at test time has been adversarially modified or not.
Detectors are trained on a dataset made up of clean and attacked inputs, after the network has been trained.
While simply training the detector on the inputs has been tried, using their intermediate embeddings works better (Carlini &
Wagner (2017b)).
Detection methods vary by which activations to use and how to process them to extract the features that are fed to the classifier that tells clean samples from attacked ones.
We make two contributions.
First, we propose an adversarial detector that is based on the view of neural networks as dynamical systems that move inputs in space, time represented by depth, to separate them before applying a linear classifier (Weinan (2017)).
Our detector follows the trajectory of samples in space, through time, to differentiate clean and attacked images.
The statistics that we extract are the positions of the internal embeddings in space approximated by their norms and cosines to a fixed vector.
Given their resemblance to the Euler scheme for differential equations, residual networks (He et al.
(2016a;b); Weinan (2017)) are particularly amenable to this analysis.
Skip connections and residuals are basic building blocks in many architectures (e.g. EfficientNet (Tan & Le (2019)) and MobileNetV2 (Sandler et al.
(2018))), and ResNets and their variants (e.g. WideResNet (Zagoruyko & Komodakis (2016)) and ResNeXt (Xie et al.
(2017))) remain competitive (Wightman et al.
Also, Wu et al.
(2020) show an increased vulnerability of residual-type architectures to transferable attacks, precisely because of the skip connections.
This motivates the need for a detector that is well adapted to residual-type architectures.
But the analysis and implementation can extend immediately to any network where most layers have the same input and output dimensions.
We test our detector on seven attacks (white-box and black-box), three datasets and networks, comparing it favorably to the reference Mahalanobis detector (Lee et al.
that we largely outperform.
Building on this dynamical viewpoint, our second contribution is to use the transport regularization during training proposed in Karkar et al.
(2020) to make the activations of adversarial samples more distinguishable from those of clean samples, thus making adversarial detectors perform better, while also improving generalization.
We prove that the regularization achieves this by making the network more regular on the support of the data distribution.
This does not make it more robust to attacks, but it makes its behavior on them more distinguishable, since they tend to lie outside the data manifold.
R ELATED W ORK
An early idea for detection was to use a second network (Metzen et al.
However, this network can also be adversarially attacked.
More popular statistical approaches include LID (Ma et al.
(2018)), which trains the detector on the local intrinsic dimensionality of activations, or the Mahalanobis detector (Lee et al.
(2018)), which trains the detector on the Mahalanobis distances between the activations and a Gaussian fitted to them during training.
Detectors trained in the Fourier domain of activations have also been proposed in Harder et al.
See Aldahdooh et al.
for a review.
Lipschitz regularity has been linked to adversarial robustness since their introduction (Szegedy et al.
Our second contribution is to regularize the network in a way that makes it Hölder-continuous (a weaker version of Lipschitz continuity), but only on the support of the data distribution.
This won’t make it more robust to attacks, but it will make the activations of the clean samples closer to each other and further from those of abnormal out-of-distribution samples, thus making adversarial detection easier.
This is illustrated in a toy example in 2 dimensions in Figure 1.
(green) after blocks 6 and 9 of a ResNet-9.
In the second row, we add our proposed regularization during training, which makes the movements of the clean points (red and blue) more similar to each other and more different from the movements of the green out-of-distribution points than when using the vanilla network in the first row.
In particular, without the regularization, the green points have a similar dynamic to the red points lying on the data distribution support which is undesirable.
Estimations of the Lipschitz constant of a network have been used as estimates of its robustness to attacks in Weng et al.
; Szegedy et al.
; Virmaux & Scaman (2018); Hein & An- driushchenko (2017), and making the network more Lipschitz (e.g. by penalizing an upper bound on its Lipschitz constant) has been used to make it more robust to attacks in Hein & Andriushchenko (2017) and Cisse et al.
These regularizations (Cisse et al.
(2017); Gu & Rigazio (2015)) often work directly on the weights of the network, therefore making the network more regular on the entire input space, as opposed to our approach.
Adversarial attacks tend to lie outside the data manifold, particularly in its co-dimensions.
This is a common observation and explanation for why adversarial attacks are easy to find in high dimensions (Gilmer et al. ; Tanay & Griffin (2016); Song et al. ; Ma et al. ; Samangouei et al.
; Khoury & Hadfield-Menell (2018); Alemany & Pissinou (2022); Feinman et al. (2017). To the best of our knowledge, Pang et al. (2018) is the only other method that attempts to improve detection by encouraging the network to learn latent representations that are more different between clean and attacked samples.
They do this by replacing the cross-entropy loss by a reverse cross-entropy that encourages uniform softmax outputs among the non-predicted classes. However, this method negatively impacts test accuracy in some cases, while our regularization always improves it, and we test our method on bigger datasets and networks.
In the context of defending against adversarial attacks, the view of residual networks as transport systems was used in Wang et al.
(2019) to relate injecting noise into residual networks and ensembling them to approximating the Feynman-Kac formula for convection-diffusion, which makes the level sets more regular and the network more robust to adversarial attacks.
B ACKGROUND
Our detector is based on the dynamical viewpoint of neural networks that followed from the analogy between residual networks and the Euler scheme made in Weinan (2017). We present this in Section 3.2. The trajectory regularization we use was proposed in Karkar et al.
(2020) to improve generalization and we also present it in Section 3.2. The regularity results that this regularization provides for the network require the use of optimal transport theory, which we present in Section 3.1.
O PTIMAL T RANSPORT
Let α and β be absolutely continuous densities on a compact set Ω ⊂ R d .
The Monge problem is to look for a map T : R d → R d moving α to β , i.e. T ♯ α = β , with minimal Euclidean transport cost:
[Equation]
and this problem has a unique solution T ⋆ .
An equivalent formulation of the Monge problem in this setting is the dynamical formulation.
Here, instead of directly pushing points from α to β through T , we continuously displace mass from time 0 to 1 according to velocity field v t : R d → R d .
We denote ϕ x t the position at time t of the particle that was at x ∼ α at time 0.
This position evolves according to ∂ t ϕ xt
= v t ( ϕ xt ) .
Rewriting the constraint, Problem (1) is equivalent to the dynamical formulation: min
[Equation]
s [Equation] v t
[Equation] ϕ xt ) for x ∈ support ( α ) and t ∈ [0 , 1[ , ϕ · 0 = id , ( ϕ · 1 ) ♯ α = β
3. L EAST A CTION P RINCIPLE R ESIDUAL N ETWORKS
A residual network made up of M residual blocks applies x m +
=
x m + hr m ( x m ) for 0 ≤ m < M , with x 0 being the input and h =1 in practice.
The final point x M is then classified by a linear layer.
The dynamic view considers a residual network as an Euler discretization of a differential equation:
[Equation]
(3) where r m approximates the vector field v t at time t = m/M .
The dynamic view allows to consider that ResNets are transporting their inputs in space by following a vector field to separate them, the depth representing time, before classification by a linear layer.
Karkar et al.
(2020) look for a network that solves the task while having minimal transport cost:
[Equation]
[Equation]
where T is made up of the M residual blocks, α is the data distribution, F is the classification head and L (( F, T ♯ α )) is the (cross-entropy) loss obtained from classifying the transformed data distribution T ♯ α through F .
Given Section 3.1, the corresponding dynamical version of (4) is inf v,F (cid:90)
[Equation]
subject to ∂ t ϕ xt =
v t ( ϕ xt ) for x ∈ support ( α ) and t ∈ [0 , 1[ , ϕ · 0 = id , L ( F, ( ϕ · 1 ) ♯ α ) =(5)
Karkar et al.
(2020) show that (4) and (5) are equivalent and have a solution such that T is an optimal transport map.
In practice, (5) is discretized using a sample D from α and an Euler scheme, which gives a residual architecture with residual blocks r m (parametrized along with the classifier by θ ) that approximate v .
This gives the following problem
[Equation]
subject to φ xm[-+
-]
[Equation] = x for all x ∈ D [Equation] L ( θ ) =(6)
In practice, we solve Problem (6) using a method of multipliers, which amounts to penalizing the discrete transport cost C (see Section 4.2.1).
Our contribution is to show, theoretically and experimentally, that besides improving test accuracy, this makes adversarial attacks easier to detect.
M ETHOD
We take the view that a residual network moves its inputs according to a discrete vector field to separate them.
Points in each class follow similar trajectories.
Heuristically, for a successful adversarial sample that lies close to clean samples, the vector field it follows has to be different at some step from that of the clean samples (in its original class and in other classes), so that it can join the trajectory of the points in another class.
In Section 4.1, we present how to detect adversarial attacks by looking at these trajectories.
In Section 4.2, we discuss the link between the regularity of neural networks and their robustness to adversarial attacks and use the transport regularization (6) to improve detectability of adversarial attacks.
4. D ETECTION
Given a residual network that applies x m +
=
x m + hr m ( x m ) to an input x 0 for 0 ≤ m<M , we consider the embeddings x m for 0 <m ≤ M , or the residues r m ( x m ) for 0 ≤ m<M .
To describe their positions in space, we take their norms and their cosine similarities with a fixed vector as features to train our adversarial detector on.
Using only the norms already gave good detection accuracy. Cosines to other orthogonal vectors can be added to better locate the points at the price of increasing the numbers of features. We found that using only one vector already gives state-of-the-art detection, so we only use the norms and cosines to a fixed vector of ones.
We train the detector (a random forest in practice, see Section 5.2) on these features.
The embeddings x m and the residues r m ( x m ) can equivalently describe the trajectory of x 0 in space through the blocks, but in practice we find that the residues r m ( x m ) give slightly better detection performance.
The number of features is twice that of residual blocks (a norm and a cosine per block).
This is of the same order as for other popular detectors such as Mahalanobis (Lee et al.
and LID (Ma et al.
that extract one feature per residual stage (a residual stage is a group of residual blocks that keep the same dimension).
Even for common large architectures, twice the number of residual blocks is still a small number of features for training a binary classifier (e.g. a ResNet152 has 50 blocks).
More importantly, the features we extract (norms and cosines) are simple and quick to calculate, whereas those of other methods require more involved statistical computations on the activations.
We include in Appendix D.7 a favorable time comparison of our detector to the Mahalanobis detector.
Another advantage of our detector is that it does not have a hyper-parameter to tune unlike the
Mahalanobis and LID detectors.
4. R EGULARIZATION
Regularity of neural networks (typically Lipschitz continuity) has been used as a measure of their robustness to adversarial attacks (Weng et al.
; Szegedy et al.
; Virmaux & Scaman (2018); Hein & Andriushchenko (2017);
Cisse et al.
Indeed, the smaller the Lipschitz constant L of a function f satisfying ∥ f ( x ) − f ( y ) ∥
≤ L ∥ x − y ∥ , the less f changes its output f ( y ) for a perturbation (adversarial or not) y of x .
Regularizing a network to make it more Lipschitz and more robust has therefore been tried in Hein & Andriushchenko (2017) and Cisse et al.
For this to work, the regularization has to apply to adversarial points, i.e. outside the support of the clean data distribution.
Indeed, the Lipschitz continuity obtained though most of these methods and analyses apply on the entire input space R d as they regularize the weights of the network’s matrices.
We propose here an alternative approach where we regularize the network only on the support of the input distribution, making it η -Hölder on this support (a function f is η -Hölder on X if ∀ a, b ∈ X , we have ∥ f ( a ) − f ( b ) ∥ ≤ C ∥ a − b ∥ η for some constants C> 0 and 0 <η ≤ 1 , and we denote this f ∈ C 0 ,η ( X ) ).
Since this result does not apply outside the input distribution’s support, particularly in the adversarial spaces, then this regularity that only applies to clean samples can serve to make adversarial samples more distinguishable from clean ones, and therefore easier to detect.
We show experimentally that the behavior of the network will be more distinguishable between clean and adversarial samples in practice in Section 5.1.
We discuss the implementation of the regularization in Section 4.2.1 and prove the regularity it endues the network with in Section 4.2.2.
4.2. I MPLEMENTATION
We regularize the trajectory of the samples by solving Problem (6).
This means finding, among the networks that solve the task (condition L ( θ ) =
in (6)), the network that moves the points the least, that is the one with minimal kinetic energy.
The residual functions r m we find are then our approximation of the vector field v that solves the continuous version (5) of the problem.
We solve Problem (6) via a method of multipliers: since L ≥ 0 , Problem (6) is equivalent to the min-max problem min θ max λ> 0 C ( θ ) + λ L ( θ ) , which we solve, given growth factor τ > 0 , and starting from initial weight given to the loss λ 0 and initial parameters θ 0 , through
[Equation]
We use SGD for s> 0 steps (i.e. batches) for the minimization in the first line of (7), starting from the previous θ i .
When using a ResNeXt, where a residual block applies x m +1 = ReLU ( x m + r m ( x m )) , we regularize the norms of the true residues x m +1 − x m instead of r m ( x m ) .
The true residue x m +1 − x m can be used for any non-residual architecture on layers that have the same input and output dimension, allowing to apply the method to any network with many such layers.
4.2. T HEORETICAL ANALYSIS
We first prove a regularity result on each residual r m that applies on the clean data points moving according to v solution to (5).
We take Ω ⊂ R d convex and compact and the data distribution α ∈ P (Ω) absolutely continuous and such that δ Ω is α -negligible.
We suppose that there exists an open bounded convex set X ⊂ Ω such that α is bounded away from zero and infinity on X and is zero on its complement X ∁ .
From Karkar et al.
(2020), we know that problems (4) and (5) are equivalent and have solutions ( T, F ) and ( v, F ) such that T is an optimal transport map between α and β :=
T ♯ α .
We suppose that β is absolutely continuous and that there exists an open bounded convex set Y ⊂
Ω such that β is bounded away from zero and infinity on Y and is zero on Y ∁ .
In the rest of this section, v solves (5) and we suppose that we find a solution to the discretized problem (6) that is an ε/ 2 -approximation of v , i.e. ∥ r m − v t m ∥ ∞ ≤ ε/ 2 for all 0 ≤ m < M , with t m = m/M .
Theorem 4.1.
For a, b ∈ support ( α t m ) where α t := ( ϕ · t ) ♯ α with ϕ solving (5) along with v , we have
[Equation]
for constants K > 0 and 0 < ζ 1 ≤ ζ 2 ≤ 1 ,
Proof.
A solution v to (5) exists and is linked to an optimal transport map T that is a solution to (4) by v t
= ( T − id ) ◦ T − 1 t where T t :=
(1 − t ) id + tT which is invertible (see Appendix A).
By Theorem A.1 in Appendix A, being an optimal transport map, T is η -Hölder on X .
So for all a, b ∈ support ( α t ) and t ∈ [0 , 1[ , where α t = ( ϕ · t ) ♯ α = ( T t )
♯ α with ϕ solving (5) with v , we have
[Equation]
Since ( α t ) 1 t =0 is a geodesic between α and β =
α 1 = T ♯ α , then ( α s ) ts =0 is a geodesic between α and α t (modulo reparameterization to [0 , 1] ).
And since α s = ( T s )
♯ α , the map T t is an optimal transport map between α and α t .
Therefore its inverse T − 1 t is an optimal transport map (see Theorem
A.2 in Appendix A) and is η t -Hölder with 0 <η t ≤ 1 (being a push-forward by T t , the support of α t satisfies the conditions of Theorem A.1 in Appendix A).
Therefore, for all a, b ∈ support ( α t )
[Equation]
and for all a, b ∈ support ( α t m )
[Equation]
10) by the hypothesis on r and the triangle inequality.
Let K := max m C [Equation]
η min m η t m and ζ 2 :=
max m η t m .
Then, we have the desired result immediately from (10).
Remark 4.2 .
If the convexity hypothesis on the support Y of the target distribution β is too strong, we still get the same results almost everywhere.
More precisely, if the set Y such that β is bounded away from zero and infinity on Y and is zero on Y ∁ is open and bounded but not convex, then the solution map T is η -Hölder almost everywhere on X (see Appendix A).
Remark 4.3 .
If the distributions α and β in Theorem 4.1 are C k,η (i.e all derivatives up to the k -th derivative are η -Hölder), then the optimal transport map T is C k +1 ,η .
This means that the more regular the data, the more regular the network we find.
We now look at the activations for inputs that are not necessarily in X .
For inputs a 0 and b 0 , the intermediate embeddings are a m +
= a m + hr m ( a m ) and b m +
=
b m + hr m ( b m ) , and the residues we (and other methods) use to compute features for adversarial detection are r m ( a m ) and r m ( b m ) .
We therefore want to bound ∥ r m ( a m ) − r m ( b m ) ∥ as a function of the input variation ∥ a 0 − b 0 ∥ .
This is usually done by multiplying the Lipschitz constants of each residual block up to depth m , which leads to an overestimation (Huster et al. (2019)), or through more complex estimation algorithms (Virmaux & Scaman (2018) ; Latorre et al.
(2020); Bhowmick et al.
Bound (8) allows us avoid multiplying the Hölder constants of the block through T − 1 t . We have the following result (proof in Appendix C), which shows the increased regularity guarantee when the inputs a 0 and b 0 are in X :
Theorem 4.4.
For all a 0 , b 0 ∈ X , we have, for constants C, L > 0 , that
[Equation]
The term ∥ a m − ϕ a 0 t m ∥ (likewise ∥ b m − ϕ b 0 t m ∥ ) is the distance between the point a m after m residual blocks and the point ϕ a 0 t
m we get by following the vector field v up to time t m .
Since (8) applies only on the data support, an extra term has to be introduced if a 0 is not in X (see the proof).
Under more regularity hypotheses on v , it is possible to bound these terms.
Indeed, if v is C 1 and Lipschitz in x (this is not stronger than the regularity we get on v through our regularization, because it does not give something similar to bound (8)), we have, for constants R, S > 0 , and for all a 0 , b 0 ∈ R d , that ∥ r
[Equation]
E XPERIMENTS
White-box attacks have access to the network’s architecture and weights, while black-box attacks don’t have any knowledge about the network.
We test our method on five popular white-box attacks: FGM (Goodfellow et al.
, PGD (Madry et al.
BIM (Kurakin et al.
(2017)), DF (Moosavi-Dezfooli et al.
and CW Carlini & Wagner (2017a), and two popular black-box attacks: HSJ (Chen et al. and BA (Brendel et al.
We present these attacks in Appendix D.1.
For FGM, PGD and BIM, we use a maximal perturbation size of ϵ = 0 .
For CW, we use the L 2 version.
We use the adversarial robustness toolbox (ART) (Nicolae et al.
to generate the attacks.
We compare our method to the popular Mahalanobis detector (Lee et al.
We use three architectures and three datasets for our experiments: ResNeXt50 on CIFAR100,
ResNet110 on CIFAR10 and WideResNet on TinyImageNet.
For each architecture-dataset pair, the network is trained with the same hyper-parameters with and without the transport regularization.
For the regularization, we use (7) with τ = 1 , s = 1 and λ 0 = 1 for all networks.
We don’t fine-tune these hyper-parameters much and they are chosen to improve validation accuracy during training not adversarial detection.
The other training details are in Appendix D.3.
A network trained with the regularization is called a LAP-network (for Least Action Principle).
5. P RELIMINARY EXPERIMENTS
First, the regularization improves generalizations. The vanilla ResNeXt50 has a test accuracy of 76 . 4% on CIFAR100, while the LAP-ResNeXt50 has a test accuracy of 79% . The vanilla ResNet has a test accuracy of 92 .
7% on CIFAR10, while the LAP-ResNet110 has a test accuracy of 93 .
The vanilla WideResNet has a test accuracy of 65 . 14% on TinyImageNet, while the LAP-WideResNet has a test accuracy of 65 . 34% . We further show that the regularization improves training stability by allowing to train deep networks without batch-normalization in Figure 4 in Appendix D.4. We see in Figure 2 that regularizing the transport cost C makes it more distinguishable between clean and attacked samples.
We see that simply using the empirical quantiles of C on clean samples allows us to detect some attacks with near perfect recall and a chosen fixed false positive rate. D ETECTION OF SEEN ATTACKS For detection training, the test set is split in 0 . 1 proportions into two datasets, B1 and B2.
For each image in B1 (respectively B2), an adversarial attack is generated and a balanced detection training set (respectively a detection test set) is created.
We try augmenting the dataset with a randomly perturbed version of each image, to be considered as clean during detection training as in Lee et al.
, but we find that this does not improve detection accuracy.
This dataset creation protocol is standard and is depicted in Figure 3 in Appendix D.2.
We tried limiting the dataset to successfully attacked images only (i.e. images that are classified correctly but whose adversarial versions fool the network) as in Lee et al.
, but we did not find a significant improvement in detection (even on successfully attacked images only).
Points in the detection training set are fed through the network and the features for each detection method are extracted.
The classifiers (logistic regression, random forest and support vector machines from scikit-learn (Pedregosa et al. (2011))) are trained on these features for each detection method and tested on the features extracted from the points in the detection test set. A random forest always works best.
This is done for both the vanilla version of the network and its LAP version.
We tried two commonly used methods to improve detection accuracy for both detection methods: class-conditioning and ensembling.
In class-conditioning, the features are grouped by the class predicted by the network for the image, and a detector is trained for every class.
At test time, the detector trained on the features of the predicted class is used.
A detector trained on all samples regardless of the predicted class is also trained to be used in case a particular class is never targeted by adversarial attacks (which might target certain classes).
We also tried ensembling the class-conditional detector with the class-agnostic one.
By ensembling we mean that an input is considered an attack if at least one detector says so. In the results below, we report the results of the best performing version, which is often this ensemble of the class-conditional detector and the class-agnostic one. 5.2. W HITE - BOX ATTACKS We report the detection accuracy of each detection method on the detection test set for both the vanilla version of the network and its LAP version in Table 1 below.
In each cell, the first number corresponds to the vanilla network and the second to the regularized LAP-network.
These results are averaged over 5 runs and the standard deviations are in Tables 4, 5 and 6 in Appendix D.5.
Our results show two things.
First, our detection method performs better than the Mahalanobis detector, with or without the regularization.
Second, both detection methods work better on the LAP versions of the networks most times.
The Mahalanobis method benefits more from the regularization, but on all attacks, the best detector is always the transport detector.
Results for ResNeXt50 on CIFAR100 are in the first row of Table 1.
Our detector outperforms the
Mahalanobis detector by 1 to 5 percentage points depending on the attack on the vanilla ResNeXt50.
The transport regularization improves the detection accuracy of our detector and of the Mahalanobis detector by 0.7 percentage points on average.
Results for ResNet110 on CIFAR10 are in the second row of Table 1.
Our detector outperforms the
Mahalanobis detector by 9 to 13 percentage points depending on the attack on the vanilla ResNet110.
The transport regularization improves the detection accuracy of our detector by 1.3 percentage points on average and that of the Mahalanobis detector by a substantial 8 percentage points on average.
Results for a WideResNet on TinyImageNet are in the third row of Table 1.
Our detector outperforms the Mahalanobis detector by 11 to 12 percentage points depending on the attack on the vanilla
WideResNet.
The transport regularization does not change the detection accuracy of our detector and improves that of the Mahalanobis detector by 0.8 percentage points on average.
[Table]
B LACK - BOX ATTACKS
Since they are much slower to generate
[Table]
we test detection of seen black-box attacks only on the network-data pair of ResNet110-CIFAR10 in Table 2 below.
The experimental protocol and the
ResNet110 and LAP-ResNet110 networks are the same as above.
Our proposed detector outperforms the Mahalanobis detector by 13 percentage points on HSJ and by 16 percentage points on BA.
The regularization improves the detection accuracy of the Mahalanobis detector by 8 percentage points on HSJ and by 9 percentage points on BA, and that of the transport detector slightly.
The transport detector is still the best detector by 5 percentage points on HSJ and by 7 percentage points on BA.
[Table]
5. D ETECTION OF UNSEEN ATTACKS
An important setting is when we do not know which attack might be used at test time or we only have time to train our detector on a few attacks.
We still want our detector to generalize well to other unseen attacks.
To test this, we use the same networks as above.
The detectors are now trained on the detection training set created using the simplest and quickest attack (FGM) and tested on the detection test sets created using the six other attacks (PGD, BIM, DF, CW, HSJ and BA).
We use only a tenth of the detection test sets for black-box attacks HSJ and BA because they are much slower.
Results are in Table 3 below.
We see that our detection method has very good detection generalization to unseen attacks, even those very different from FGM, comfortably better than the Mahalanobis detector, by up to 18 percentage points in some cases.
These results are averaged over 5 runs and the standard deviations are in Tables 7, 8 and 9 in Appendix D.6.
[Table]
However, this last experiment shows that our approach has some limitations.
We see in Tables 7, 8 and 9 in Appendix D.6 that the regularization does not improve detection accuracy as much anymore, and reduces it in some cases.
It improves it for the Mahalanobis detector on all attacks on the ResNet110-CIFAR10 and WideResNet-TinyImageNet experiments (by up to 10 percentage points), but not on most attacks on the ResNeXt50-CIFAR100 experiment.
It often reduces detection generalization on our detector.
We claim this is because the regularization reduces the variance of features extracted during training on the seen attack, thus harming generalization to unseen attacks.
This explains why detection of PGD and BIM, which are variants of FGM, still often improves.
Fixing this could be an area of future research.
C ONCLUSION
We proposed a method for detecting abnormal samples, based on the dynamical viewpoint of neural networks.
The method examines the discrete vector field moving the inputs to distinguish clean and adversarial samples.
The detector requires minimal processing and computation to extract the features it needs for detection and achieves state-of-the-art detection performances on 3 different architectures and datasets, both on seen and unseen attacks.
We have also used a transport regularization that was previously shown to improve test accuracy to improve the detection accuracy of adversarial detectors.
This regularization makes the embeddings closer to each other on the support of the input distribution, thus making them more distinguishable from those of abnormal samples.
R EPRODUCIBILITY STATEMENT
We provide implementation details in Appendix D.3.
The code is provided as supplementary material and at github.com/advadvadvadvadv/adv , along with weights for the trained ResNet models on CIFAR10 that were used
A B ACKGROUND ON OPTIMAL TRANSPORT
The Wasserstein space W 2 (Ω) with Ω a convex and compact subset of R d is the space P (Ω) of probability measures over Ω , equipped with the distance W 2 given by the solution to the optimal transport problem
[Equation]
where Π( α, β ) is the set of probability distribution over Ω × Ω with first marginal α and second marginal β , i.e. Π( α, β )
= { γ ∈ P (Ω × Ω) | π 1 ♯ γ = α, π 2 ♯
γ = β } where π 1 ( x, y ) =
x and π 2 ( x, y ) =
y .
The optimal transport problem can be seen as looking for a transportation plan minimizing the cost of displacing some distribution of mass from one configuration to another.
This problem indeed has a solution in our setting (see for example Santambrogio (2015); Villani (2008)).
If α is absolutely continuous and ∂ Ω is α -negligible then the problem in (12) (called the Kantorovich problem) has a unique solution and is equivalent to the following problem, called the Monge problem,
[Equation]
(13) and this problem has a unique solution T ⋆ linked to the solution γ ⋆ of (12) through γ ⋆
= ( id , T ⋆ )
♯ α .
Another equivalent formulation of the optimal transport problem in this setting is the dynamical formulation (Benamou & Brenier (2000)).
Here, instead of directly pushing samples of α to β using
T , we can equivalently displace mass, according to a continuous flow with velocity v t : R d → R d .
This implies that the density α t at time t satisfies the continuity equation ∂ t α t + ∇ · ( α t v t ) = 0 , assuming that initial and final conditions are given by α 0 =
α and α 1 = β respectively.
In this case, the optimal displacement is the one that minimizes the total action caused by v :
[Equation]
Instead of describing the density’s evolution through the continuity equation, we can describe the paths ϕ xt taken by particles at position x from α when displaced along the flow v .
Here ϕ xt is the position at time t of the particle that was at x ∼ α at time 0.
The continuity equation is then equivalent to ∂ t ϕ xt =
v t ( ϕ xt ) .
See chapters 4 and 5 of Santambrogio (2015) for details.
Rewriting the conditions as necessary, Problem (14) becomes
[Equation]
and the optimal transport map T ⋆ that solves (13) is in fact T ⋆ ( x ) =
ϕ x 1 for ϕ that solves the continuity equation together with the optimal v ⋆ from (15).
The optimal vector field is related to the optimal map through v ⋆ t =
( T ⋆ − id ) ◦ ( T ⋆ t ) − 1 , where T ⋆ t = (1 − t ) id + tT ⋆ and is invertible.
This simply means that the points move in straight lines and with constant speed from x to T ⋆ ( x ) .
The W 2 (Ω) space is a metric geodesic space, and the geodesic between α and β is the curve α t found while solving (14).
It is also given by α t = ( π t )
♯
γ ⋆ =
( T ⋆t )
♯ α , where π t ( x, y ) =
(1 − t ) x
+ ty .
We refer to Section 5.4 of Santambrogio (2015) for these results on optimal transport.
Optimal transport maps have some regularity properties under some boundedness assumptions.
We mention the following result from Figalli (2017):
Theorem A.1.
Suppose there are X, Y , bounded open sets, such that the densities of α and β are null in their respective complements and bounded away from zero and infinity over them respectively.
Then, if Y is convex, there exists η > 0 such that the optimal transport map T between α and β is
C 0 ,η over X .
If Y isn’t convex, there exists two relatively closed sets A, B in X, Y respectively such that T ∈
C 0 ,η ( X \ A, Y \ B ) , where A and B are of null Lebesgue measure.
Moreover, if the densities are in C k,η , then C 0 ,η can be replaced by C k +1 ,η in the conclusions above.
In particular, if the densities are smooth, then the transport map is a diffeomorphism.
A final result that we mention is the following, which says that the inverse of the optimal transport map between α and β is the optimal transport map from β to α ,
Theorem A.2.
If α and β are absolutely continuous measures supported respectively on compact subsets X and Y of R d with negligible boundaries, then there exists a unique couple ( T, S ) of functions such that the five following points hold
• T : X → Y and S :
Y → X • T # α = β and S # β = α • T is optimal for the Monge problem from α to β • S is optimal for the Monge problem from β to α • T ◦ S
β − a.s.
= id and S ◦ T
α − a.s. =
id
B B ACKGROUND ON NUMERICAL METHODS FOR ODE S
We refer to Quarteroni et al.
(2007) for this quick background on numerical methods for ODEs.
Consider the Cauchy problem x ′ =
f ( t, x ) with initial condition x
( t 0 )
= x 0 and a subdivision t 0 < t 1 < .. <
t N = t
+ T of [ t 0 , t 0 + T ] .
Denote the time-steps h n :=
t n +1 − t n for 0 ≤ n < N and define h max := max h n .
In a one-step method, an approximation of x ( t n ) is x n given by x n +1 − x n h n = ϕ ( t n , x n , h n )
For ϕ ( t, x, h ) =
f ( t, x ) , we get Euler’s method: x n +
= x n + h n f ( t n , x n ) .
Definition B.1.
(Consistency and order)
For a one-step method, the consistency errors e n , for 0 ≤ n < N , are
[Equation]
where x is solution.
The local (truncation) errors are h n e n .
The method is consistent if max | e n | goes to zero as h max goes to zero.
For p ∈ N ∗ , the method has order p if max | e n | ≤ Ch p max for a constant C that depends on f, t 0 and T .
Theorem B.2.
(Consistency criterion)
If f and ϕ are continuous then the one-step method is consistent if and only if ϕ ( t, x, 0) =
f ( t, x ) for all ( t, x ) .
Theorem B.3.
(Order criterion)
If f is C p and ϕ is C p in h then the one-step method is of order p if and only if ∂ kh ϕ
( t, x, 0)
=
1 k +1 f
[ k ] ( t, x ) for all ( t, x ) and 0 ≤ k < p where f [0]
= f and f
[ k ] = ∂
t f [ k − 1] + f∂ x f [ k − 1] .
Corollary B.4.
(Consistency and order of Euler’s method)
If f is continuous then Euler’s method is consistent.
If f is C 1 then Euler’s method has order 1.
Definition B.5.
(Zero-stability)
A one-step method is zero-stable (or stable ) if ∃ S > 0 such that for all ( x n ) 0 ≤ n ≤ N , (˜ x n ) 0 ≤ n ≤ N and ( ε n ) 0 ≤ n<N satisfying x n +1 − x n h n = ϕ ( t n , x n , h n ) and ˜ x n +1 − ˜ x n h n = ϕ ( t n , ˜ x n , h n ) + ϵ n (16) for 0 ≤ n < N , we have max n ∥ ˜ x n − x n ∥ ≤ S ( ∥ ˜ x 0 − x
∥
+ T max n | ϵ n | ) , where ϵ n = ε n /h n .
The constant S is the stability constant of the method.
Theorem B.6.
(Zero-stability criterion)
If ϕ is uniformly L -Lipschitz in its second variable, then the one-step method is stable with constant e LT .
Corollary B.7.
(Zero-stability of Euler’s method)
If f is Lipschitz in its second variable, then Euler’s method is stable.
Definition B.8.
(Convergence)
A numerical method converges if its global error max n ∥ x ( t n ) − x n ∥ goes to zero as h max goes to zero.
Theorem B.9.
(Convergence criterion)
If a method is consistent and stable with stability constant S , then it converges and max n ∥ x ( t n ) − x n ∥ ≤ ST max | e n | .
If the method is of order p with constant
C , then max n ∥ x ( t n ) − x n ∥ ≤ STCh p max
Corollary B.10.
(Convergence of Euler’s method) Euler’s method converges if f is C 0 and Lipschitz in x .
If f is also C 1 then it converges with speed O
( h max ) .
C P ROOF OF T HEOREM 4.
Proof.
Since T − 1 t ( ϕ xt ) = x , we have for any a 0 , b 0 ∈ X by the triangle inequality
[Equation]
where L = max m L m and L m is the Lipschitz constant of r m (which is Lipschitz being a composition of matrix multiplications and activations such as ReLU).
This is the first bound in the theorem.
In this bound, the term ∥ a m − ϕ a 0 t m ∥ (and likewise ∥ b m − ϕ b 0 t m ∥ ) represents the distance between the point a m we get after m residual blocks (i.e. after m Euler steps using the approximation r of v ) and the point ϕ a 0 t m we get by following the solution vector field v up to time t m .
By the properties of the Euler method (consistency and zero-stability, see Corollaries B.4, B.7 and B.10 in Appendix
B), under more regularity conditions on v , it is possible to bound this term.
Indeed, if v is C 1 and
M -Lipschitz in x (this is not stronger than the regularity we get on v through our regularization, because we still need to use (8)), we have for constants R, S > 0 ,
[Equation]
where ˜ a m comes from the Euler scheme with access to v (i.e. ˜ a m +1 :
=
˜ a m + hv t m (˜ a m ) and ˜ a 0 :
= a 0 ),
R is the consistency constant of the Euler method and S is its zero-stability constant.
Likewise, we get the same bound for ∥ b m − ϕ b 0 t m ∥ .
If a 0 , b 0 / ∈ X , we need to introduce ˆ a 0 :=
Proj X ( a o ) and ˆ b 0 :=
Proj X ( b o ) to apply (8).
We now get ∥
[Equation]
m
[Equation]
Bounding the terms ∥ a m − ϕ
ˆ a 0 t m ∥ and ∥ b m − ϕ
ˆ b 0 t m ∥ now gives
[Equation]
where ˜ a m now comes from the Euler scheme with access to v that starts at ˆ a 0 (meaning ˜ a m +1 :
=
˜ a m + hv t m (˜ a m ) and ˜ a 0 :
=
ˆ a 0 ).
Likewise, we get the same bound for ∥ b m − ϕ
ˆ b 0 t m ∥ .
Since ∥ a 0 − ˆ a 0 ∥ = dist ( a 0 , X ) and ∥ b 0 − ˆ b
∥ = dist ( b 0 , X ) , we get the second bound in the theorem.
Note that if we use the stability of the ODE instead of the Euler method to bound ∥ a m − ϕ ˆ a 0 t m ∥
we get the same result.
Indeed, if ˜ a m again comes from the Euler scheme with access to v that starts at a 0 (meaning ˜ a m +1 :
=
˜ a m + hv t m (˜ a m ) and ˜ a 0 :
= a 0 ), we can write, for some constant F >
[Equation]
since
[Equation]
where we get the last line by Gronwall’s lemma.
D A DDITIONAL EXPERIMENTS D. A DVERSARIAL ATTACKS
The Fast Gradient Method (FGM) (Goodfellow et al. (2015)) takes a perturbation step in the direction of the gradient that maximizes the loss. Projected Gradient Descent (PGD) (Madry et al. and the Basic Iterative Method (BIM) (Kurakin et al.
(2017)) are iterative versions of FGM.
Two slower but more powerful attacks are DeepFool (DF) (Moosavi-Dezfooli et al.
(2016)), which iteratively perturbs an input in the direction of the closest decision boundary, and Carlini-Wagner (CW) (Carlini & Wagner (2017a)), which solves an optimization problem to find the perturbation.
Black-box attacks don’t have any knowledge about the network.
We use two such attacks: Hop-Skip-Jump (HSJ) (Chen et al.
(2020)), which estimates the gradient direction at the decision boundary, and the Boundary
Attack (BA) (Brendel et al.
(2018)), which starts from a large adversarial input and moves towards the boundary decision to minimize the perturbation.
D. A DVERSARIAL DETECTION TRAINING PROTOCOL
D. I MPLEMENTATION DETAILS
For ResNeXt50 (Xie et al.
on CIFAR100 (Krizhevsky (2009)), we train for 300 epochs using
SGD with a learning rate of 0 .
1 (divided by ten at epochs 150, 225 and 250)
For ResNet110 (He et al.
(2016a))
on CIFAR10 (Krizhevsky (2009)), we train for 300 epochs using
SGD with a learning rate of 0 .
1 (divided by ten at epochs 150, 225 and 250), orthogonal initialization with a gain of 0.05, a batch size of 256, weight decay of 0 . 0001 and gradient clipping at 5.
For WideResNet (Zagoruyko & Komodakis (2016)) on TinyImageNet, we train for 300 epochs using
SGD with a learning rate of 0 .
1 (divided by ten at epochs 150, 225 and 250), orthogonal initialization with a gain of 0 . 1 , a batch size of 114 and weight decay of 0 . 0001 .
For the magnitude parameter of the Mahalanobis detector, we try all the values tried in their paper for the magnitude
and we report the best results
D. P RELIMINARY EXPERIMENTS
We see in Figure 4 below that training deep ResNets without batch-normalization is near impossible, whereas LAP-ResNets maintain the same performance and stability without ResNets for up toblocks.
LAP-ResNets are also compared in this regard to the small step method of Zhang et al.
(2019), which simply adds a small weight h of around 0 .
1 in front of the residue function to make ResNets more stable.
The Least Action Principle has the same improved stability when training without batch-normalization in Figure 4 as this method, while also improving the test accuracy (Karkar et al.
(2020)) which the small step method does not claim.
D. D ETECTION OF SEEN ATTACKS
In the tables below, VAN corresponds to detectors trained on a vanilla network and LAP to detectors trained on a transport-regularized network.
‘Maha’ refers to the Mahalanobis detector and ‘Tra’ or ‘Transport’ refers to our detector.
[Table]
[Table] [Table]
D. D ETECTION OF UNSEEN ATTACKS
[Table]
[Table]
[Table]
D.
[Table]
T IME COMPARISON
With a ResNeXt50 on CIFAR100 and a Tesla V100 GPU
[Table]
it takes our method (including the time to generate FGM attacks) 66 seconds to extract its features from both the clean and the adversarial samples, while it takes the Mahalanobis method 110 seconds. Mahalanobis also extracts some statistics from the training set prior to adversarial training, which takes an additional 35 seconds. Our feature vector is of size 32, compared to 5 for the Mahalanobis detector. So our random forest takes only 4 more seconds to train than the Mahalanobis one (7 vs 3 seconds).