A FREQUENCY DOMAIN ANALYSIS OF GRADIENTBASED ADVERSARIAL EXAMPLES
Anonymous authors
Paper under double-blind review
A BSTRACT
It is well known that deep neural networks are vulnerable to adversarial examples
We attempt to understand adversarial examples from the perspective of frequency analysis .
Several works have empirically shown that the gradient-based adversarial attacks perform differently in the low-frequency and high-frequency part of the input data.
But there is still a lack of theoretical justiﬁcation of these phenomena.
In this work, we both theoretically and empirically show that the adversarial perturbations gradually increase the concentration in the low-frequency domain of the spectrum during the training process of the model parameters.
And the log-spectrum difference of the adversarial examples and clean image is more concentrated in the high-frequency part than the low-frequency part.
We also ﬁnd out that the ratio of the high-frequency and the low-frequency part in the adversarial perturbation is much larger than that in the corresponding natural image.
Inspired by these important theoretical ﬁndings, we apply low-pass ﬁlter to potential adversarial examples before feeding them to the model.
The results show that this preprocessing can signiﬁcantly improve the robustness of the model.
I NTRODUCTION
Recently, deep neural networks (DNN) have achieved great success in the ﬁeld of image processing, but it was found that DNNs are vulnerable to some synthetic data called adversarial examples ((Szegedy et al., 2013), (Kurakin et al., 2016)).
Adversarial examples are natural samples plus adversarial perturbations , and the perturbations between natural samples and adversarial examples are imperceptible to human but able to fool the model.
Typically, generating an adversarial example can be considered as ﬁnding an example in an (cid:15) -ball around a natural image that could be misclassiﬁed by the classiﬁer.
Recent studies designed Fast Gradient Sign Method (FGSM, (Goodfellow et al., 2014)), Fast Gradient Method (FGM, (Miyato et al., 2016)), Projected Gradient Descent (PGD, (Madry et al., 2017)) and other algorithms (Carlini & Wagner (2017), (Su et al., 2019), (Xiao et al., 2018), (Kurakin et al., 2016), (Chen et al., 2017)) to attack the model.
Since the phenomenon of adversarial examples was discovered, many related works have made progress to study why they exist.
Several works studied this phenomenon from the perspective of feature representation.
Ilyas et al.
(2019) divided the features into non-robust ones that are responsible for the model’s vulnerability to adversarial examples, and robust ones that are close to human perception.
Further, they showed that adversarial vulnerability arises from non-robust features that are useful for correct classiﬁcation.
Another way to characterize adversarial examples is to investigate them in the frequency domain via
Fourier transform.
Wang et al.
(2020a) divided an image into low-frequency component (LFC) and high-frequency component (HFC) and they empirically showed that human can only perceive LFC, but convolutional neural networks can obtain useful information from both LFC and HFC.
Yin et al.
(2019) ﬁlters out the input data with low-pass or high-pass ﬁlters to study the sensitivity of the additive noise with different frequencies.
Wang et al.
(2020b) claimed that existing adversarial attacks mainly concentrate in the high-frequency part.
Sharma et al.
(2019) found that when perturbations are constrained to the low-frequency subspace, they are generated faster and are more transferable, and will be effective to fool the defended models, but not for clean models.
All of these works showed that spectrum in frequency domain is an reasonable way to study the adversarial examples.
However, there is a lack of theoretical understanding about the dynamics of the adversarial perturbations in frequency domain along the training process of the model parameters.
In this work, we focus on the frequency domain of the adversarial perturbations to explore the spectral properties of the adversarial examples generated by FGM (Miyato et al., 2016) and PGD (Madry et al., 2017).
In this work, we give a theoretical analysis in frequency domain of natural images for the adversarial examples.
• For a two-layer neural network with non-linear activation function, we prove that adversarial perturbations by FGM and l 2 -PGD attacks gradually increase the concentration in the low-frequency part of the spectrum during the training process over model parameters.
• Meanwhile, the log-spectrum difference of the adversarial examples (the deﬁnition will be clariﬁed in Section 2.2) will be more concentrated in the high-frequency part than the low-frequency part. Furthermore, we show that the ratio of the high-frequency and the low-frequency part in the adversarial perturbation is much larger than that in the corresponding clean image.
Empirically • we design several experiments on the two-layer model and Resnet-32 with CIFAR-10 to verify the above ﬁndings.
• Based on these phenomena, we ﬁlter out the high-frequency part of the adversarial examples before feeding them to the model to improve the robustness.
Compared with the adversarially trained model with the same architecture, our method achieves comparable robustness with the similar computational cost of normal training and almost no loss of accuracy.
The rest of the paper is organized as follows:
In Section 2, we present preliminaries and some theoretical analysis on the calculation of the spectrum and log-spectrum .
Then we provide our main results about the frequency domain analysis of gradient-based adversarial examples in Section 3.
Furthermore, some experiments for supporting our theoretical ﬁndings are shown in Section 4.
Finally we conclude our work and conduct a discussion about the future work in Section 5.
All the details about the proof and experiments are shown in the Appendix.
B ACKGROUND 2. P RELIMINARIES
Notations We use { 0 , 1 , ..., d } to denote the set of all integers between 0 and d and use (cid:107) · (cid:107) p to denote l p norm.
Speciﬁcally, we denote by (cid:107) · (cid:107) the l 2 norm.
For a d -dimensional vector x , we use x µ to denote its µ -th component with index starting at 0.
For a scalar function f ( x ) : R d (cid:55)→ R , ∇ x f and ∂ µ f denote the gradient vector and its µ -th component.
We let sgn ( x ) = 1 for x > 0 and −for x < 0 .
Normal training refers to training on the original training data for learning the optimal weights for the neural networks.
˜ x
= F ( x ) denotes the Discrete Fourier Transform (DFT) of x .
Discrete Fourier Transform
The k -th frequency component ˜ g
[ k ] of one-dimensional DFT of a vector g is deﬁned by
[Equation]
where k ∈ {− d/ 2 , − d/ 2 + 1 , ..., d/ 2 } if d is even and k ∈ {− ( d − 1) / 2 , ..., ( d − 1) / 2 } if d is odd.
For convenience, we always consider an odd d and one can easily generalize the case to even dimensions.
For an integer cut-off ( cutoff frequency ) k r ∈ (0 , ( d − 1) / 2) , the Low Frequency
Components (LFC) and High Frequency Components (HFC) of ˜ g are denoted by ˜ g l [ k ] = (cid:26) ˜ g
[ k ] if − k c ≤ k ≤ k c , otherwise ˜
g
h [ k ] = (cid:26)if − k c ≤ k ≤ k c , ˜ g
[ k ] otherwise . Then the frequency space R d can be decomposed as S l (cid:76) S h where ˜ g l ∈ S l and ˜ g h ∈ S h .
Let N ˜
g
= (cid:113)(cid:80) d/ 2 k
= − d/ 2 | ˜ g
[ k ] | 2 , then the ratio of a frequency component with respect to the whole frequency spectrum is denoted by
[Equation]
and the ratio of LFC is denoted by
[Equation]
Setup In this paper, we consider a two-layer neural network f : R d (cid:55)→ R
[Equation]
where σ is an activation function, a = ( a 1 , a 2 , ..., a m ) T is an m -dimensional vector and w r ’s are d -dimensional weight vectors.
Van der Schaaf & van Hateren (1996) showed that natural images obey a power law in the frequency domain.
Therefore, to make our settings closer to widely studied image related tasks, we also assume our input x obey the power law
[Equation]
to imitate LFC-concentrated images, where the constant α ≥ 1 and | ˜ x [0] | = α 0 is another constant.
Besides, we choose the cut-off k c in such a way that
[Equation]
which indicates that τ (˜ x ∈ S l ) > τ (˜ x ∈ S h ) and can be easily satisﬁed for a small k c .
Gradient-based adversarial attack For a loss function (cid:96) , we ﬁnd l p -norm (cid:15) -bounded adversarial perturbations by solving the optimization problem
[Equation]
where ( x , y ) ∈ R d × R is d -dimensional input and output following the joint distribution D .
To solve the above optimization problem, Goodfellow et al.
(2014) proposed FGSM, an attack for an l ∞ -bounded adversary, and computed an adversarial example as x adv = x + (cid:15) sgn ( ∇ x (cid:96) ( θ, x , y )) .
Miyato et al.
(2016) proposed FGM with an l 2 -bounded adversary, x
adv = x + (cid:15) ∇ x (cid:96) ( x ,y )
(cid:107)∇ x (cid:96) ( x ,y ) (cid:107) 2 .
Madry et al.
(2017) introduced PGD which training adversarial examples in an iterative way that x t [-+
-]+1 =
Π x + S ( x t + α sgn ( ∇ x (cid:96) ( θ, x , y ))) or x t +1 = Π x
[Equation] S ( x [Equation] +
[Equation] ∇ [Equation]
To simplify the analysis, we make the following assumptions on the normal training process:
Assumption 1 There exist β (cid:48) , β > {+
+} such that β (cid:48) ≤ | ∂(cid:96)/∂f | ≤ β .
Assumption 2 If (cid:96) = 12 ( y − f ( x , θ )) 2 , | ∂(cid:96)/∂f | should be a small quantity such that | ∂(cid:96)/∂f | = {+(cid:15) 
 ++} v if the network is well trained in the end stage of normal training, where 0 < v < 1 .
Assumption 3 There exist λ, γ > {+
+} such that 0 ≤ σ (cid:48) ≤ λ and 0 ≤ σ (cid:48)(cid:48) ≤ γ 1 .
Assumption 4 a r ’s are i.i.d drawn
2. E XPLORING THE LOG - SPECTRUM AND SPECTRUM
Wang et al.
(2020a) considered the log-spectrum of an image: 20 ∗ log ( |F ( x ) | ) , where x is a 2 ddimensional matrix representing an image.
One way to calculate the ”log-spectrum difference of the adversarial examples” is to consider the difference between the log-spectrum of the adversarial and natural image and as follows,
[Equation]
We can observe that this quantity is approximately proportional to the ratio of the perturbation’s over the natural image’s frequency.
Moreover, the average Relative Change in discrete cosine Transforms (RCT) in (Wang et al., 2020b)) approximately equals to it.
Fig.
5 empirically shows that this ratio is large for high frequency component but small for low frequency one.
However, Van der Schaaf & van Hateren (1996) claimed that the LFC of F ( x ) is much bigger than its HFC, and the right side of Eq.
3 is inversely proportional to F ( x ) .
The denominator of the right side of Eq.(3) in low-frequency part is much larger than it in high frequency part, so it may be imprudent to consider the frequency distribution of the adversarial perturbations only based on the ratio in the right side of
Eq.(3).
Yin et al.
(2019) provided another spectral measurement for adversarial examples.
They considered the ”spectrum of the adversarial perturbation” |F
( δ ) | , which directly reﬂects the distribution of adversarial perturbations in the frequency domain.
Unfortunately, they claimed that ‘The adversarial perturbations for the normally trained model are uniformly distributed across frequency components’ without sufﬁcient empirical and theoretical justiﬁcation.
In this work, we provide a totally different theoretical ﬁnding that the adversarial perturbations for the normally trained model are gradually concentrated in the low-frequency domain, further veriﬁed by extensive experiments.
Fig.
1 visually shows the calculation methods of the spectrum of perturbations F ( δ ) and the logspectrum difference of adversarial examples | log( |F ( x + δ ) | ) − log( |F ( x ) | ≈ |F ( δ ) | |F ( x ) | .
M AIN R ESULTS
[Figure]
In Section 3.1, we show that, for a well trained two-layer neural network described in Eq.(1) with LFC-concentrated inputs, l 2norm adversarial perturbations generated by
FGM (Theorem 1) prefer to concentrate in the low frequency domain.
Section 3.2 theoretically proves the HFC-concentration of the logspectrum difference of adversarial examples.
Section 3.3 attempts to demonstrate the effectiveness of masking the HFC of the adversarial examples to improve model’s robustness despite that perturbations can concentrate in the low frequency domain.
All technical proofs are deferred to Appendix A. Results on l 2 PGD perturbations are presented in Appendix B.
In this work, we consider the two-layer neural networks and the loss function (cid:96) =
12 ( y − f ( x , θ )) 2 .
For any x in this setting, we have
[Equation]
where a (cid:48) r = a r σ (cid:48) ( w Tr x ) and a (cid:48) = ( a (cid:48) 1 , a (cid:48) 2 , ..., a (cid:48) m ) .
At the ( t + 1) -th step of gradient descent in the normal training process of the classiﬁer, weight w is updated as
[Equation]
We use L x and H x to denote the LFC and HFC of the clean input
[Equation]
3. S PECTRAL TRAJECTORIES OF l 2 - NORM FGM PERTURBATIONS
Consider the l 2 -norm FGM perturbation δ and its DFT ˜ δ [ k ]
[Equation]
where (cid:102) ∇ f [ k ] =
F ( ∇ x f )
[ k ] .
Since the coefﬁcient before (cid:102) ∇ f [ k ] in (6) is same for different frequencies k , we only need to analyze the spectrum of (cid:102) ∇ f [ k ] to investigate the ratio of certain frequency component over the whole frequency spectrum, i.e.
[Equation]
In this subsection, we explore the evolving of τ (˜ δ [ k ]) along the normal training process.
For a randomly initialized network, the FGM-attack perturbation will not have bias towards either HFC or
LFC.
We attack this model with FGM at each step of the normal training to study the trajectory of these perturbations in the frequency domain.
According to Eq.(4), ∇ x f is updated to the order of η at the ( t + 1) -th step as
[Equation]
m (cid:88) r
= (1 − η ( t ) r ) a (cid:48) ( t )
r w ( t ) r − ˜ η
( t ) x + O ( η 2 ) , where η ( t ) r = ηa r (cid:107) x (cid:107) 2 ∂(cid:96)∂f ( t ) σ (cid:48)(cid:48) ( t ) ( w ( t ) T r x ) and ˜ η ( t ) =
η ∂(cid:96)∂f ( t ) (cid:107) a (cid:48) ( t ) (cid:107) 2 are composite learning rates for easing our notation.
For ReLU activation function σ ( x ) = max (0 , x ) , we have that η ( t ) r =
E i ∈{ 1 , 2 ,...,m } [ η ( t ) i ] , therefore, considering that Softplus has similar shape as ReLU, we derive that η ( t ) r = ¯ η ( t ) + O ( η 2 ) , where ¯ η ( t ) =
η (cid:107) x (cid:107) 2 ∂(cid:96)∂f ( t ) E r ∈{ 1 , 2 ,...,m } (cid:104) a r σ (cid:48)(cid:48) ( t ) ( w ( t ) T r x ) (cid:105) .
In this way, the 1 d -DFT of ∇ x f ( t +1) has the following form,
[Equation]
We are now ready to study the trajectory of τ (˜ δ [ k ]) along the training step t .
Let (cid:52) ϕ ( t ) k denote the difference of phases between (cid:102) ∇ f ( t )
[ k ] and ˜ x [ k ] for frequency k at the t -th step.
And for ease of notation, denote
[Equation]
k [Equation] [Equation] where ˜ η ( t ) (cid:52) τ ( t ) l and ˜ η ( t ) (cid:52) τ ( t ) h are approximately changed amounts for LFC and HFC of | (cid:102) ∇ f ( t ) |at the t -th step of the normal training of the network, respectively.
Since the network is randomly initialized and the clean inputs x are highly concentrated in the low frequency domain, we always study the case where τ (˜ δ (0) ∈ S l ) <
τ (˜ x ∈ S l ) .
We now, taking Eq.(7) into consideration , present our main theorem about l 2 norm FGM perturbations as follows.
Theorem 1 (The spectral trajectory of l 2 FGM perturbation)
During the training process of the two-layer neural network f ( x ) in (1), the l 2 norm FGM adversarial perturbation will change its ratio of LFC, τ (˜ δ ∈ S l ) , at the ( t + 1) -th step as follows,
[Equation]
there will be a t 1 s.t.
∀ t ≥ t 1 , we have τ (˜ δ ( t +1)
∈ S l ) > τ ( ˜ δ ( t ) ∈
S l ) .
Remark
According to theorem 1, during the normal training process for a randomly initialized network with LFC-concentrated input, there will be a t 0 s.t.
∀ t ≥ t 0 , we have (cid:52) τ ( t ) l > (cid:52) τ ( t ) h , and τ (cid:16) ˜
δ ( t +1) ∈ S l (cid:17) > τ (cid:16) ˜ δ ( t ) ∈ S l (cid:17) if τ (cid:16) ˜ δ ( t ) ∈ S l (cid:17) < τ (cid:16) ˜ δ ( t )
∈ S h (cid:17) .
For ReLU activation function, starting with τ (˜ δ (0) ∈ S l ) = 1 / 2 − ζ and (cid:52)
τ (0) l = (cid:52) τ (0) h + bζ , we have t 0 = max (cid:110) 0 , − [Equation]
(cid:48)
[Equation] x
[Equation] x [Equation] (cid:111) , where n = min t ∈ [0 ,t 0 ] (cid:107) a (cid:48) ( t ) (cid:107) 2 .
Besides, given such a network trained with at least t 0 steps, there will be a t 1 > t 0 such that (cid:52) τ ( t 1 ) l τ (cid:16) ˜ δ ( t 1 )
∈ S h (cid:17) ≥ (cid:52) τ ( t 1 ) h τ (cid:16) ˜ δ ( t 1 ) ∈ S l (cid:17) and FGM perturbation will increase its ratio of LFC for all t > t 1 no matter whether it is smaller than 1 / 2 . Finally, if τ (cid:16) ˜ δ ( t 2 ) ∈ S l (cid:17) > τ (cid:16) ˜ δ ( t 2 ) ∈
S h (cid:17) for some t 2 > t 0 , this relation will hold for any t ≥ t 2 .
3. H IGH - FREQUENCY CONCENTRATION FOR LOG - SPECTRUM DIFFERENCE OF ADVERSARIAL EXAMPLES
In this subsection, we explore spectral trajectories of log-spectrum difference of l 2 FGM adversarial examples
[Equation]
along normal training step t as in Section 3.1 to study their HFC-concentration phenomenon.
Since | (cid:102) ∇ f (0)
[ k ] | for a randomly initialized network (1) has no bias towards high frequency or low frequency while | ˜ x | ∝ k − α , we adopt the following setting
[Equation]
without loss of generality and present below our result on the log-spectrum difference of the adversarial examples.
Theorem 2 (Concentration of the log spectrum difference)
For l 2 FGM perturbations of the network in (1), for any k > k (cid:48) > 0 which satisﬁes that R (0) k > R (0) k (cid:48) , there exists a t (cid:48) such that
R ( t ) k > R ( t ) k (cid:48) for all t < t (cid:48) steps of the normal training.
For ReLU activation function, we have
[Equation]
while if the initialization satisﬁes that R (0) k cos(
[Equation] then t (cid:48) = ∞ .
The theorem shows that although ˜ δ itself can concentrate in the low frequency domain, it does not concentrate as dense in the low frequency domain as the inputs ˜ x which obey the power law.
Therefore, the log-spectrum difference ( i.e. equivalent to spectrum of the ratio of the perturbation to the clean data) of adversarial examples will express a HFC-concentrated phenomenon instead (Fig.1).
3. M ASKING HFC OF THE ADVERSARIAL EXAMPLES TO IMPROVE ROBUSTNESS
In this subsection, we compare the ratio of LFC of the adversarial perturbations τ (˜ δ ∈ S l ) and the ratio of LFC of original images τ (˜ x ∈ S l ) .
Theorem 3 (Comparison on the LFC ratio of clean data and perturbations) For l 2 FGM perturbation of the two-layer neural network in (1), at the t -th step of normal training, let ζ = τ ( ˜ x ∈
S l ) − τ (˜ δ (0) ∈ S l ) > 0 , then we will have τ (˜ δ ( t ) ∈ S l ) < τ (˜ x ∈ S l ) for all t > 0 if the initilization of the netowrk satisﬁes that
[Equation]
Remark When both (cid:52) τ (0) l and (cid:52) τ (0) h are positive, condition (12) states that the ratio of low frequency component of (cid:52) τ (0) l + (cid:52) τ (0) h is smaller than that of clean data, which can be easily satisﬁed for a randomly initialized network.
The ratio of LFC of the perturbations δ will be less than that of clean data x if the condition Eq.(12) is satisﬁed for l 2 FGM perturbations during the normal training process.
Masking the HFC of an adversarial example x + δ will then erase more amount of the perturbation than that of the clean data which leads to a new ”not-perturbed-much” adversarial example.
As a result, one can expect the improvement for robustness of the model, as demonstrated in Section 4.3 later.
E XPERIMENT
In this section, we conduct several experiments to validate our theoretical conclusions.
Setup.
We use the Resnet-32 (He et al., 2016), a fully connected network as the last layer and cross entropy loss, and train all the parameters on the CIFAR-10 dataset with Adam optimizer (Kingma & Ba, 2014).
We use PGD-attack with (cid:15) = 8 / 255 , 40 iterations and step size ξ = 4 / 255 .
We supplement more experiments in Appendix C to ﬁll the gap between our theory and more complicated settings: we use the MSE loss and keep other conditions unchanged to resolve the discrepancy of loss function; experiments to support our theorems on two-layer neural network with ﬁxed outer-layer parameters are also provided.
4. T HE INCREASE OF LFC CONCENTRATION
In this part, we empirically verify the theorem in Section 3.1, to show that adversarial perturbations gradually concentrate more in the low-frequency part of the spectrum during the training process.
Consider a neural network with random initialization, each time before updating all the parameters of neural network, we randomly sample 100 images and apply the l 2 FGM / PGD attack to them.
For each iteration in PGD attack, we calculate their ratios of LFC and plot them in Fig.
It shows that the LFC ratios of the perturbations τ (˜ δ ∈ S l ) are gradually increasing along the normal training.
[Figure]
[Figure]
For a sufﬁciently trained neural network, we choose an image, use PGD-attack to obtain its adversarial examples and perturbations, and calculate its spectrum.
We randomly select 10 images in the test set and show the original images and the spectrum of perturbations in Fig.
We also randomly select 1000 images in the test set and average their spectrum to show the expectation of the spectrum in Fig. 4(a).
They show that the spectrum of adversarial perturbations are more concentrated in the low-frequency domain after a sufﬁcient training.
Ortiz-Jimenez et al.
(2020) show that the distance to the boundary (margin) in different frequency bands is heavily dependent of the distribution used for training, which also supports our claims.
4. HFC CONCENTRATION OF LOG - SPECTRUM DIFFERENCE
We ﬁrst use PGD-attack to get the adversarial example of a image for a sufﬁciently trained neural network.
We then make the log-spectrum of the original image and its adversarial example and calculate the difference of their log-spectrum (Fig.5).
Fig.4(b) displays the expectation of the spectrum for 1000 random images in the test set.
They both show that the log-spectrum difference of adversarial examples is generally concentrated around.
4. L OW - PASS FILTER IMPROVES ROBUSTNESS
In this part, we empirically show how a low-pass ﬁlter (which ﬁlters out HFC of the inputs) can improve the robustness of the model with a cheat sheet in the sense that the attacker does not know the existence of the ﬁlter while the model is aware of the frequency distribution of the gradient-based adversarial examples.
The procedure of our low-pass ﬁlter basically has three steps: performing DFT on the inputs, then masking their high frequency components (set as 0) in the frequency domain and ﬁnally performing inverse DFT on them to give the desired inputs with only low frequency components preserved.
We evaluate two types of accuracy.
One is ﬁltering the clean images out of high frequencies and feed them into the model to test accuracy, named as ” no-attack accuracy ”.
The other is ﬁrstly PGD-attacking the input images in the normally trained model, and then check the test accuracy of ﬁltered adversarial images, called “ PGD-attack accuracy ”. Here, we use data augmentation to train the model to the best.
We show the results in Fig.
As the cut-off increases, the no-attack accuracy gradually increases, and the PGD-attack accuracy ﬁrst increases and then decreases.
The highest robustness achieves 51.7% when the cut-off k = 12 and its robustness accuracy is 81.7%.
And the normal accuracy (no ﬁlter, no attack) is 93.03%.
We also adversarially train the model with PGD-attacks.
Its normal accuracy is 79% and PGD-attack accuracy is 48% without the ﬁltering.
Our low-pass ﬁlter model not only performs better on both no-attack accuracy and PGD-attack accuracy, but also does not need computationally expensive adversarial training. [Figure]
Theorem 3 provides a reasonable explanation for the above results.
When the cut-off is low, the ﬁlter masks most of the original image and the adversarial perturbation, so both of the no-attack and
PGD-attack models perform not well.
When the cut-off is proper, there remains enough information of original images but little of perturbations, so the PGD attacker cannot fool the model thoroughly.
Therefore, with a suitable cut-off, the low-pass ﬁlter can indeed improve the model’s robustness without computationally expensive adversarial training.
C ONCLUSION
We investigate the adversarial examples through the lens of frequency analysis.
Our work both theoretically and empirically clariﬁes the deﬁnition of log-spectrum difference of the adversarial examples in existing literature.
Besides, we devote to understanding the spectral trajectories of adversarial examples: l 2 FGM perturbations gradually increase the concentration in the low-frequency part of the spectrum but their ratios of LFC will never exceed that of clean data during the training process over model parameters.
Inspired by these ﬁndings, we ﬁnd that a low-pass ﬁlter can improve the robustness of the model and give a reasonable explanation for this phenomenon.
Future work can focus on analysis of other perturbations in the frequency domain such as FGSM and l ∞ PGD perturbations or providing theoretical understanding of the phenomenon that the ratio of the LFC of the perturbation for adversarially trained model is much higher than that for normally trained model (Appendix D) A P ROOFS A. P ROOF OF THEOREM According to (8), the update of τ (˜ δ [ k ]) for k -th frequency component of l 2 norm FGM adversarial perturbation at the ( t + 1) -th step is proportional to [Equation]
We adopt the following representations to see trends of cos (cid:16) (cid:52) ϕ
( t ) k (cid:17) through the training process:
[Equation]
then | (cid:102) ∇ f ( t +1)
[ k ] ||
˜ x
[ k ] | cos (cid:16) (cid:52) ϕ
( t ) k (cid:17) will have the form of (cid:10)
[Equation]
[Equation]
=
[Equation]
[Equation]
) at the ( t + 1) -th step.
At the ( t + 1) -th step, we have
[Equation]
[Equation]
which means that if (cid:52) τ ( t ) l > (cid:52) τ ( t ) h then we must have (cid:52)
τ ( t (cid:48) )
l > (cid:52) τ ( t (cid:48) )
h for all t (cid:48) ≥ t .
Besides, there must be a step which leads to this condition since L x > H x .
Let L ( t ) =
{+(cid:80) k c k =
+}
| (cid:102) ∇ f [ k ] | 2 and
H ( t ) =
(cid:80) ( d − 1) /
2 k = k c +1 | (cid:102) ∇ f
[ k ] | 2 .
At the ( t + 1) -th step, the changed amount of τ (˜ δ ∈ S l ) is
[Equation]
If t > t 0 and τ (cid:16) ˜ δ ( t )
∈ S h (cid:17) > τ (cid:16) ˜ δ ( t )
∈ S l (cid:17) then the above changed amount will be positive and gradient descent will increase τ (cid:16)
˜ δ ∈ S l (cid:17) at this step.
[Equation]
< (cid:52) τ ( t ) h τ (cid:16) ˜
δ ( t )
∈
S l (cid:17) , then gradient descent will increase τ (cid:16) ˜
δ ( t )
∈
S h (cid:17) untill the step t 1 at which (cid:52) τ ( t 1 ) l τ (cid:16) ˜
δ ( t 1 ) ∈
S h (cid:17) ≥ (cid:52) τ ( t 1 ) h τ (cid:16) ˜ δ
( t 1 ) ∈ S l (cid:17) , then for any t > t 1 , we have τ (cid:16) ˜
δ
( t +1)
∈ S l (cid:17) > τ (cid:16) ˜ δ ( t )
∈ S l (cid:17) .
A. P ROOF OF THEOREM
We provide the proof for ReLU activation function for positive ˜ eta , the case for negative ˜ η is similar.
The update rules of R k at the ( t + 1) -th step are
[Equation]
For any k > k (cid:48) > 0 with R (0) k > R (0) k (cid:48) , we have
R
[Equation]
)2 k
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
at the ( t + 1) -th step.
If R (0) k cos (cid:16) (cid:52) ϕ (0) k (cid:17) − R (0) k (cid:48) cos (cid:16) (cid:52) ϕ (0) k (cid:48) (cid:17) > 0 , considering the case ˜ η
( t (cid:48) )
=
˜ η max for any t (cid:48) and the condition that the left L.H.S of the above equation is larger than 0 gives us
Eq.(11).
On the other hand, if R (0) k cos (cid:16) (cid:52) ϕ (0) k (cid:17) − R (0) k (cid:48) cos (cid:16) (cid:52) ϕ (0) k (cid:48) (cid:17) < 0 , then R ( t ) k > R ( t ) k (cid:48) for all t ≥ 0 .
A. P ROOF OF THEOREM
We consider the case for l 2 FGM perturbations with ˜ η > 0 , the case for negative ˜ η is similar.
Let the initialization of the network always satisfy the condition
[Equation]
(16) such that τ (˜ x ∈ S l ) − τ (˜ δ (0) ∈ S l ) = ζ > 0 .
Then at the ( t ) -th step, we can express τ (˜ δ (0)
∈ S l ) as follows
[Equation]
Let a =
[Equation]
then
[Equation]
τ
[Equation]
0) h and
L ( t
[Equation]
H
[Equation]
If
[Equation]
then we must have
[Equation]
) at the t -th step.
A stronger condition is that
[Equation]
then we will have τ (˜ δ ( t )
∈ S l ) <
τ (˜ x ∈ S l ) for all t .
This condition approximately states that the ”ratio” of low frequency component of (cid:52) τ (0) l + (cid:52) τ (0) h is smaller than that of clean data.
B l 2 PGD PERTURBATIONS
The PGD update rule for ﬁnding perturbations of x with learning rate ξ at step j + 1 is:
[Equation]
where B (0 , (cid:15) ) is a ball centered at 0 with radius (cid:15) in Euclidean space and P is the projection operator deﬁned as
[Equation]
Note that in this part quantities removing the step script j ( e.g. ∂(cid:96)∂f and ∇ x f ) refers to not involving perturbations.
For convenience, we use the same learning rate ξ for all steps j and instead explore the update of
[Equation]
to the order of δ to see trends of PGD perturbations in frequency domain alongside PGD iterations since τ (˜ δ [ k ]) =
τ (˜ κ [ k ]) .
For ease of notation, we adopt the following representations: Let (cid:52) φ ( j ) k denote the difference of phases between (cid:102) ∇ f [ k ] and ˜ κ ( j )
[ k ] ; let ¯ β ( j ) =
∂(cid:96)/∂f + δ ( j ) · ∇ x f where ¯ β (0) > 0 and one can drive similar results for ¯ β (0) < 0 ; we denote (cid:52) ¯ τ ( j ) l (cid:44)k c (cid:88) k
= | ˜ κ ( j ) [ k ] || (cid:102) ∇ f [ k ] | cos (cid:16) (cid:52) φ ( j ) k (cid:17) and (cid:52) ¯ τ ( j
[Equation]
where ¯ β ( j ) (cid:52) ¯ τ ( j ) l and ¯ β ( j ) (cid:52) ¯ τ ( j ) h are changed amounts of LFC and HFC of | κ | 2 at the ( j + 1) - th step of PGD iteration.
We provide below our results on frequency spectrum of l 2 -norm PGD perturbations.
Theorem 4 (The spectral trajectory of l 2 PGD perturbation)
At the ( j + 1) -th step of PGD, iteration of PGD will change the ratio of LFC of l 2 norm PGD adversarial perturbation for a neural network (1) which satisﬁes | ∂(cid:96)/∂f | = (cid:15) 1+ ν with 0 < ν < 1 as follows,
[Equation]
Remark If the two-layer neural network in (1) is trained with at least t ≥ t 1 steps ( t 1 determined by theorem 1) such that τ ( (cid:102) ∇ f ∈ S l ) >
τ ( (cid:102) ∇ f ∈ S h ) , then, according to theorem 4, there exists
[Equation]
0 | (cid:102) ∇ f [ k ] | 2 − (cid:80) k =(
d − 1) / 2 k = k c | (cid:102) ∇ f [ k ] | 2 (cid:17)   , (21) where ˜ β
= max i ∈ [0 ,j 0 ] ¯ β ( i ) ,
such that τ (˜ δ
( j +1) ∈ S l ) > τ (˜ δ ( j ) ∈ S l ) for all j > j 0 if τ (˜ δ ( j ) ∈
S l ) < τ (˜ δ ( j ) ∈ S h ) .
B. P ROOF OF T HEOREM
At the (j + 1)-th step of PGD update for κ ( j +1) , if
[Equation]
[Equation]
[Equation]
In either case
[Equation]
the quantity τ (˜ κ [ k ]) is proportional to
[Equation]
since the term δ ∂(cid:96)/∂f <
δ 2 and can be dropped.
Therefore
[Equation]
we now consider
[Equation]
(22) at the ( j + 1) -th step of PGD in the frequency domain to explore ratio of frequency k to the whole frequency spectrum of perturbations
[Equation]
Lemma 1 (Dynamics of l 2 Norm PGD in the Frequency Domain)
If the initialization of l 2 norm
PGD adversarial perturbation satisﬁes ∂(cid:96)∂f + δ (0) ·∇ x f > 0 , then it will be positive at every iteration of PGD 3 .
Similar to Eq.(15), one can derive the changed amount of τ (˜ δ ∈ S l ) in theorem 4 at the ( j + 1)
-th step and ﬁnd the necessary condition.
C S UPPLEMENTARY EXPERIMENT
In this part, we present experiments on models with MSE loss and two-layer neural networks with ﬁxed outer-layer parameters in Section 4.
C. MSE LOSS
We show the supplementary experiment of Section 4.1 in Fig.
7 and Fig.
They have similar results to show that the spectrum of adversarial perturbations are more concentrated in the low-frequency domain after a sufﬁcient training.
Then we show the supplementary experiment of Section 4.2 in Fig.
9 and Fig.
They have similar results to show the log-spectrum difference of adversarial examples is generally concentrated around.
In general, there is little empirical gap between MSE loss and CrossEntropy loss.
C. T WO - LAYER NEURAL NETWORK
We use two-layer neural networks with ﬁxed outer-layer parameters.
The input layer is of size 3 ∗ 32 ∗ 32 followed by a ReLU activation function, and the hidden layer is of size 500 followed by a sigmoid activation function.
The dataset is still CIFAR10.
We use random initialization and SGD optimizer without any training tricks.
This setting can be used in practice and get higher than 40% test accuracy.
The attack method is the same as the setup in Section 4.
We show the LFC ratio of the adversarial perturbations during the training process in Fig.11.
After sufﬁcient training, we show the expectation of the spectrum in Fig.12(a) and the expectation of the log-spectrum difference in Fig.12(b).
[Figure]
[Figure]
D PERTURBATIONS OF PGD- ATTACK ADVERSARIAL TRAINED MODEL ARE MORE CONCENTRATED IN LFC
We do use the same setup in Section 4, and the PGD-attack adversarial training is also PGD-attack with (cid:15) = 8 / 255 , 40 iterations and step size ξ = 4 / 255 in each training step.
Then we test the ratio of LFC of original images, perturbations in a normally trained model and perturbations in a PGDattack adversarially trained model.
The results are shown in Fig.
It shows that the LFC ratio of perturbations in PGD-attack adversarially trained model is generally higher than that in normally trained model.
[Figure]