Rethinking and Improving Robustness of Convolutional Neural Networks: a Shapley Value-based Approach in Frequency Domain
Abstract
The existence of adversarial examples poses concerns for the robustness of convolutional neural networks (CNN), for which a popular hypothesis is about the frequency bias phenomenon: CNNs rely more on high-frequency components (HFC) for classification than humans, which causes the brittleness of CNNs.
However, most previous works manually select and roughly divide the image frequency spectrum and conduct qualitative analysis.
In this work, we introduce
Shapley value, a metric of cooperative game theory, into the frequency domain and propose to quantify the positive (negative) impact of every frequency component of data on CNNs.
Based on the Shapley value, we quantify the impact in a fine-grained way and show intriguing instance disparity.
Statistically, we investigate adversarial training(AT) and the adversarial attack in the frequency domain.
The observations motivate us to perform an in-depth analysis and lead to multiple novel hypotheses about i) the cause of adversarial robustness of the
AT model; ii) the fairness problem of AT between different classes in the same dataset; iii) the attack bias on different frequency components.
Finally, we propose a Shapley-value guided data augmentation technique for improving the robustness.
Experimental results on image classification benchmarks show its effectiveness.
Introduction
Though convolutional neural networks (CNNs) have shown their great generalization power in various vision tasks, the existence of adversarial examples [37, 16, 24] show that CNNs are prone to be affected by adversarial noises that are imperceptible to humans.
As inspecting the impact of frequency components, i.e. signals of a certain frequency in the frequency domain, one popular hypothesis about the brittleness of CNNs is that CNNs can exploit high frequency components (HFCs) that are not perceivable to humans [41, 42, 1, 21].
Based on this hypothesis, a line of works propose to utilize
HFC filtering to improve the attack or defense methods [13, 22, 51].
However, towards the above frequency bias phenomenon, most works [41, 42, 21] intuitively divide the image frequency spectrum into two parts as low frequency components (LFCs) and HFCs then evaluate the contribution of each part based on the performance of models on the dataset, overlooking the possible disparity in sample-level and frequent component-level.
Moreover, current works [17, 32] reveal that some successful adversarial attacks also depend on LFCs, which reveals the limitation of previous works: i) overlooking the contribution of each individual frequency component in LFCs and HFCs; ii) only do qualitative dataset-level analysis.
Instead of studying the impact of HFCs and LFCs at the overall dataset level, in this paper we resort to quantify the contribution of each frequency component with the help of Shapley value [31].
Specifically, Shapley value is a metric from cooperative game theory which has been widely used in explainable machine learning [9, 2, 23, 36].
Since these works measure pixel-level importance for prediction in the spatial domain, our work is the first to apply Shapley value to the frequency domain.
Shapley value satisfies several desirable properties for quantifying the contribution of each frequency component, and we justify that applying Shapley value in the frequency domain rather than spatial domain is a more natural approach on CNNs (c.f. Remark 1).
Based on the Shapley value, we quantify the contribution of each frequency component by measuring the average expected difference between the model’s output brought by the absence of the frequency component regarding all possible circumstances where other frequency components are either masked or not.
An illustration of the quantifying method is presented in Fig.
Through our quantification, one is able to inspect the contribution of each frequency components of each sample (instead of at dataset level).
For simplicity, we refer to the frequency components with positive contribution as positive frequency components (PFC) and frequency components with negative contribution as negative frequency components (NFC).
Our quantification results provide insights in various perspectives.
At the instance level, we present intriguing disparity between different data samples, e.g. model may rely more on HFCs to inference on some instances, as shown in the first row of Fig. 2, or rely more on LFCs to inference on the other instances, as shown in the second row of Fig.
Statistically, we inspect and analyze the adversarial training including the cause of the adversarial robustness achieved by adversarial training and the fairness problem in adversarial training (AT), which quantitatively proves that HFCs plays an important role in adversarial training.
We further inspect and analyze the adversarial attack.
We present the attack bias on different frequency components and provide a possible explanation for adversarial attacks focusing on LFCs [17, 32].
Based on our quantification, we propose a simple yet principled data augmentation method named
Class-wise Shapley value-guided Augmentation (CSA) to correct the misalignment of features between different classes.
By augmenting the data sample with features composed by NFCs of other data samples from the same class, the method boosts the adversarial robustness efficiently.
Our major contributions are as follows
: • Instead of inspecting CNNs based on the manually divided frequency spectrum as done in existing literature [41], we introduce the Shapley value metric to the frequency domain and directly quantify the contribution of each individual frequency component on CNNs, with the advantage of desirable theoretical properties from the Shapely value itself (Section 3).
To our knowledge, this is the first work to apply the Shapley value to the frequency domain on DNNs (specifically CNNs in this paper).
• As inspecting the impact of frequency components on models, we address the possible cause of adversarial robustness achieved by adversarial training and demonstrate that AT models improve robustness by sacrificing the utilization of HFCs(Sec. 4.2).
We further address the fairness problem in adversarial training and demonstrate the negative relationship between the impact HFCs have on the ST model and the robust accuracy of the AT model(Sec. 4.3).
• We inspect the impact of different frequency components on adversarial attacks and demonstrate that adversarial attacks on PFCs and HFCs are relatively more effective than attacks on NFCs and LFCs.
We further demonstrate that contribution of frequency components is not necessarily determined by its frequency and provide a possible explanation for successful adversarial attacks focusing on LFCs [17, 32](Sec. 4.4).
• Motivated by our explanatory results, we assume that NFCs compose features that are not correctly exploited by the model.
Therefore we propose to augment data with NFCs found on data samples of its class, entitled by Class-wise Shapley value-guided Augmentation (CSA), which successfully boosts robustness in our empirical studies (Section 5).
Notations and Preliminaries
We present our model in the context of image classification, while we think the concept can still be applied to other backbones e.g. GNNs.
Notation: ⟨ X , y ⟩ denotes a data sample where X ∈ R d 1 × d 2 is an image and y ∈ { 1 , . .
, C } is its label where C is the number of classes.
For simplicity, we omit the channel of images.
Frequency Domain f( ∙ ) :
Model; : Unmasked Frequency; : Masked Frequency; : Target Frequency
Shapley Value of f( )- f( ), f( )-f( ), ⋮ f( )-f( ),
Average
Discrete Fourier Transform (DFT).
DFT transforms finite signals into complex-valued functions of frequency.
The 2D-Discrete (for image data considered in this paper)
Fourier Transform is given by:
[Equation]
The inversion of DFT is computed as:
[Equation]
Adversarial Attack.
It aims to find a small noise δ bounded in ℓ p space to cause failure [37, 16].
[Equation]
Defense.
Adversarial training [24] becomes the most effective defense method [3], which is typically conducted by minimizing loss directly on adversarial training data.
[Equation]
Definition of the Shapley Value: Let N = { 1 , · · · , n } denotes the finite set of players.
Each non-empty subset
S ⊆ N is called a coalition.
A cooperative game is defined by the pair ( N , V ) , where V : 2 N → R is a mapping that assigns a real number to each coalition and satisfies V ( ∅ )
= 0 .
Let π ∈ Π( N ) denotes a permutation of player set N and π ( i ) is the position of player i ∈ N , where Π(
N ) is the set of permutations.
Then the predecessor set of player i ∈ N in permutation π is P
πi = { j ∈ N| π
( j ) <
π ( i ) } .
The Shapley value ϕ
Vi of player i is defined as:
[Equation]
The Shapley value [31] of the player is the average marginal contribution of the player to the value of predecessor set in every possible permutation.
Properties of Shapley Value:
It satisfies many desirable properties including Null player , Efficiency ,
Symmetry , Linearity .
See Appendix D for more details about the properties of Shapley value.
[Figure]
[Figure]
Shapley Value-based Quantification 3.1 Contribution quantification for frequency component
We assume that frequency components in the data sample cooperate with each other and compose the feature that the model process to infer.
To apply the Shapley value to the frequency components, we firstly construct a cooperative game in this scenario.
Considering frequency components as players, we define the set of players for data sample ⟨ X , y ⟩ as:
[Equation]
Note that F ( X )(0 , 0) determines the mean value of the image, we exclude this frequency component from the player set and keep the mean value of the image unchanged to fairly calculate the marginal contribution of the frequency components.
For a model f ( · ) , we define the output of the model on S ⊆ N as:
[Equation]
(7) where f ( · ) y denotes the y -th output of the model and ⊙ denotes Hadamard product.
σ S ∈ [0 , 1] d 1 × dis the mask generated according to S satisfying:
[Equation]
(8) where I ( · ) denotes an indicator that output 1 when the condition is satisfied and 0 otherwise.
Specifically, σ S (0 , 0)
= 1 .
The corresponding characteristic function of the game is defined as:
[Equation]
By definition, we calculate the Shapley value ϕ N X ,f u,v of frequency component F ( X )( u, v ) according to Eq. 5.
The Shapley value of a frequency component evaluates the average marginal contribution of the frequency component to the neural network’s output of the ground-truth class, which takes desirable properties including Null player , Efficiency , Symmetry and Linearity .
3.2 Justification for frequency based Shapley value on CNNs
When it comes to measuring the contribution of input features in images, the common approach is decomposing features into pixels in the spatial domain and considering each pixel as a player [9, 2, 23, 36].
Missing players that are not in a coalition S are replaced with a reference value.
The reference value are often chosen intuitively as zero or the mean value of the image.
The choice of reference value may affect the explanatory results on CNNs.
[Figure]
l
[Figure]
y v
[Figure]
Remark.
1 (Decomposition of convolution)
The output of a convolution operation f conv ( · ) on X can be decomposed into a linear function of output on frequency components on X .
Formally,
[Equation]
10) where I u,vd 1 × d 2 ∈ R
d
1 × d 2 contains zeros except the element on the u -th row and v -th column is one.
According to Remark 1, the output of a convolution operation is a linear function w.r.t.
each frequency component.
Hence the reference value for frequency components is naturally zero.
Furthermore, the Shapley value intuitively aligns more with what is important in an additive setting, as mentioned in previous works [20].
For non-additive models, Shapley value may fail to explain the importance of players.
For example, function f ( x ) =
x 1 x 2 x 3 with reference value set to be zero.
The Shapley value of each variable is 13 f ( x ) while the value of each variable may vary.
For frequency components, the CNN is approximately a linear function.
See Appendix D for more details.
Rethinking Robustness in Frequency Domain
In this section, we conduct extensive experiments to give a fundamental analysis of the contributions of different frequency components.
In Sec. 4.1, we introduce details and results of instance-level sampling for Shapley value calculation.
In Sec. 4.2, we inspect the contribution of frequency components of clean (adversarial) data with standard (adversarial) training paradigms respectively, followed by an in-depth analysis of the way how adversarial attacks work in Sec. 4.4.
We demonstrate the discussion of fairness in Sec. 4.
4.1 The Shapley Value of Frequency Components and The Instance Disparity
To get the Shapley value of each frequency component, we employ Monte Carlo Permutation
Sampling to approximate the Shapley value [31].
We sample the Shapley value on CIFAR10,
CIFAR100, Tiny ImageNet and ImageNet.
Quantification result We demonstrate our quantification result on ImageNet in Fig 2.
Each row contains the results of a data sample.
We show the original image, the reconstructed image from positive frequency components (PFCs), the reconstructed image from negative frequency components (NFCs), and the heatmap of Shapley value from left to right.
The Shapley value of each frequency component varies between different data samples.
As shown in Fig.
2, the NFCs in the first image are mainly of high frequency or medium frequency while the
PFCs are mostly low frequency.
As a result, the NFCs compose the contour of the goldfish and other textures while PFCs recover the main body of the fish just like how humans perceive it.
However, the “shark” image in the second row shows the opposite direction to “goldfish”: PFCs mainly contains the contour of the shark, i.e., the HFC information, which means the frequency bias varies at the instance-level.
See Appendix A for more details and more results.
[Table]
a v e r a g
e
s h a p (i) Class 8 (Ship)
1 2 3 4 5 6 7 8 910111213141516 frequency 0.
a v e r a g
e
s h a p (j) Class 9 (Truck)
4.2 The Cause of the Adversarial Robustness of AT model
Observations in Sec.
4.1 are based on clean examples.
We further inspect the difference between ST and AT model in the frequency domain based on our Shapley value metric.
For CIFAR10, adversarial examples are generated with PGD-20 with a Linf bound of 8 \ 255 and step size of 2 \ 255 .
We present the experiment results of ResNet-18 on CIFAR10.
Refer to Appendix B for more experiment results of vgg16 on CIFAR10 and ResNet-18 on CIFAR100 and refer to Appendix A for more details of the model training.
4.2.1 Standard Trained Models Mainly Exploit LFC but Are More Vulnerable on HFC
As shown in Fig. 3(a), first, for clean examples, LFCs have significant positive Shapley value on average for standard trained (ST) model while the average Shapley value of HFCs are nearly zero, which means ST model barely exploits HFC; Second, for adversarial examples, the mean Shapley value of HFC are significantly negative, which shows that ST model is more vulnerable on HFC.
4.2.2 Adversarial Trained
Models Improve Robustness by Sacrificing Information
in HFCs
As shown in Fig. 3(b), LFCs have high positive Shapley value on both clean and adversarial examples and the average Shapley value of HFCs are nearly zero on both clean and adversarial examples, which may be the key to improved robustness of AT.
We hypothesize that AT tends to filter out the negative impact of HFC and the outcome is that AT focuses more on LFC, overlooking information in HFC.
As Fig. 3(a) shows, the HFC of clean images plays a remarkable positive impact.
Thus we conjecture that the trade-off between the clean accuracy and the robust accuracy of AT is possibly the result of whether the model utilizes the information in HFCs.
4.3 Fairness of AT in Frequency Domain
[Figure]
The class-wise disparity of robustness of AT model is a troublesome phenomenon even on the balanced dataset such as CIFAR-10 [43, 38] which results in fairness concerns.
Though the clean accuracy of the
ST model on each classes is similar, the robust accuracy of the AT model differs between classes.
For
CIFAR10, the class 0 , 1 , 8 , 9 that belong to the same coarse class “Transportation” exhibit relatively high accuracy and robustness while other classes related to “Animals” are less robust except the class 7 “horse”.
To investigate the difference between different classes, we demonstrate the average Shapley value of clean data samples from each class for ST model in Fig.
We generally find the trend that the absolute Shapley value of HFC is relatively larger on non-robust classes ( e.g. class 2 , 3 , 4 and 5 ) than robust classes ( e.g. class 1 , 7 , 8 and 9 ).
[Figure]
[Figure]
q u e n c
[Figure]
c
[Figure]
m p o n e n
[Figure]
PFC
NFC
We further calculate the average absolute Shapley value of HFC (we intuitively take components with frequency higher than 70% of the largest frequency) for ST model.
As shown in Fig.
there is a negative relationship between the adversarial accuracy of AT model and the average absolute Shapley value of HFC for ST model (the Pearson correlation coefficient is − 0 . 8765 ).
As shown in Sec. 4.
HFCs compose useful but vulnerable information while AT tends to sacrifice HFCs for robustness.
Therefore, we conjecture that: classes, where HFCs have a larger impact on the ST model would have relatively lower adversarial accuracy under AT.
4.4 Inspecting Adversarial Attack in Frequency Domain 4.4.1 Adversarial Attacks on PFCs and HFCs Are More Effective
In this section, we inspect the performance of adversarial attacks over different frequency components.
Specifically, for an adversarial noise δ generated by Eq.
3 on ST model f and data sample ⟨ X , y ⟩ , we transform the noise δ into frequency domain as F ( δ ) , which is further applied by masks based on different metrics to determine the impact of adversarial noise on different frequency components.
We first build masks based on the division between PFCs and NFCs of the data sample and the masked noise is defined as follows, where ⊙ is Hadamard product:
[Equation]
Similar to Eq.
11, we define adversarial noises δ on HFCs and LFC as
[Equation]
where r ( u, v ) is the frequency of the frequency component, r max is the maximum frequency in this data sample, ϵ low and ϵ high is the intuitively determined threshold for low frequency and high frequency.
In our experiment, ϵ low = 0 .
4 , ϵ high = 0 .
We test the error rate of ST model as the input data sample is attacked with various noises, as shown in Fig.
The error rate is nearly zero for origin data sample X and nearly one hundred percent for data samples attacked with the unmasked noises X + δ .
For adversarial noise on PFCs and NFCs,
X + δ PFC reaches a similar error rate of ST model as X + δ which is significantly larger than the error rate with X + δ NFC .
It indicates that the adversarial attack on PFCs is more effective.
Similar for HFCs and LFCs, the adversarial attack on HFCs is more effective than NFCs.
A Possible Explanation for Adversarial Attacks Focusing on LFCs
As demonstrated in Sec.
4.2 and Sec.
4.3, we quantitatively proved that models are more vulnerable on HFCs than on LFCs.
Previous works hypothesize that exploiting HFCs causes the brittleness of
[Table]
CNNs [41, 40, 45].
However, it conflicts with the success of the adversarial attacks that are limited on LFCs [17, 32] as addressed in [4, 25] and demands a closer look.
To explain such an intriguing conflict, we further calculate the ratio of PFCs and NFCs for each frequency over the frequency spectrum.
As shown in Fig.
7, both PFCs and NFCs that have a significant impact take up a fair ratio (at least 0 . 1 ) over the whole frequency spectrum.
Therefore, We argue that the contribution of frequency components on the prediction of CNNs are not necessarily related to its frequency .
The fact that PFCs and NFCs actually co-exist on every frequency basis provides a possible explanation for the success of the adversarial attacks that are limited on LFCs [17, 32].
Since the ratio of
PFCs and NFCs are even higher at low frequency, it is possible for adversarial attacks that focus on
LFCs to reduce the impact of PFCs and boost the impact of NFCs.
Improving Robustness via Class-wise Shapley Value-guided Augmentation
Motivation.
As defined in Sec. 3, frequency components with negative Shapley value make the model output lean to other classes rather than the ground-truth class, which implies that the distribution of image features composed by NFC is mistakenly aligned with that of other classes.
Intuitively, by proactively correcting the misalignment of features of the model, we could improve the adversarial robustness for future attacks.
Moreover, our observations in Sec.
4.3 reveal that there exists obvious disparity at the class level of distribution of NFC.
Therefore, we propose to actively augment data with features composed by NFC of the same class, named Class-wise Shapley value-guided Augmentation (CSA).
To be specific, based on a CNN f , we obtain the CSA p fX of data X by masking its PFC as follows, with the indicator function I [ · ] :
[Equation]
Implementation Since it takes a large number of samplings for obtaining a stable Shapley value (c.f.
Eq. 5) for each sample, it is necessary to downgrade the time complexity.
Fortunately, we find that for the same class, the feature distribution composed by NFC aligns well between samples, and depending on only a small ratio of whole data ( 3% in our experiment) brings out an obvious improvement.
Detailed analyses at a statistical level and experimental level can be found in Appendix B.
For each class c , we randomly take data samples ⟨ X 1 , c ⟩ , ⟨ X 2 , c ⟩ , · · · , ⟨ X n , c ⟩ and generate a set of
CSAs with a pretrained model ˆ f as: P ˆ fc = { p ˆ fX 1 , p ˆ fX 2 , · · · , p ˆ fX n } and train our model as follows:
[Equation]
(14) where the α is a coefficient of the CSA p ˆ f .
Experiment Results We conduct experiments on CIFAR10 and CIFAR100 [19] to verify the effectiveness of our CSA.
We train ResNet18 [18] on CIFAR10 and CIFAR100 under PGD-AT [24] [Table] [Table] [Table]
and TRADES [49].
Attacks are crafted within ℓ ∞ bound with ϵ = 8 / 255 .
We select PGDattack [24] and AutoAttack (AA) [12] as the attack baselines.
As Table 1 shows, under various attacks, our CSA consistently achieves better robustness over defense baselines with little cost of natural accuracy.
With the Shapley value calculated with ResNet-18, we further apply our CSA on
VGG16 [33] and WideResNet-28-10 [46] on CIFAR10.
As shown in Table 2, our CSA also greatly improves the adversarial robustness of VGG16 and WRN-28-10.
It indicates that our CSA method is model-agnostic and could be applied to various model architectures.
Note that the CSA set of each class contains the NFC of the first 150 images without shuffling for CIFAR10 and 30 images for
CIFAR100, i.e. , 3% of the training set in CIFAR10 and 6% of the training set in CIFAR100, which shows that obtaining CSA only takes little overhead and the feasibility of our method in realistic settings.
Training details and ablation study can be found in Appendix C.
Comparison between CSA and high frequency filtering defense method High frequency filtering methods have been proposed to defend adversarial attacks [13, 22, 51].
Most high frequency filtering methods are proposed to deal with l 2 norm adversarial attack and are compared with other nonadversarial training defend methods.
We take the suppressing high frequency method proposed in [51] as baseline, which masks the high frequency components and trains the model with TRADES, and test the robust accuracy of ResNet18 on CIFAR10 with ℓ ∞ attack where ϵ = 8 / 255 .
As demonstrated in Tab. 3, for l ∞ attack, suppressing high frequency components even slightly degrade the testing accuracy on CIFAR10.
Related Work
Adversarial Attack Since the seminal works [37, 16] have been proposed that reveal the vulnerability of DNNs, there is an increasing interest in studying stronger adversarial attacks, such as FGSM [16],
C&W attack [5], PGD attack [24], and AutoAttack [12].
The robustness of CNNs in this work has been thoroughly evaluated on these above attack baselines.
Adversarial Defense
The development between attack and defense is just like an arms race: the existence of adversarial examples promotes the development of defense methods, among which adversarial training (AT) has been considered the most effective one [3].
However, AT has its own limitations: i) the trade-off between accuracy and robustness [34, 39, 48, 44], ii) robust overfitting [30, 10], which brings a large robustness generalization gap.
Many works have been proposed to improve
AT by sample-wise importance re-weighting [50], adding more unlabeled data [27, 6] or regularizing the embedding space [29].
A recent work [28] proposes to augment data with the approximated confounders from a causal perspective for improved robustness while these confounders are class- agnostic and lack theoretical guarantee.
In this work, we quantify the frequency importance via the theoretically stable Shapley value and design a class-specific data augmentation strategy, which achieves great improvement in robustness with better robustness generalization.
Understanding Robustness in Frequency Domain One way to inspect the intriguing nature of adversarial attack and defense is through the lens of frequency spectrum.
Many works explore the sensitivity of CNNs and frequency properties of the model [40, 45, 1] and study whether convolution itself has an intrinsic bias in frequency domain [7].
A hypothesis is that CNNs exploit HFCs which leads to the lack of robustness [41].
Driven by this hypothesis, many pre-processing defend methods have been proposed [13, 22, 51].
Meanwhile, some works also come up with adversarial attacks focusing on LFC [17, 32] which leads to assumptions that constituent frequencies of adversarial examples are dependent on the dataset [4, 25].
Efforts to quantify and inspect the contribution of each frequency component have been made [42, 21].
We employ Shapley value, a tool that has preferable properties, to quantify the contribution of each frequency component on the fine-grained level.
Explainability with Shapley Value Shapley Value [31] is proposed to distribute contributions among players in a game.
It has been widely applied for the explainability of deep learning.
Some works propose its approximations [23, 9, 2].
While another line of works focus on the condition to apply
Shapley value [14, 15].
Also, the interaction between input features [36, 35, 47] is recently studied.
In our work, we apply Shapley value in the frequency domain to quantify the contribution of frequency components.
The quantification have many desirable theoretical properties.
Conclusion and Outlook
We have made efforts to quantify the contribution of each frequency component on CNNs.
We employ Shapley value, a method with theoretically desirable properties, to quantify the contribution of a single frequency component on the instance-level.
Through our quantification, we explore the difference between standard trained (ST) model and adversarial trained (AT) model.
Our analysis of the fairness in AT provides insights of the cause of robustness difference over classes.
We further examine the attack effect of adversarial noises on different frequency components.
Based on our findings, we then propose a simple yet effective class-wise data augmentation method CSA which augments data with the NFC found in this class.
Experimental results have shown its effectiveness.
Limitation & future work:
Our current results are limited mainly to CNNs while we believe it is also rewarding to develop new techniques to other backbone e g. Transformer and GNNs based on our framework, which we leave for immediate future work.
We hope that this work could inspire future explanations and defense algorithms of adversarial attacks.
Due to the nature of this work, there may not be any potential negative social impact that is easily predictable.
Acknowledgement
This work was partly supported by National Key Research and Development Program of China (2020AAA0107600), National Natural Science Foundation of China (61972250, 72061127003) and
Shanghai Municipal Science and Technology Project (2021SHZDZX0102, 22511105100)
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
In this Supplementary Material, we first present details of the Shapley value sampling (Appendix A).
Then we give more experimental results on CIFAR-100 and stability analysis of Shapley value (Appendix B).
We further show the training details and ablation study for the sample complexity of our CSA (Appendix C).
Finally, we add properties of the Shapley value and proof of decomposition of CNNs in frequency domain (Appendix D).
A Details of the Shapley Value Sampling
In this section, we introduce the details of the Shapley value sampling.
A.1 Details of the Model for the Shapley Value Sampling
We sample the Shapley value for models trained on CIFAR10, CIFAR100 and ImageNet.
For
CIFAR10 and CIFAR100, we employ ResNet-18 and train them ourselves.
The ST model on
CIFAR10 and CIFAR100 is trained for 100 epochs with SGD optimizer where learning rate is set to be 0 . 1 and decayed by 0 .
1 at 75 -th epoch and 90 -epoch.
The momentum is set to be 0 .
9 and weight decay is set to be 2 e − 4 .
The AT model on CIFAR10 and CIFAR100 is trained with adversarial examples generated with PGD-10 [24] with a L-inf bound where epsilon is set to be 8 / 255 and stepsize is set to be 2 / 255 .
The other settings of the training of AT model are the same of the ST model.
For ImageNet, we employ standard ResNet-50 provided in robustbench [11].
A.2 Details of the Sampling of the Shapley Value
We employ Monte-Carlo sampling method for the Shapley value [8] to sample the Shapley value.
For CIFAR10 and CIFAR100, we divide the image into 16 × 16 patches in the frequency domain where each patch contains 2 × 2 frequency components and sample 2000 times for each data sample.
For ImageNet, we divide the image into 32 × 32 patches in the frequency domain where each patch contains 7 × 7 frequency components and sample 1000 times for each data sample.
A.3 Analysis about the Error Bound of our Shapley Value Sampling
[Figure]
As discussed in previous work [26], once we know the variance of the marginal contributions we have an bound for the estimation error of the Shapley
Value.
For our estimated Shapley value ˆ ϕ , assume the probability of the estimation error being greater than ϵ is less than δ that is
[Equation]
With Chebyshev’s inequality, for marginal contributions with variance as σ 2 , the required sample times should be
[Equation]
≥
[Equation]
We calculate the average variance of the marginal contribution of each patch of the images in the first 200 images of each class of the train set of CIFAR10.
The average variance is 1 .
814 × 10 − 3 .
With δ = 0 . 05 and ϵ = 0 .
005 the required sample times is around 1450 .
Therefore, we set the sample times on
CIFAR10 and CIFAR100 as 2000 .
Limited by the calculation complexity, we set the sampling times for ImageNet at 1000 .
A.4 Shapley Value Quantification Results
We demonstrate more Shapley Value
Quantification Results on ImageNet and TinyImageNet in
Fig. 9 and Fig.
Each row contains results of a data sample.
We display the origin image, image reconstructed with PFCs, image reconstructed with NFCs and the heatmap of Shapley value from left to right.
Note that the for the heatmap of Shapley value, red represents positive and blue represents negative.
The closer to the center the lower the frequency.
To further demonstrate the instance disparity, we present the average Shapley value with the standard deviation between different instances in Fig.
8 which shows a large standard deviation.
B Extra Experiment ResultsB.1 Comparison between CSA and High Frequency Filtering Method
High frequency filtering methods have been proposed to defend adversarial attacks [13, 20, 45].
Most high frequency filtering methods are proposed to deal with l 2 norm adversarial attack and arecompared with other non-adversarial training defend methods [13, 20].
We take the suppressinghigh frequency method proposed in [45] as baseline and test the robust accuracy of ResNet18 on
CIFAR10.
As demonstrated in Tab. 5, for l ∞ attack, suppressing high frequency components even slightly 501 degrade the testing accuracy on CIFAR10.
B.2 Experiment Results of vgg16 on CIFAR10
In this section we present the experiment results of vgg16 on CIFAR10, the results are similar tothe results of ResNet-18.
The dimension of fully connected layers of vgg16 is changed from 4096to 512 in order to suit the scale of the image in CIFAR10.
Fig.
8 shows the similar differencebetween adversarial trained vgg16 and standard trained vgg16.
Fig.
9 demonstrate a similar negative
A.5 Time Complexity Analysis for The Sampling of the Shapley Value
The main cost of the sampling of the Shapley value on CNNs comes from generating masks and performing model inference.
Assume that frequency components are devided in to d × d patches, for the size d , the time complexity is O ( N 2 ) .
For sampling times, the time complexity is O ( N ) .
We report the time consumed for sampling Shapley value for ResNet-18 on CIFAR10 with Intel(R)
Core(TM) i7-7820X CPU @ 3.60GHz and one GeForce RTX 2080 Ti in Tab. 4.
sampling
[Table] [Table]
[Table]
relationship between the adversarial accuracy of AT model and the average absolute Shapley value of
HFC for ST model with vgg16.
Fig.
10 demonstrate the same attack bias for vgg16 as ResNet18.
[Figure]
B.3 Experiment Results on CIFAR100
In this section, we present the experiment results on CIFAR100.
Shapley value in frequency domain For frequency domain analysis, we conduct the same experiments on CIFAR100 as on CIFAR10.
As shown in Fig.
11, we present the average Shapley value
B Extra Experiment Results B.1 Experiment Results of vgg16 on CIFAR
In this section we present the experiment results of vgg16 on CIFAR10, the results are similar to the results of ResNet-18.
The dimension of fully connected layers of vgg16 is changed fromto 512 in order to suit the scale of the image in CIFAR10.
Fig.
11 shows the similar difference between adversarial trained vgg16 and standard trained vgg16.
Fig.
12 demonstrate a similar negative relationship between the adversarial accuracy of AT model and the average absolute Shapley value of
HFC for ST model with vgg16.
Fig.
13 demonstrate the same attack bias for vgg16 as ResNet18.
B.2 Experiment Results on CIFAR
In this section, we present the experiment results on CIFAR100.
Shapley value in frequency domain For frequency domain analysis, we conduct the same experiments on CIFAR100 as on CIFAR10.
As shown in Fig.
14, we present the average Shapley value of frequency components of the first 10 images of each class over frequency basis.
The results are similar to the results on CIFAR10 except that the absolute average Shapley value of adversarial examples on ST model are relatively smaller.
[Figure] [Figure]
[Figure] [Figure]
B.3 Stability analysis of Shapley Value
We show the variance of Average Shapley Value between different data samples to measure the statistical stability on Fig.
We plot the error bars of the average Shapley value of frequency components of clean samples on ST model and AT model on CIFAR10 and CIFAR100.
The variance is relatively big on LFCs and small on HFCs.
For HFCs, the average Shapley value and the variance on AT model is smaller than that on ST model and near zero which shows that AT model are less impacted by HFCs.
[Figure]
C.1 Training Details of CSA
As introduced in Eq. 13, we extract the CSA of images in the train set of CIFAR10 and CIFAR for ST model.
We train model in the same manner as the AT model trained in Appendix A. Models are trained for 100 epochs with SGD optimizer where learning rate is set to be 0 . 1 and decayed by 0 .
1 at 75 -th epoch and 90 -epoch.
The momentum is set to be 0 .
9 and weight decay is set to be 2 e − 4 .
For other data augmentations on train data, we employ random horizontal flip with the probability set to 0 .
The adversarial examples are generated with PGD-10 [24] with a L-inf bound where epsilon is set to be 8 / 255 and stepsize is set to be 2 / 255 .
For TRADES[49], the trade-off regularization parameter β is set to 6 .
The α in Eq.
14 linearly increase from 0 to 0 .
5 through the training process.
C.2 Ablation study for the size of our CSA
[Figure]
The number of CSA we used for data augmentation may affect the effectiveness of the data augmentation.
We generate CSA for the first 150 images of each class of CIFAR10 that is 3% of whole train set.
We train ResNet-18 as described in Appendix C.1 with different number of CSA and test the model with AA [12] and PGD-20 [24] at the last epoch.
The result is shown in Fig.
As the number of CSA for each class on CIFAR10 increasing, the test accuracy under AA and PGD-20 attack gradually increase.
At the same time the data augmentation with CSA is already effective with very few CSA i.
e
CSA per class which means the CSA we generate could represent features that are not correctly learned by the model.
D Justification for the Shapley Value on CNNs in Frequency domain D.1 Properties of the Shapley value
As proved in the seminal work [31], the Shapley value is the only solution that satisfies the following desirable properties:
• Null player : A player i is a null player if ∀ S ⊆ N , V ( S ∪ { i } )
=
V ( S ) .
The Shapley value of every null player is zero.
• Efficiency : Shapley value satisfies that (cid:80) i ∈N ϕ N ,V
i
[Equation]
• Symmetry :
Two players i, j ∈ N are symmetric if ∀ S ⊆
N\{ i, j } , V ( S ∪ { i } )
=
V ( S ∪ { j } ) .
The Shapley value satisfies that for all symmetric players i, j , ϕ N ,V i
[Equation]
• Linearity :
For two games ( N , U ) , ( N , V ) , if a third game is defined as ( N , W ) where ∀ S ⊆ N , W ( S ) =
U ( S )
+ V ( S ) .
It holds that ϕ N ,W i
[Equation]
D.2 The Decomposition of Convolution in Frequency Domain
As shown in Remark 1, the output of a convolution operation can be decomposed into sum of output of frequency components of X .
For a convolution kernel C ∈ R d 3 × d 4 , the element at i -th row and j -th column of the output of convolution function with the kernel f C is
[Equation]
We represent the d 1 × d 2 matrix where all the elements equal to zero except the element on u -th row and v -th column is one with I u,vd 1 × d 2 .
We have
[Equation]
Then we get
[Equation]
Therefore
[Equation]
we get
[Equation]