Rethinking and Improving Robustness of Convolutional Neural Networks: a Shapley Value-based Approach in Frequency Domain
Abstract
The existence of adversarial examples poses concerns for the robustness ofconvolutional neural networks (CNN), for which a popular hypothesis is aboutthe frequency bias phenomenon: CNNs rely more on high-frequency components(HFC) for classification than humans.
Most works manually select and roughlydivide the image frequency spectrum for their analysis.
In this work, we introduce
Shapley value, a metric of cooperative game theory, into the frequency domain andpropose to quantify the positive (negative) impact of every frequency componentof data on CNNs.
Based on the Shapley value, we show that the importanceproportion between low frequency components (LFC) and HFC varies at aninstance-level.
We further investigate the distinctive behavior of standard trained(ST) and adversarial trained (AT) models on clean and adversarial examples atthe dataset-level.
The observations motivate us to perform an in-depth analysis andlead to multiple novel hypotheses about i) the difference between ST model and
AT model; ii) the generalization limitation of AT including the fairness problem;iii)
the attack bias at different frequency component level.
Finally, we proposea Shapley-value guided data augmentation technique for improving the robustness.
Experimental results on image classification benchmarks show its effectiveness.
Introduction
Though convolutional neural networks (CNNs) have shown their great generalization power in variousvision tasks, the existence of adversarial examples [32, 16, 22] show that CNNs are prone to beaffected by adversarial noises that are imperceptible to humans.
As inspecting the impact of frequencycomponents, i e. signals of a certain frequency in the frequency domain, one popular hypothesisabout the brittleness of CNNs is that CNNs can exploit high frequency components (HFCs) that arenot perceivable to humans [36, 37, 1, 19].
Based on this hypothesis, a line of works propose to utilize
HFC filtering to improve the attack or defense methods [13, 20, 45].
However, towards the above 25 frequency bias phenomenon, most works [36, 37, 19] intuitively divide the image frequency spectruminto two parts as low frequency components (LFCs) and HFCs then evaluate the contribution ofeach part based on the performance of models on the dataset, overlooking the possible disparityin sample-level and frequent component-level.
Moreover, current works [17, 28] reveal that somesuccessful adversarial attacks also depend on LFCs, which reveals the limitation of previous works:i) overlooking the contribution of each individual frequency component in LFCs and HFCs; ii) onlydo qualitative dataset-level analysis.
Previous work [37] quantify and inspect the contribution of each frequency component by the changeof model prediction brought by masking frequency components i.e. replace the frequency componentwith zero value.
Following the spirit, in this paper we resort to a more sophisticated tool, Shapley value [27] to quantify the contribution of each frequency component.
Specifically, Shapley valueis a metric from cooperative game theory which has been widely used in explainable machinelearning [9, 2, 21, 31].
Since these works measure pixel-level importance for prediction in the spatialdomain, our work is the first to apply Shapley value to the frequency domain.
Shapley value satisfiesseveral desirable properties for quantifying the contribution of each frequency component, and wejustify that applying Shapley value in the frequency domain rather than spatial domain is a morenatural approach on CNNs (c.f. Remark 1).
Based on the Shapley value, we quantify the contribution of each frequency component by measuringthe average expected difference between the model’s output brought by the absence of the frequencycomponent regarding all possible circumstances where other frequency components are either maskedor not.
An illustration of the quantifying method is presented in Fig.
Through our quantification,one is able to inspect the contribution to the classification task of different frequency componentsof each sample (instead of dataset level).
For simplicity, we refer to the frequency componentswith positive contribution as positive frequency components (PFC) and frequency components withnegative contribution as negative frequency components (NFC).
We inspect the impact brought by different frequency components.
At the instance level, we presentintriguing disparity between different data samples, e.g. model may rely more on HFCs to inferenceon some instances, as shown in the first row of Fig.
2, or rely more on LFCs to inference onthe other instances, as shown in the second row of Fig.
At the dataset level, we inspect andanalyze: i) the cause of the adversarial robustness achieved by adversarial training; ii) the fairnessproblem in adversarial training (AT); iii) the attack bias on different frequency components.
Wefurther propose a simple yet principled data augmentation method named Class-wise Shapley valueguided Augmentation (CSA) to correct the misalignment of features between different classes.
Byaugmenting the data sample with features composed by NFCs of other data samples from the sameclass, the method boosts the adversarial robustness efficiently.
Our major contributions are as follows:
• Instead of inspecting CNNs based on the manually divided frequency spectrum as done in existingliterature [36], we introduce the Shapley value metric to the frequency domain and directly quantifythe contribution of each individual frequency component on CNNs, with the advantage of desirabletheoretical properties from the Shapely value itself (Section 3).
To our knowledge, this is the firstwork to apply the Shapley value to the frequency domain on DNNs (specifically CNNs in this paper).
• As inspecting the impact of frequency components on models, we address the possible cause ofadversarial robustness achieved by adversarial training(Sec.
4.2) and the fairness problem in theadversarial problem in adversarial training(Sec. 4.3).
We quantitatively prove that HFCs play animportant role in adversarial training.• We inspect the impact of different frequency components on adversarial attacks and demonstratethat adversarial attacks on PFCs and HFCs are relatively more effective than attacks on NFCsand LFCs.
We further demonstrate that contribution of frequency components is not necessarilydetermined by its frequency and provide a possible explanation for successful adversarial attacksfocusing on LFCs [17, 28](Sec.
4.4).• Motivated by our explanatory results, we assume that NFCs compose features that are not correctlyexploited by the model.
Therefore we propose to augment data with NFCs found on data samplesof its class, entitled by Class-wise Shapley value-guided Augmentation (CSA), which successfullyboosts robustness in our empirical studies (Section 5).
Notations and Preliminaries
We present our model in the context of image classification, while we think the concept can still beapplied to other backbones e.g. GNNs.
Notation: ⟨ X , y ⟩ denotes a data sample where X ∈ R d 1 × d 2 is an image and y ∈ { 1 , . .
, C }is its label where C is the number of classes.
For simplicity, we omit the channel of images.f ( · , θ ) :
R d 1 × d 2 → R C denotes a neural network that takes image as input and output a predictionwhere θ represent its parameters.
We use ℓ ( · , · ) to denote loss function and ∥ · ∥ p to denote ℓ pnorm.
F ( · ) : R d 1 × d 2 → C d 1 × d 2 denotes the Discrete Fourier Transfom (DFT).
F − 1 ( · ) : C d 1 × d 2 → 86 R d× d 2 denotes the inverse Discrete Fourier Transform.
Frequency Domain f
[Equation]
Model; : Unmasked Frequency; : Masked Frequency; : Target Frequency
Shapley Value of f
[Equation]
Average
Discrete Fourier Transform (DFT).
DFT transforms finite signals into complex-valued functions offrequency.
The 2D-Discrete (for image data considered in this paper)
Fourier Transform is given by:
[Equation]
The inversion of DFT is computed as:
[Equation]
Adversarial Attack.
It aims to find a small noise δ bounded in ℓ p space to cause failure [32, 16].
[Equation]
Defense.
Adversarial training [22] becomes the most effective defense method [3], which is typicallyconducted by minimizing loss directly on adversarial training data.
[Equation]
Definition of the Shapley Value: Let N = { 1 , · · · , n } denotes the finite set of players.
Each 94 non-empty subset
S ⊆ N is called a coalition.
A cooperative game is defined by the pair ( N , V ) , 95 where V :
N → R is a mapping that assigns a real number to each coalition and satisfies V ( ∅ ) = 0 .
Let π ∈ Π( N ) denotes a permutation of player set N and π ( i ) is the position of player i ∈ N , 97 where Π(
N ) is the set of permutations.
Then the predecessor set of player i ∈ N in permutation π is 98 P
πi = { j ∈ N| π
( j ) <
π ( i ) } .
The Shapley value ϕ
Vi of player i is defined as:
[Equation]
[Equation]
The Shapley value [27] of the player is the average marginal contribution of the player to the value ofpredecessor set in every possible permutation.
Properties of Shapley Value:
It satisfies many desirable properties including Null player , Efficiency , 102 Symmetry , Linearity .
See Appendix D for more details about the properties of Shapley value.
[Figure] [Figure]
[Figure]
Shapley Value-based Quantification
[Figure]
3.1 Contribution quantification for frequency component
[Figure]
We assume that frequency components in the data sample cooperate with each other and compose thefeature that the model process to infer.
To apply the Shapley value to the frequency components, wefirstly construct a cooperative game in this scenario.
Considering frequency components as players,we define the set of players for data sample ⟨ X , y ⟩ as:
[Equation]
Note that F ( X )(0 , 0) determines the mean value of the image, we exclude this frequency componentfrom the player set and keep the mean value of the image unchanged to fairly calculate the marginalcontribution of the frequency components.
For a model f ( · ) , we define the output of the model on S ⊆ N as:
[Equation]
(7) where f ( · ) y denotes the y -th output of the model and ⊙ denotes Hadamard product.
σ S ∈ [0 , 1] d 1 × d 2is the mask generated according to S satisfying:
[Equation]
(8) where I ( · ) denotes an indicator that output 1 when the condition is satisfied and 0 otherwise.
Specifically, σ S (0 , 0)
= 1 .
The corresponding characteristic function of the game is defined as:
[Equation]
By definition, we calculate the Shapley value ϕ N X ,f u,v of frequency component F ( X )( u, v ) accordingto Eq. 5.
The Shapley value of a frequency component evaluates the average marginal contributionof the frequency component to the neural network’s output of the ground-truth class, which takesdesirable properties including Null player , Efficiency , Symmetry and Linearity .
3.2 Justification for frequency based Shapley value on CNNs
When it comes to measuring the contribution of input features in images, the common approach isdecomposing features into pixels in the spatial domain and considering each pixel as a player [9,2, 21, 31].
Missing players that are not in a coalition S are replaced with a reference value.
Thereference value are often chosen intuitively as zero or the mean value of the image.
The choice ofreference value may affect the explanatory results on CNNs.
[Figure]
Remark.
1 (Decomposition of convolution)
The output of a convolution operation f conv ( · ) on X 128 can be decomposed into a linear function of output on frequency components on X .
Formally,
[Equation] [Equation]
where I u,vd 1 × d 2 ∈ R
d
1 × d 2 contains zeros except the element on the u -th row and v -th column is one.
According to Remark 1, the output of a convolution operation is a linear function w.r.t.
each frequencycomponent.
Hence the reference value for frequency components is naturally zero.
Furthermore, the Shapley value intuitively aligns more with what is important in an additive setting,as mentioned in previous works [18].
For non-additive models, Shapley value may fail to explain theimportance of players.
For example, function f ( x ) =
x 1 x 2 x 3 with reference value set to be zero.
The Shapley value of each variable is 13 f ( x ) while the value of each variable may vary.
For frequencycomponents, the CNN is approximately a linear function.
See Appendix D for more details.
Rethinking Robustness in Frequency Domain
In this section, we conduct extensive experiments to give a fundamental analysis of the contributionsof different frequency components.
In Sec. 4.1, we introduce details and results of instance-levelsampling for Shapley value calculation.
In Sec. 4.2, we inspect the contribution of frequencycomponents of clean (adversarial) data with standard (adversarial) training paradigms respectively,followed by an in-depth analysis of the way how adversarial attacks work in Sec. 4.4.
We demonstratethe discussion of fairness in Sec.
4.1 The Shapley Value of Frequency Components
To get the Shapley value of each frequency component, we employ Monte Carlo Permutation
Sampling to approximate the Shapley value [27].
We sample the Shapley value on CIFAR10,
CIFAR100, Tiny ImageNet and ImageNet.
Quantification result We demonstrate our quantification result on ImageNet in Fig 2.
Each rowcontains the results of a data sample.
We show the original image, the reconstructed image frompositive frequency components (PFCs), the reconstructed image from negative frequency components(NFCs), and the heatmap of Shapley value from left to right.
The Shapley value of each frequency component varies between different data samples.
As shownin Fig. 2, the NFCs in the first image are mainly of high frequency or medium frequency while the
PFCs are mostly low frequency.
As a result, the NFCs compose the contour of the goldfish and othertextures while PFCs recover the main body of the fish just like how humans perceive it.
However, the“shark” image in the second row shows the opposite direction to “goldfish”: PFCs mainly containsthe contour of the shark, i.e., the HFC information, which means the frequency bias varies at theinstance-level.
See Appendix A for more details and more results.
[Table]
a v e r a g
e
s h a p (i) Class 8 (Ship)
1 2 3 4 5 6 7 8 910111213141516 frequency 0.
a v e r a g
e
s h a p (j) Class 9 (Truck)
4.2 The Cause of the Adversarial Robustness of AT model
Observations in Sec.
4.1 are based on clean examples.
We further inspect the difference between STand AT model in the frequency domain based on our Shapley value metric.
For CIFAR10, adversarialexamples are generated with PGD-20 with a Linf bound of 8 \ 255 and step size of 2 \ 255 .
We presentthe experiment results of ResNet-18 on CIFAR10.
Refer to Appendix B for more experiment resultsof vgg16 on CIFAR10 and ResNet-18 on CIFAR100 and refer to Appendix A for more details of themodel training.
4.2.1 Standard Trained Models Mainly Exploit LFC but Are More Vulnerable on HFC
As shown in Fig. 3(a), first, for clean examples, LFCs have significant positive Shapley value onaverage for standard trained (ST) model while the average Shapley value of HFCs are nearly zero,which means ST model barely exploits HFC; Second, for adversarial examples, the mean Shapleyvalue of HFC are significantly negative, which shows that ST model is more vulnerable on HFC.
Adversarial Trained Models Improve Robustness as an HFC Filter
As shown in Fig. 3(b), LFCs have high positive Shapley value on both clean and adversarial examplesand the average Shapley value of HFCs are nearly zero on both clean and adversarial examples, whichmay be the key to improved robustness of AT.
We hypothesize that AT tends to filter out the negativeimpact of HFC and the outcome is that AT focuses more on LFC, overlooking information in HFC.
As Fig. 3(a) shows, the HFC of clean images plays a remarkable positive impact.
Thus we conjecturethat the trade-off between the clean accuracy and the robust accuracy of AT is possibly the result ofwhether the model utilizes the information in HFCs.
4.3 Fairness of AT in Frequency Domain
[Figure]
The class-wise disparity of robustness of AT modelis a troublesome phenomenon even on the balanceddataset such as CIFAR-10 [38, 33] which results infairness concerns.
For CIFAR10, the class 0 , 1 , 8 , 9that belong to the same coarse class “Transportation”exhibit relatively high accuracy and robustness whileother classes related to “Animals” are less robustexcept the class 7 “horse”.
To investigate the difference between differentclasses, we demonstrate the average Shapley valueof clean data samples from each class for ST modelin Fig.
Though the class-wise accuracy of ST aresimilar, we generally find the trend that the absolute
Shapley value of HFC is relatively larger on nonrobust classes ( e.g. class 2 , 3 , 4 and 5 ) than robustclasses ( e.g. class 1 , 7 , 8 and 9 ).
[Figure]
[Figure]
PFC
NFC
We further test the adversarial accuracy of AT model and calculate the average absolute Shapley valueof HFC (we intuitively take components with frequency higher than 70% of the largest frequency)for ST model.
As shown in Fig.
5, there is a negative relationship between the adversarial accuracyof AT model and the average absolute Shapley value of HFC for ST model (the Pearson correlationcoefficient is − 0 . 8765 ).
As AT tends to filter out HFCs for robustness, we conjecture that: classes, 202 where HFCs have a larger impact on the ST model, would have relatively lower adversarial accuracyunder AT.
4.4 Attack Bias in Frequency Domain
In this section, we inspect the performance of adversarial attacks over different frequency components.
Specifically, for an adversarial noise δ generated by Eq.
3 on ST model f and data sample ⟨ X , y ⟩ , wetransform the noise δ into frequency domain as F ( δ ) , which is further applied by masks based ondifferent metrics to determine the impact of adversarial noise on different frequency components.
4.4.1 Adversarial Attacks on PFCs and HFCs Are More Effective
We first build masks based on the division between PFCs and NFCs of the data sample and themasked noise is defined as follows, where ⊙ is Hadamard product:
[Equation]
Similar to Eq.
11, we define adversarial noises δ on HFCs and LFC as
[Equation]
12) where r ( u, v ) is the frequency of the frequency component, r max is the maximum frequency inthis data sample, ϵ low and ϵ high is the intuitively determined threshold for low frequency and highfrequency.
In our experiment, ϵ low = 0 .
4 , ϵ high = 0 .
We test the error rate of ST model as the input data sample is attacked with various noises, as shownin Fig.
The error rate is nearly zero for origin data sample X and nearly one hundred percent fordata samples attacked with the unmasked noises X + δ .
For adversarial noise on PFCs and NFCs,
X + δ PFC reaches a similar error rate of ST model as X + δ which is significantly larger than theerror rate with X + δ NFC .
It indicates that the adversarial attack on PFCs is more effective.
Similarfor HFCs and LFCs, the adversarial attack on HFCs is more effective than NFCs.
4.4.2 A Closer Look on the Relationship between HFC, LFC and PFC, NFC
As demonstrated in Fig. 3(a), the average Shapley value of LFC are significantly positive, it seemsthat most LFCs are PFCs.
However, results in Sec.
4.4.1 imply that the attack exhibits some bias in the frequency domain: the attack effect of adversarial noises on PFCs is better than the adversarialnoises on NFCs, while the adversarial noises on HFCs is better than that on LFCs, which seems tocontradict Fig. 3(a) and demands a closer look.
To explain such an intriguing conflict, we furthercalculate the ratio of PFCs and NFCs for each frequency over the frequency spectrum.
As shown in
Fig.
7, both PFCs and NFCs that have a significant impact take up a fair ratio (at least 0 . 1 ) of eachfrequency over the whole frequency spectrum.
Therefore, the existence of PFC in high frequency and
NFC in low frequency may lead to the result in Sec. 4.4.1.
The existence of PFCs and NFCs over the whole frequency spectrum also provides a possibleexplanation for the conflict between the hypothesis that exploiting HFCs causes the brittleness of
CNNs and the success of the adversarial attacks that are limited on LFCs [17, 28] which is addressedin [4, 23].
Since the ratio of PFCs and NFCs are even higher at low frequency, it is possible foradversarial attacks that focus on LFCs to reduce the impact of PFCs and boost the impact of NFCs.
We argue that the contribution of frequency components on the prediction of CNNs are not necessarilyrelated to its frequency.
Improving Robustness via Class-wise Shapley Value-guided Augmentation
Motivation.
As defined in Sec. 3, frequency components with negative Shapley value make the modeloutput lean to other classes rather than the ground-truth class, which implies that the distribution ofimage features composed by NFC is mistakenly aligned with that of other classes.
Intuitively, byproactively correcting the misalignment of features of the model, we could improve the adversarialrobustness for future attacks.
Moreover, our observations in Sec.
4.3 reveal that there exists obviousdisparity at the class level of distribution of NFC.
Therefore, we propose to actively augment data withfeatures composed by NFC of the same class, named Class-wise Shapley value-guided Augmentation(CSA).
To be specific, based on a CNN f , we obtain the CSA p fX of data X by masking its PFC asfollows, with the indicator function I [ · ] :
[Equation]
Implementation Since it takes a large number of samplings for obtaining a stable Shapley value (c.f.
Eq. 5) for each sample, it is necessary to downgrade the time complexity.
Fortunately, we find that forthe same class, the feature distribution composed by NFC aligns well between samples, and dependingon only a small ratio of whole data ( 3% in our experiment) brings out an obvious improvement.
Detailed analyses at a statistical level and experimental level can be found in Appendix B.
For each class c , we randomly take data samples ⟨ X 1 , c ⟩ , ⟨ X 2 , c ⟩ , · · · , ⟨ X n , c ⟩ and generate a set of
CSAs with a pretrained model ˆ f as: P ˆ fc = { p ˆ fX 1 , p ˆ fX 2 , · · · , p ˆ fX n } and train our model as follows:
[Equation]
14) where the α is a coefficient of the CSA p ˆ f .
Experiment Results We conduct experiments on CIFAR10 to verify the effectiveness of our CSA.
We train models on CIFAR10, CIFAR100 and TinyImageNet under PGD-AT [22] and TRADES [43].
Training details can be found in Appendix C. Attacks are crafted within ℓ ∞ bound with ϵ = 8 / 255 .
Weselect PGD-20 attack [22] and AutoAttack (AA) [12] as the attack baselines.
As Table 1 shows, undervarious attacks, our CSA consistently achieves better robustness over defense baselines with little costof natural accuracy.
See Appendix B for more experiment results.
Note that the CSA set of each classcontains the NFC of the first 150 images without shuffling, i.e. , 3% of the training set, which showsthat obtaining CSA only takes little overhead and the feasibility of our method in realistic settings.
Related Work
Adversarial Attack Since the seminal works [32, 16] have been proposed that reveal the vulnerabilityof DNNs, there is an increasing interest in studying stronger adversarial attacks, such as FGSM [16],
C&W attack [5], PGD attack [22], and AutoAttack [12].
The robustness of CNNs in this work hasbeen thoroughly evaluated on these above attack baselines.
Adversarial Defense
The development between attack and defense is just like an arms race: theexistence of adversarial examples promotes the development of defense methods, among which
[Table]
adversarial training (AT) has been considered the most effective one [3].
However, AT has its ownlimitations: i) the trade-off between accuracy and robustness [29, 34, 42, 39], ii) robust overfitting [26,10], which brings a large robustness generalization gap.
Many works have been proposed to improve
AT by sample-wise importance re-weighting [44] or adding more unlabeled data [25, 6].
In thiswork, we pave a new way of class-specific Shapley value-based data augmentation towards betterrobustness, which is orthogonal to the above strategies.
Understanding Robustness in Frequency Domain One way to inspect the intriguing nature ofadversarial attack and defense is through the lens of frequency spectrum.
Many works explore thesensitivity of CNNs and frequency properties of the model [35, 40, 1] and study whether convolutionitself has an intrinsic bias in frequency domain [7].
A hypothesis is that CNNs exploit HFCs whichleads to the lack of robustness [36].
Driven by this hypothesis, many pre-processing defend methodshave been proposed [13, 20, 45].
Meanwhile, some works also come up with adversarial attacksfocusing on LFC [17, 28] which leads to assumptions that constituent frequencies of adversarialexamples are dependent on the dataset [4, 23].
Efforts to quantify and inspect the contribution of eachfrequency component have been made [37, 19].
We employ Shapley value, a tool that has preferableproperties, to quantify the contribution of each frequency component on the fine-grained level.
Explainability with Shapley Value Shapley Value [27] is proposed to distribute contributions amongplayers in a game.
It has been widely applied for the explainability of deep learning.
Some workspropose its approximations [21, 9, 2].
While another line of works focus on the condition to apply
Shapley value [14, 15].
Also, the interaction between input features [31, 30, 41] is recently studied.
Inour work, we apply Shapley value in the frequency domain to quantify the contribution of frequencycomponents.
The quantification have many desirable theoretical properties.
Conclusion & Limitations
In this work, we have made efforts to quantify the contribution of each frequency component on CNNs.
We employ Shapley value, a method with established theoretically desirable properties, to quantifythe contribution of a single frequency component on the instance-level.
Through our quantification,we explore the difference between standard trained (ST) model and adversarial trained (AT) model.
Our analysis of the fairness in AT provides insights of the cause of robustness difference betweenclasses.
We further examine the attack effect of adversarial noises on different frequency components.
Based on our findings, we further propose a simple yet effective class-wise data augmentation method
CSA which augments data with the NFC that we found in this class.
Experimental results have shownthe effectiveness of our method.
Note that our current results are limited mainly to CNNs while webelieve it is also rewarding to develop new techniques to other backbone e.g. Transformer and GNNsbased on our framework, which we leave for immediate future work.
We hope that this work couldinspire future explanations and defense algorithms of adversarial attacks.
Due to the nature of thiswork, there may not be any potential negative social impact that is easily predictable.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes](b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
[Table] [Table]
[Table]
In this Supplementary Material, we first present details of the Shapley value sampling (Appendix A).
Then we give more experimental results on CIFAR-100 and stability analysis of Shapley value(Appendix B).
We further show the training details and ablation study for the sample complexity ofour CSA (Appendix C).
Finally, we add properties of the Shapley value and proof of decompositionof CNNs in frequency domain (Appendix D).
A Details of the Shapley Value Sampling
In this section, we introduce the details of the Shapley value sampling.
A.1 Details of the Model for the Shapley Value Sampling
We sample the Shapley value for models trained on CIFAR10, CIFAR100 and ImageNet.
For
CIFAR10 and CIFAR100, we employ ResNet-18 and train them ourselves.
The ST model on
CIFAR10 and CIFAR100 is trained for 100 epochs with SGD optimizer where learning rate is set tobe 0 . 1 and decayed by 0 .
1 at 75 -th epoch and 90 -epoch.
The momentum is set to be 0 .
9 and weightdecay is set to be 2 e − 4 .
The AT model on CIFAR10 and CIFAR100 is trained with adversarialexamples generated with PGD-10 [22] with a L-inf bound where epsilon is set to be 8 / 255 andstepsize is set to be 2 / 255 .
The other settings of the training of AT model are the same of the STmodel.
For ImageNet, we employ standard ResNet-50 provided in robustbench [11].
A.2 Details of the Sampling of the Shapley Value
We employ Monte-Carlo sampling method for the Shapley value [8] to sample the Shapley value.
For CIFAR10 and CIFAR100, we divide the image into 16 × 16 patches in the frequency domainwhere each patch contains 2 × 2 frequency components and sample 2000 times for each data sample.
For ImageNet, we divide the image into 32 × 32 patches in the frequency domain where each patchcontains 7 × 7 frequency components and sample 1000 times for each data sample.
A.3 Analysis about the Error Bound of our Shapley Value Sampling
As discussed in previous work [24], once we know the variance of the marginal contributions we havean bound for the estimation error of the Shapley Value.
For our estimated Shapley value ˆ ϕ , assumethe probability of the estimation error being greater than ϵ is less than δ that is
[Equation]
With Chebyshev’s inequality, for marginal contributions with variance as σ 2 , the required sampletimes should be m ≥ ⌈ σ 2 δϵ 2 ⌉
We calculate the average variance of the marginal contribution of each patch of the images in thefirst 200 images of each class of the train set of CIFAR10.
The average variance is 1 .
814 × 10 − 3 .
With δ = 0 . 05 and ϵ = 0 .
005 the required sample times is around 1450 .
Therefore, we set thesample times on CIFAR10 and CIFAR100 as 2000 .
Limited by the calculation complexity, we set thesampling times for ImageNet at 1000 .
A.4 Shapley Value Quantification Results
We demonstrate more Shapley Value
Quantification Results on ImageNet and TinyImageNet in
Fig. 8 and Fig.
Each row contains results of a data sample.
We display the origin image, imagereconstructed with PFCs, image reconstructed with NFCs and the heatmap of Shapley value from leftto right.
Note that the for the heatmap of Shapley value, red represents positive and blue representsnegative.
The closer to the center the lower the frequency.
A.5 Time Complexity Analysis for The Sampling of the Shapley Value
The main cost of the sampling of the Shapley value on CNNs comes from generating masks andperforming model inference. Assume that frequency components are devided in to d × d patches, forthe size d , the time complexity is O ( N 2 ) .
For sampling times, the time complexity is O ( N ) .
We report the time consumed for sampling Shapley value for ResNet-18 on CIFAR10 with Intel(R) [Table] [Table]
B Extra Experiment ResultsB.1 Comparison between CSA and High Frequency Filtering Method
High frequency filtering methods have been proposed to defend adversarial attacks [13, 20, 45].
Most high frequency filtering methods are proposed to deal with l 2 norm adversarial attack and arecompared with other non-adversarial training defend methods [13, 20].
We take the suppressinghigh frequency method proposed in [45] as baseline and test the robust accuracy of ResNet18 on
CIFAR10.
As demonstrated in Tab. 5, for l ∞ attack, suppressing high frequency components even slightly 501 degrade the testing accuracy on CIFAR10.
B.2 Experiment Results of vgg16 on CIFAR10
In this section we present the experiment results of vgg16 on CIFAR10, the results are similar tothe results of ResNet-18.
The dimension of fully connected layers of vgg16 is changed from 4096to 512 in order to suit the scale of the image in CIFAR10.
Fig.
8 shows the similar differencebetween adversarial trained vgg16 and standard trained vgg16.
Fig.
9 demonstrate a similar negative
[Table]
B.1 Comparison between CSA and High Frequency Filtering Method High frequency filtering methods have been proposed to defend adversarial attacks [13, 20, 45].
Most high frequency filtering methods are proposed to deal with l 2 norm adversarial attack and arecompared with other non-adversarial training defend methods [13, 20]. We take the suppressinghigh frequency method proposed in [45] as baseline and test the robust accuracy of ResNet18 on CIFAR10.
[Table] [Table]
relationship between the adversarial accuracy of AT model and the average absolute Shapley value of
HFC for ST model with vgg16.
Fig.
10 demonstrate the same attack bias for vgg16 as ResNet18.
B.3 Experiment Results on CIFAR100
In this section, we present the experiment results on CIFAR100.
Shapley value in frequency domain For frequency domain analysis, we conduct the same experiments on CIFAR100 as on CIFAR10.
As shown in Fig.
11, we present the average Shapley value
As demonstrated in Tab. 3, for l ∞ attack, suppressing high frequency components even slightlydegrade the testing accuracy on CIFAR10.
[Table]
B.2 Experiment Results of vgg16 on CIFAR10
In this section we present the experiment results of vgg16 on CIFAR10, the results are similar tothe results of ResNet-18.
The dimension of fully connected layers of vgg16 is changed from 4096to 512 in order to suit the scale of the image in CIFAR10.
Fig.
10 shows the similar differencebetween adversarial trained vgg16 and standard trained vgg16.
Fig.
11 demonstrate a similar negative relationship between the adversarial accuracy of AT model and the average absolute Shapley value of
HFC for ST model with vgg16.
Fig.
12 demonstrate the same attack bias for vgg16 as ResNet18.
[Figure]
[Figure]
[Figure]
model. [Figure] [Table] B.3 Experiment Results on CIFAR100
In this section, we present the experiment results on CIFAR100.
Shapley value in frequency domain For frequency domain analysis, we conduct the same experiments on CIFAR100 as on CIFAR10.
As shown in Fig.
13, we present the average Shapley valueof frequency components of the first 10 images of each class over frequency basis.
The results aresimilar to the results on CIFAR10 except that the absolute average Shapley value of adversarialexamples on ST model are relatively smaller.
Class-wise Shapley value-guided augmentation We train ResNet-18 augmented with our proposed
CSA where the training settings are the same as the settings on CIFAR10.
For each class, we extract
CSA from the first 30 images.
The result is presented in Table 4.
The data augmentation with CSAcould effectively improve the robustness of the model.
B.4 Stability analysis of Shapley Value
We show the variance of Average Shapley Value between different data samples to measure thestatistical stability on Fig.
We plot the error bars of the average Shapley value of frequencycomponents of clean samples on ST model and AT model on CIFAR10 and CIFAR100.
The varianceis relatively big on LFCs and small on HFCs.
For HFCs, the average Shapley value and the varianceon AT model is smaller than that on ST model and near zero which shows that AT model are lessimpacted by HFCs.
[Figure]
C Details of Our Proposed Data Augmentation
C.1 Training Details of CSA
As introduced in Eq. 13, we extract the CSA of images in the train set of CIFAR10 and CIFAR100for ST model.
We train model in the same manner as the AT model trained in Appendix A. Modelsare trained for 100 epochs with SGD optimizer where learning rate is set to be 0 . 1 and decayed by0 .
1 at 75 -th epoch and 90 -epoch.
The momentum is set to be 0 .
9 and weight decay is set to be 2 e − 4 .
For other data augmentations on train data, we employ random horizontal flip with the probability setto 0 .
The adversarial examples are generated with PGD-10 [22] with a L-inf bound where epsilonis set to be 8 / 255 and stepsize is set to be 2 / 255 .
For TRADES[43], the trade-off regularizationparameter β is set to 6 .
The α in Eq.
14 linearly increase from 0 to 0 .
5 through the training process.
C 2 Ablation study for the size of our CSA
[Figure]
The number of CSA we used for data augmentation may affect the effectiveness of the dataaugmentation.
We generate CSA for the first150 images of each class of CIFAR10 that is3% of whole train set.
We train ResNet-18 asdescribed in Appendix C.1 with different number of CSA and test the model with AA [12]and PGD-
[22] at the last epoch.
The result isshown in Fig.
As the number of CSA for each class on CIFAR10 increasing, the test accuracy under AAand PGD-20 attack gradually increase.
At thesame time the data augmentation with CSA is already effective with very few CSA i.e. 30 CSAper class which means the CSA we generatecould represent features that are not correctlylearned by the model.
D Justification for the ShapleyValue on CNNs in Frequency domain
D.1 Properties of the Shapley value
As proved in the seminal work [27], the Shapley value is the only solution that satisfies the followingdesirable properties:
• Null player : A player i is a null player if ∀ S ⊆ N , V ( S ∪ { i } ) =
V ( S ) .
The Shapley value of every null player is zero.
• Efficiency : Shapley value satisfies that (cid:80) i ∈N ϕ N ,V i
[Equation]
• Symmetry :
Two players i, j ∈ N are symmetric if ∀ S ⊆
N\{ i, j } , V ( S ∪ { i } )
=
V ( S ∪ { j } ) .
The Shapley value satisfies that for all symmetric players i, j , ϕ N ,V i
[Equation]
• Linearity :
For two games ( N , U ) , ( N , V ) , if a third game is defined as ( N , W ) where ∀ S ⊆N , W ( S ) =
U ( S )
+ V ( S ) .
It holds that ϕ N ,W i =
ϕ N ,U i [Equation] V i
D.2 The Decomposition of Convolution in Frequency Domain
As shown in Remark 1, the output of a convolution operation can be decomposed into sum of outputof frequency components of X .
For a convolution kernel C ∈ R d 3 × d 4 , the element at i -th row and j -th column of the output of convolution function with the kernel f C is
[Equation]
We represent the d 1 × d 2 matrix where all the elements equal to zero except the element on u -th rowand v -th column is one with I u,vd 1 × d 2 .
We have
[Equation] [Equation]
Then we get
[Equation]
Therefore
[Equation]
we get
[Equation]