I NEQUALITY PHENOMENON IN l ∞ - ADVERSARIAL TRAINING , AND ITS UNREALIZED THREATS
Anonymous authors
Paper under double-blind review
A BSTRACT
The appearance of adversarial examples raises attention from both academia and industry.
Along with the attack-defense arms race, adversarial training is the most effective against adversarial examples.
However, we find inequality phenomena occur during the l ∞ -adversarial training, that few features dominate the prediction made by the adversarially trained model.
We systematically evaluate such inequality phenomena by extensive experiments and find such phenomena become more obvious when performing adversarial training with increasing adversarial strength (evaluated by ϵ ).
We hypothesize such inequality phenomena make l ∞ - adversarially trained model less reliable than the standard trained model when few “important features” are influenced.
To validate our hypothesis, we proposed two simple attacks that either perturb or replace important features with noise or occlusion.
Experiments show that l ∞ -adversarially trained model can be easily attacked when a few important features are influenced.
Our work shed light on the limitation of the practicality of l ∞ -adversarial training.
I NTRODUCTION
Szegedy et al. (2013) discovered adversarial examples of deep neural networks (DNNs), which pose significant threats to the deep learning-based applications such as autonomous driving and face recognition.
Prior to deploying DNN-based applications in real-world scenarios safely and securely, we must defend against adversarial examples.
After the emergence of adversarial examples, several defensive strategies have been proposed (Guo et al., 2018; Prakash et al., 2018; Mummadi et al., 2019; Akhtar et al., 2018).
By retraining l p -norm bounded adversarial samples generated in each training loop, adversarial training (Goodfellow et al., 2015; Zhang et al., 2019; Madry et al., 2018b) is regarded as the most effective defense against adversarial attacks.
The most prevalent adversarial training is l ∞ adversarial training, which applies adversarial samples with l ∞ bounded perturbation by ϵ .
Numerous works have been devoted to theoretical and empirical comprehension of adversarial training (Andriushchenko & Flammarion, 2020; Allen-Zhu & Li, 2022; Kim et al., 2021). For example,
Ilyas et al.
(2019) proposed that an adversarially trained model learns robust features from adversarial examples and discards non-robust features.
Engstrom et al.
(2019) also proposed that adversarial training forces the model learning to be invariant to signals to which humans are also invariant.
Therefore, the adversarial training results in feature representations that are more comparable to humans.
Theoretically validated by Chalasani et al.
(2020), the l ∞ -adversarial training suppresses the significance of the redundant features, and the adversarially trained model, therefore, produces sparser and better-behaved feature representations.
In general, previous research indicates that robust models have a sparse representation of features and view this sparse representation as advantageous because it is more human-aligned.
Several works investigate this such property of an adversarially trained model and attempt to transfer it to standard trained model using various methods (Ross & Doshi-Velez, 2018;
Salman et al., 2020; Deng et al., 2021).
However, contrary to the augmentation of previous work regarding such sparse feature representation as an advantage, we find that such sparseness also indicates inequality phenomena that may pose unanticipated threats to adversarially trained models.
During l ∞ -adversarial training, the model not only suppresses the significance assigned to redundant features (Chalasani et al., 2020) but also suppresses the importance of other features including robust ones.
The degree of suppression is pro- portional to the adversarial attack budget (evaluated by ϵ ).
Hence, for an l ∞ -adversarially trained model, only a handful of features dominate the prediction with the provided input images.
Intuitively, standard-trained models make decisions based on various features, and some redundant features serve as a “bulwark” when a few crucial features are accidentally distorted.
However, in the case of a l ∞ robust model, the decision is primarily determined by a small number of characteristics, so the prediction is susceptible to change when these significant characteristics are modified (see Figure 1).
As shown in Figure 1, an l ∞ -adverarially trained model recognizes a street sign using very few regions of the sign.
Even with very small occlusions, the adversarially-trained model cannot recognize a street sign if we obscure the region that the model considers to be the most important (but are well recognized by humans and standard-trained model).
Even if an autonomous vehicle is deployed with an adversarially trained model that achieves high adversarial robustness against worst-case adversarial examples, it will still be susceptible to small occlusions.
Thus, the applicability of such an adversarially trained model is debatable.
With given feature attribution maps that attribute the importance of each feature, we occlude the image’s features of high importance with small patches.
The resultant image fools the adversarially trained model successfully. ) In this work, we name such phenomenon that only a few features are extreme crucial for models’ recognition as “inequality phenomenon”. we first study the “inequality phenomenon” of l ∞ - adversarial training quantitatively and qualitatively as primarily reflected in two aspects: 1) global inequality: characterized by the dominance of a small number of features.
2) regional inequality: characterized by the tendency of features deemed significant by the model to cluster in particular regions.
Such phenomena are analyzed on ImageNet and Cifar10 with models with distinct architectures.
We further devise attacks to expose the vulnerabilities resulting from such inequality based on our findings.
Experiments demonstrate that, under the premise that human observers can recognize the resulting images,, l ∞ -adversarially trained models are significantly more susceptible than the standard-trained models. Furthermore, they are susceptible to occlusion and noise with error rates of 100% and 94%, respectively, whereas standard-trained models are only affected by 30.1% and 34.5%.
In summary, our contribution can be summed up in the following manner
: • We identify the occurrence of the inequality phenomenon during l ∞ -adversarial training.
We design correlative indices and assess such inequality phenomena from various perspectives (global and regional).
We systematically evaluate such phenomena by conducting extensive experiments on broad datasets and models.
• Then, we identify unrealized threats posed by such inequality phenomena that l ∞ - adversarially trained models are much more vulnerable than standard trained ones under inductive occlusion or noise.
In this case, during the l ∞ -adversarial training, the adversarial robustness is is achieved at the expense of another.
• Our work provides an intuitive understanding of the weakness of l ∞ -adversarially trained model’s feature representation from a novel perspective.
Moreover, our work sheds light on the limitation and the hardness of l ∞ -adversarial training.
This paper is organized as follows.
Background and related work are discussed in Section 2.
The proposed methodology is described in Section 3, then evaluated in Section 4.
Finally, sectionprovides a discussion and conclusion about our work.
B ACKGROUND AND R ELATED W ORK 2. A DVERSARIAL EXAMPLE
Given a model denoted as f ( x ; θ ) : x → R k and training dataset denoted as D , empirical risk minimization (ERM) is a standard way (denoted as standard training) to train the model f through:
[Equation]
where y ∈ R k is the one-hot label for the image and loss ( x, y ) is usually cross-entropy loss.
With such a training scheme, the model typically performs well on clean test samples.
Adversarial examples (Szegedy et al., 2013) aim to generate perturbation superimposed on clean sample x to fool a well-trained model f .
Adversarial example x ′ can be crafted by either following the direction of adversarial gradients (Goodfellow et al., 2015; Kurakin et al., 2016; Madry et al., 2018a) or optimizing perturbation with a given loss (Carlini & Wagner, 2017; Chen et al., 2018).
For most adversarial attacks, the generated adversarial perturbation is bounded by a l p norm ball, where l ∞ - norm is the most commonly used norm.
2. A DVERSARIAL TRAINING
Several defensive strategies are proposed to improve the models’ adversarial robustness (Wong &
Kolter, 2018; Akhtar et al., 2018; Meng & Chen, 2017; Raghunathan et al., 2018).
However, analysis by Athalye et al.
(2018) shows that among various defensive strategies against adversarial examples, only the adversarial training does not rely on the obfuscated gradient and truly increases the model’s robustness.
A model is considered robust against adversarial examples if:
[Equation]
where ϵ represents the magnitude of the perturbation. Therefore, the core idea of adversarial training is to train models with adversarial examples, formally:
[Equation]
The objective max | σ |≤ ϵ loss ( x + σ, y ) introduces the model to minimize empirical risk on the training data points while also being locally stable in the (radius- ϵ ) neighborhood around each of data points x .
The objective is approximated via gradient-based optimization methods such as PGD (Madry et al., 2018b).
Several works attempt to improve adversarial training (Shafahi et al., 2019; Sriramanan et al., 2021;
Jia et al., 2022; Cui et al., 2021).
Interestingly, Ilyas et al.
(2019) proposes that by suppressing the importance of non-robust features, adversarial training makes the trained model more focused on robust and perceptually-aligned feature representations.
In this process, both the gradient and feature representation becomes more sparse.
Chalasani et al.
Salman et al.
(2020); Utrera et al.
(2020) suggests that the feature representation generated by an adversarially trained model is concise as it is sparse and human-friendly.
It only assigns the feature that is truly predictive of the output with significant contributions.
2. H OW INEQUALITY FORMS DURING l ∞ ADVERSARIAL TRAINING
In (Chalasani et al., 2020), they theoretically prove the connection between adversarial robustness and sparseness: During l ∞ -adversarial training, supposed the adversarial perturbation σ satisfying || σ || ∞ ≤ ϵ , the model attempts to find robust features serving as strong signals against perturbation. Meanwhile, the weights of non-robust ones which serve as relatively weak signals, are more aggressively shrunk toward zero. The shrinkage rate is proportional to adversaries’ strength (evaluated by ϵ ).
In other words, standard training can result in models where many non-robust features have significant importance for models, whereas l ∞ -adversarial training tends to selectively reduce the magnitude of the significance of non-robust features with weakly relevant or irrelevant signals and push their significance close to zero.
In the end, the feature attribution maps generated by gradientsbased feature attribution methods (Smilkov et al., 2017; Lundberg & Lee, 2017; Sundararajan et al., 2017) look more sparse.
They regard such sparseness as a merit of adversarial training as it produces more concise and human-aligned feature attributions.
However, we further study such sparseness and find it introduces a phenomenon of extreme inequality, which results in unanticipated threats to adversarially trained model.
M ETHODOLOGY
In this section, we introduced the index used to measure the inequality degree of the test data point’s feature attribution map.
Then we propose two types of attacks to validate our hypothesis: extreme inequality results in threats to l ∞ -adversarially trained model.
3. M EASURING THE INEQUALITY OF A TEST DATA POINT
Firstly, feature attribution maps are required to characterize the inequality degree by given test data point x and model f .
Several feature attribution methods have been proposed in recent years (Smilkov et al., 2017; Lundberg & Lee, 2017; Sundararajan et al., 2017).
In general, feature attribution methods rank the input features according to their purported importance in model prediction.
To be specific, we treat the input image x as a set of pixels x = { x i , i = 1 ...M } and denote the generated feature attribution map of x of model f as A f ( x ) , where A f ( x ) is composed of a i .
Feature attribution methods attribute an effect a i to each feature x i , and summing the effects of all feature attributions approximates the output f ( x ) .
x i achieves the top-most score ( a i ) is considered the most important feature for prediction, whereas those with the bottom-most score are considered least important.
With a given sorted A f ( x ) = { a i , i = 1 ...
M | a i < a i +1 } generated by a typical feature attribution method, if the prediction f(x) can be approximated with the sum of N most important features and
N is much less than M , we name such distribution of A f ( x ) is unequal.
Namely, the prediction on x made by model f is dominated by few pixels.
Formally, we use Gini index (Dorfman, 1979) to measure the inequality of the distribution of a given feature attribution map.
Formally, given a population set indexed in non-decreasing order Φ = { ϕ i , i = 1 ...n | ϕ i ≤ ϕ
i +1 } , Gini coefficient can be calculated as: gini
[Equation]
An advantage of the gini ( · ) index is that inequality of the entire distribution can be summarized by using a single statistic that is relatively easy to interpret.
The Gini index ranges from 0, when the value of every ϕ i is equal, to 1, when a single ϕ i takes all the sum.
This allows us to compare the inequality degree among feature attributions with different sizes.
We define two types of inequality as follows:
• Global inequality: Given a feature attribution map A f ( x ) = { a i , i = 1 ...
M | a i < a
i +1 } on test data point x , we only consider the inequality degree of the global distribution of A f ( x ) and take no into account for other factors, the inequality degree is calculated with gini g
( A f ( x ))
directly.
The higher of gini g ( A f ( x )) , the more unequal the distribution
A f ( x ) , the fewer pixels take the most prediction power.
When gini g ( A f ( x )) is equal to 1, it indicates one pixel dominates the prediction while all the other pixels have no contribution to the current prediction.
• Regional inequality: We also consider inequality degree together with spatial factor, whether important feature tends to cluster at specific regions.
A region is defined as a block with size of n ∗ n of input space.
We first divide pixels into different regions and calculate the sum of pixels’ importance by regions, formally, A fr ( x ) = (cid:8) a r i , i = 1 ...m | a r i < a r
i +1 (cid:9) , where a r is the sum of a i in the region. Therefore, the Gini value on A fr ( x ) reflects the inequality degree of different regions of input space.
The higher the value of gini r ( A fr ( x )) , the more important features are tend to cluster in the specific regions.
When gini r ( A fr ( x )) is equal to 1, it represents all pixels contribute to the prediction cluster in one region (block).
In what follows, we propose potential threats caused by such inequality (global and regional inequality).We devise attacks utilizing common corruptions (noise and occlusion) to reveal the unreliability of such decision pattern by l ∞ -adversarially trained model.
3. A TTACK ALGORITHMS
We propose two simple attacks to validate potential threats caused by such inequality phenomena:
1) inductive noise attack.
2) inductive occlusion attack.
3.2. I NDUCTIVE N OISE A TTACK
We evaluate the models’ performance under attacks designed by two types of noise.
• Noise (Type I):
Given a image x , we perturb the most influential pixels of images with Gaussian noise σ ∈ N (0 , 1) via masking M .
Formally:
[Equation]
where M
[Equation]
0 ,a i < a tre
[Equation]
where a tre represents the threshold. x i with value that is below to the a tre will be kept, and x i whose a i ≥ a tre is perturbed by Gaussian noise.
• Noise (Type II): About the second type of noise attack, we directly replace important pixels with Gaussian noise, formally x ′ =
M ∗ x + M ∗ σ , where M represents reverse mask of M .
Compared to Noise-I, Noise-II replace important pixels totally and disturbs images more severely.
If the model’s decision pattern is extremely unequal, the performance will be highly influenced when important features are corrupted by inductive noise attacks.
3.2. I NDUCTIVE O CCLUSION A TTACK
With respect to inductive occlusion attack, we obscure regions of important pixels with occlusions gradually.
During the attack, the max count of occlusions is N with radius at max R . The order of regions to perturb is decided by the value of A fr ( x ) , that region of higher a r i is perturbed in priority by occlusions with size r ∈ { 1 ...R } . The number of occlusions is constrained by n ∈ { 1 ...N } .
We also consider occlusion with different colors to reflect potential real-world occlusion.
The inductive occlusion attack algorithm is listed as follows:
Require:
Test data point ( x, y ) , Model f , Regional Attribution map A fr ( x ) , Max count and radius
N, R , Perturb color c .
Ensure: f ( x ) =
y
▷ Ensure the test data x is correctly classified by model f . break end return
Note the intention of this work is not to propose strong adversarial attacks.
Although either noise or occlusion is beyond the threat model considered in l ∞ adversarial training, we intend to reveal the threats caused by such inequality phenomena that previous work ignored.
In summary, the extreme inequality decision pattern of l ∞ -trained adversarial models result in themselves more fragile under some corruptions.
E XPERIMENTS
In this section, we first outline the experimental setup.
We then evaluate the inequality degree (by Gini) of different models.
Then, we evaluate the performance of under the proposed attacks.
Finally, we perform an ablation study about the selection of feature attribution methods.
4. E XPERIMENTAL SETTINGS
Dataset and models.
We perform a series of experiments on ImageNet Deng et al.
(2009) and CIFAR10 Krizhevsky et al. With respect to experiments on ImageNet, we use ResNet18 (He et al., 2016), ResNet50, WideResNet50 (Zagoruyko & Komodakis, 2016) provided by Microsoft 1 . For CIFAR10, we use ResNet18, DenseNet (Huang et al., 2017).
See A.1 for more details of models including standard and robust accuracy.
Regarding feature attribution methods (implementation by Captum 2 ), we consider methods including Input X Gradients (Shrikumar et al., 2016), Integrated Gradients (Sundararajan et al., 2017), Shapley Value (Lundberg & Lee, 2017) and SmoothGrad (Smilkov et al., 2017).
Considering space and time efficiency, we present experimental results based on Integrated Gradients and perform an ablation study on the other feature attribution methods.
Metrics.
For all the tests about the models’ performance, we use error rate (%) as the metric to evaluate the model’s performance under corruptions (e.g., noise and occlusions), which is the proportion of misclassified test images among the total number of test images defined as
N (cid:80) Nn
=
[ f ( x ) ̸
= f ( x ′ )] , where x represents clean test images, and x ′ represents test images corrupted by noise and occlusions.
4. I NEQUALITY TEST
In this section, we first evaluate the inequality degree (both global and regional inequality) of l ∞ - adversarially trained models and standard trained models with different architectures (ResNet18,
ResNet50, WideResNet, DenseNet) trained on ImageNet and Cifar10.
We also evaluate the inequality degree of different models adversarially trained with increasing l ∞ -norms ( ϵ = 1, 2, 4, 8).
In the case of the evaluation on Gini, We applied the Gini index to the sorted absolute value of the flattened feature attribution maps.
On evaluating regional inequality, the region’s size is 16 ∗ 16 for experiments on ImageNet and 4 ∗ 4 for CIFAR10.
The results are presented in table 1.
As shown in Table index across different models.
We evaluate the Gini coefficient of different models trained with different ϵ on ImageNet and Cifar
[Table]
1, on Cifar10, the global inequality degree of the standard trained model with different architectures is around 0.58.
The gini (global inequality) for l ∞ -adversarially trained model is around 0.73 when ϵ = 8 .
However, on ImageNet, the inequality phenomena are much more obvious.
Especially for an adversarially trained Resnet50 ( ϵ = 8 ), the gini achieves 0.94, which indicates that only a handful of pixels decide the prediction.
Experiments on Cifar10 and ImageNet show that l ∞ -adversarially trained models rely on fewer pixels to support the prediction with the increasing of the adversarial strength ( ϵ ).
We also test the inequality degree on different classes belonging to ImageNet; classes related to animal tends to have higher value on gini index. For example, class ‘Bustard’ has the highest value on gini of 0.950.
Classes related to scene or stuff tend to have lower value on gini index. For example, class ‘Web site’ has the lowest inequality of 0.890 (See more results in Appendix A.7).
We visualize the features’ attribution of given images for the standard and adversarially trained ResNet50 respectively in Figure 2.
As shown in Figure 2, when the model is adversarially trained with weak adversarial strength ( ϵ = 1 ), the model has better feature attribution aligned to human observers.
However, when the adversarial strength increases, the model gradually assigns higher importance to fewer pixels and resulting in extreme inequality regarding feature attribution.
Moreover, these important features tend to gather in few specific regions ( see more visualization results in Appendix A.10).
4. E VALUATION UNDER I NDUCTIVE NOISE A TTACK
In this part, we compare the performance of standard- and adversarially- trained ResNet50 under random and inductive noise.
For a fair comparison, we first select 1000 random images from ImageNet that are correctly classified by all the models.
We set noise with different scales, including subpixels of 500, 1000, 5000, 10000, and 20000.
We present the results in Figure 3.
Under random noise, the success rate of attack on adversarially-trained model achieves 73.4%, but only 18.8% for standard-trained model.
Under Noise of Type I, the adversarially trained model is fooled by 94.0%, while the standard trained model is only fooled by 34.5%.
Under Noise of Type II, even when we control the amount of noise with a small threshold (e.g., 1000 subpixels), more than 50% of predictions made by the adversarially trained model is affected.
When we enlarge the threshold to 20000, the adversarially-trained model ( ϵ =8) is almost fooled with a 100% success rate.
In summary, compared to the standard trained model, l ∞ -adversarial model relies on fewer features to make a decision; such a decision pattern results in unstable prediction under noise.
4. E VALUATION UNDER I NDUCTIVE O CCLUSION A TTACK
In this part, we perform an inductive occlusion attack and evaluate the standard trained and l ∞ - adversarially trained ResNet50s’ performance.
We set two group experiments with different thresholds.
Also, for a fair comparison, we first select 1000 random images that are correctly classified by all the models.
[Table]
In the first group of experiments
[Table]
we generate occlusions with a max count of 5 and a max radius of 10.
In this case, the adversarially-trained model is fooled at a 71.7% error rate, but the standard trained model’ predictions are only affected by 31.6%.
When we enlarge the threshold and set max count as 10 and radius as 20 for occlusions, both ϵ = 4 and ϵ =adversarially trained model can be fooled with 100% success rate while only 41.2% attack success rate for the standard-trained model.
We visualize the results in Figure 4.
As the figure shows, l ∞ -adversarially trained model with larger ϵ could be easily attacked by smaller occlusions even under the same threshold.
For example, in Figure 4, the standard trained model can recognize ‘Bulbul‘ well with the head part occluded, but the adversarially trained model fails to recognize the ‘Bulbul’ if only the beak of the bulbul is occluded.
Moreover, compared with adversarial perturbation, occlusion is more practical as occlusion frequently appear in the real world.
4. A BLATION S TUDY
We consider four attribution methods for ablation study: Input X Gradient (Shrikumar et al., 2016),
SmoothGrad (Smilkov et al., 2017), Gradient Shapley Value (GradShap for short) (Lundberg & Lee, 2017) and Integrated Gradients (Sundararajan et al., 2017).
More details and parameter settings of feature attribution methods can be seen in Appendix A.8.
We perform an ablation study to evaluate the effect of selection on the feature attribution methods (see Table 3)
[Table] [Table]
Among various attribution methods
[Table]
SmoothGrad produces more spare feature attribution maps and thus results in higher values on Gini.
Regarding evaluation under noise, SmoothGrad increase inductive noise attack’s success rate.
Regarding evaluation under occlusion, Integrated Gradients improves the attack’s success rate on models.
In conclusion, the selection of attribution methods slightly affects attacks’ success rates but does not change our conclusion: the distribution of features’ attribution by the adversarially trained model is more unequal; such inequality makes the adversarially trained model more susceptible to inductive noise and occlusions.
D ISCUSSION AND C ONCLUSION
In this work, we study the inequality phenomena that occur during l ∞ -adversarial training.
Specifically, we find l ∞ -adversarially trained models’ feature attribution is not as aligned with human perception as we expect. An ideal human-perceptual aligned model is expected to make decisions based on a series of core feature attributions. For example, if the model classifies an input image as a bird, it should take attributions, including the eye, the beak of the bird, and the shape of the bird, all of these attributions into account. However, we find l ∞ -adversarially trained model only relies on individual attribution (only the bird’s beak) for recognization.
We name such phenomena as inequality phenomenon.
We perform extensive experiments to evaluate such inequality phenomena and find that l ∞ robust model assigns a few features with extremely high importance.
Thus, a few features dominate the prediction.
Such extreme inequality of l ∞ adversarially trained model result in unreliability.
We also design attacks (by utilizing noise and occlusion) to validate our hypothesis that adversarially trained models could be more susceptible under some scenarios.
We find an attacker can easily fool the l ∞ -trained model by modifying important features with either noise or occlusion easily.
We suggest that both noise and occlusion are common in a real-world scenario.
Robustness against either noise or occlusion is more essential and crucial than robustness against adversarial examples.
Our work reveals the limitation and vulnerability of the current l ∞ -adversarially trained model.
We also propose a strategy to release such inequality phenomena during l ∞ -adversarial training.
We combine Cutout (DeVries & Taylor, 2017) strategy with adversarial training and force the model learning features from different regions by cutting out part of training images at each iteration during the training (see the result in Appendix A.9).
The strategy slightly releases the inequality degree of the adversarially trained model.
More effective strategies releasing such extreme inequality could be a crucial and promising direction as future work.
Besides safety and security issues, this work provides an intuitive understanding of adversarial models’ feature representation.
We hope our work can motivate new research into the characteristics of adversarial training and open up further challenges for reliable and practical adversarial training.
E THICS S TATEMENT
In this paper we identify inequality phenomena occur during l ∞ -adversarial training, that l ∞ - adversarially trained model tends to use few features to make the decision.
We give a systematical evaluation of such inequality phenomena across different datasets and models with different architectures.
We further identified unrealized threats caused by such decision patterns and validated our hypothesis by designing corresponding attacks.
Our findings provide a new perspective on inspecting adversarial training.
Our goal is to understand current adversarial training’s weaknesses and make DNNs truly robust and reliable.
We did not use crowdsourcing and did not conduct research with human subjects in our experiments.
We cited the creators when using existing assets (e g., code, data, models).
R EPRODUCIBILITY S TATEMENT
We present the settings of hyper-parameters and how they were chosen in the experiment section.
We repeat experiments multiple times with different random seeds and show the corresponding standard deviation in the tables.
We plan to open the source code to reproduce the main experimental results later.
A A PPENDIX A. D ETAILS OF M ODELS A.1. D ETAILS OF M ODELS TRAINED ON I MAGE N ET
We summarize the clean and robust accuracy of models trained on ImageNet in Table ?? .
Regarding robust accuracy, we use PGD for evaluation.
During the training on ImageNet, the images are resized
[Table]
to 256 using interpolation=InterpolationMode.
BILINEAR, followed by a central crop of size=224.
Finally, the values are to then
[Table]
TRAIN TRANSFORM = t r a n s f o r m s .
Compose ( [ t r a n s f o r m s .
Resize (256) , t r a n s f o r m s .
CenterCrop (224) , t r a n s f o r m s .
RandomHorizontalFlip ( ) , t r a n s f o r m s .
ToTensor ( ) , ] )
TEST TRANSFORM = t r a n s f o r m s .
Compose ( [ t r a n s f o r m s .
Resize (256) , t r a n s f o r m s .
CenterCrop (224) , t r a n s f o r m s .
ToTensor ( ) , ] )
A.1. D ETAILS OF M ODELS TRAINED ON C IFAR
We summarize the clean and robust accuracy of models trained on CIFAR10 in Table 5.
Regarding robust accuracy, we use AutoAttack Croce & Hein (2020) for evaluation.
[Table]
TRAIN TRANSFORM = t r a n s f o r m s .
Compose ( [ t r a n s f o r m s .
RandomCrop (32 , padding =4) , t r a n s f o r m s .
RandomHorizontalFlip ( ) , t r a n s f o r m s .
ToTensor ( ) , ] )
TEST TRANSFORM = t r a n s f o r m s .
Compose ( [ t r a n s f o r m s .
ToTensor ( ) , ] )
[Figure]
F URTHER DISCUSSION ABOUT F EATURE ATTRIBUTION
A. V ISUALIZATION FOR C IFAR
We visualize feature attribution maps and attack results for Cifar10.
About the setting of attack, we set max count N = 10 , with max radius R = 4 .
With occlusion with different colors (black, white and grey), success rates on 1000 correct classified images of l ∞ -adversarially trained model are 60.4%, 60.5%, and 38.1% repsectively.
And success rates for standard trained model are 34.
%, 36.7% and 24.1% respectively.
We visualize corresponding results in Figure 8.
[Table]
A. E QUALITY DEGREE OF DIFFERENT CLASSES
We test inequality degree of feature attributions’ distribution of 50000 samples from 1000 classes in
ImageNet.
We present results in Table 6.
[Table]
Classes with top-5 inequality (gini value) are: Bustard, Manhole cover, Oystercatcher, Redshank and Pomeranian.
And their Gini values are 0.950, 0.949, 0.949, 0.949 and 0.949.
Classes with bottom-5 inequality (gini value) are: Web site, Slot, Grocery store, Grille and Comic book.
Their corresponding gini values are: 0.890, 0.900, 0.901, 0.901 and 0.905.
Regarding regional inequality, we present results in Table 7.
Classes with high regional inequality
[Table]
are similar to classes with high global inequality.
Specifically, classes of the top 5 (regional inequality) are Redshank, American coot, Oystercatcher, Bustard and Gazelle.
And the gini r ( .
A. D ETAILS ABOUT F EATURE A TTRIBUTION M ETHOD
We use attribution methods including Input X Gradient, Smooth Gradient (short for SmoothGrad),
Gradient Shapley Value (short for GradShap) and Integrated Gradients.
Input X Gradient: The Input X Gradient multiplies input with the gradient with respect to each input feature.
It is a baseline approach for computing the attribution.
GradShap: Shapley Values aims compute each feature’s attribution based on cooperative game theory.
GradShap approximates Shapley Values by computing the expectations of gradients by randomly sampling from the distribution of baselines .
It adds noise to each input sample n samples times, selects a random baseline from the baselines’ distribution, and a random point along the path between the baseline and the input.
Then it computes the gradient of outputs with respect to those selected random points.
In our evaluation, we set n samples = 20 for experiments with ImageNet and n samples = 10 for experiments with Cifar10.
We set baseline = 0 for all the experiments.
Integrated Gradients: Integrated Gradients is an axiomatic model which assigns an importance score to each input feature by approximating the integral of gradients of the model’s output with respect to the inputs along the path (straight line) from given baseline to inputs.
Previous work points out Integrated Gradients method is sensitive to the choice of path.
To reduce such sensitivity,
Integrated Gradients are usually repeated for n step steps.
For all the experiments, we set n step = 20 , and baseline ∈ N (0 , 1) .
SmoothGrad:
SmoothGrad adds gaussian noise to each input in the batch n samples times, then applies the given attribution algorithm to each of the samples.
It returns the mean of the sampled attributions.
SmoothGrad returns a sparser feature attribution map than other methods.
In our experiment, we set n samples =
A. H OW TO RELEASE INEQUALITY PHENOMENON
We also try to propose a strategy to release the inequality phenomenon in adversarial training.
Intuitively, we hope adversarially trained models learns to find robust features from the whole image rather than focus on a specific robust feature.
Towards this end, we incorporate Cutout with adversarial training that Cutout enables the model to learn features from multiple spatial spaces.
We evaluate our strategy on Cifar10.
The results are presented in Table 8.
[Table]
As the table indicates, the Cutout strategy can slightly release the inequality of l ∞ -adversarial training.
But at a price, both clean accuracy and adversarial accuracy slightly decrease.
Regarding performance under noise and occlusion, the strategy does not improve the adversarial trained model’s performance under noise but improves its performance under occlusions.
A more effective strategy to release inequality phenomena is highly required.
[Figure]